DIY Vendor agnostic Smart Shunt & Victron Smart Solar MPPT data logger based on ESP8266

[rin67630]

using Serial - it doesn't respond to any command.....

It can't: Serial is busy with the Victron port.
You could try with TERM_IS_SOFTSER to check, but Telnet is ways superior.

 

[cbfraser]

yes switched to time.google.com - it works (but not every time)

I meant where in the code would you prefer to see these definitions?

 

[cbfraser]

Yours (in Canada i assume), of course.

so to recap:
compiles without errors (ESP32) serial & telnet
wifi gets IP address
NTP works (not always - will try different NTP servers)
in SERIAL mode console input not responded to - if I replace Console1.read with Serial.readString(); menu system
runs but not serialRun - is Console1.read a library function?
added TelnetStream library but my telnet client will not connect, but I can ping it
tried softser but need library for <SoftwareSerial.h> - is it softserialtx -----found it ESPsoftserial, will update after trying that
thanks

 

[rin67630]

so to recap:
compiles without errors (ESP32) serial & telnet

An ESP32 is not ideal, I haven't actually tested it with the actual code.

wifi gets IP address
NTP works (not always - will try different NTP servers)

Hmm why? Is your network wonky?

in SERIAL mode console input not responded to - if I replace Console1.read with Serial.readString(); menu system
runs but not serialRun - is Console1.read a library function?

Console1 is a macro defined in a_Libs_Vars:

#if defined(TERM_IS_SOFTSER)
#include <SoftwareSerial.h>
#define Console1 SoftwareSerial
#endif

#if defined(TERM_IS_SERIAL)
#define Console1 Serial
#endif

#if defined(TERM_IS_TELNET)
#include <TelnetStream.h>
#define Console1 TelnetStream
#endif

So during compilation it is replaced by Serial. or SoftwareSerial. or TelnetStream. respectively.

added TelnetStream library but my telnet client will not connect, but I can ping it

Maybe because your network connection is wonky (NTP is unreliable)?

tried softser but need library for <SoftwareSerial.h> - is it softserialtx -----found it ESPsoftserial, will update after trying that
thanks

The SoftwareSerial library is normally built in the Arduino IDE. You don't have to install anything.

 

[rin67630]

yes switched to time.google.com - it works (but not every time)
I meant where in the code would you prefer to see these definitions?

Your should replace de.pool.ntp.org with ca.pool.ntp.org if you are in Canada.
Tell me if it works reliably.

 

[cbfraser]

I have been able to get softserial working on an ESP32. See below. When I get time
I'll get back to V on Steriods and sort out the issues there and update you. Thank you again for all your help.

15:32:20.189 PID= 0XA057
15:32:20.190 FW= 161
15:32:20.190 SER#= HQ23189PW9P
15:32:20.191 Battery Voltage: V= 25470
15:32:20.193 I= 1300
15:32:20.194 VPV= 40420
15:32:20.195 Panel Power: PPV= 36
15:32:20.197 CS= 3
15:32:20.197 MPPT= 2
15:32:20.198 OR= 0X00000000
15:32:20.200 ERR= 0
15:32:20.200 LOAD= ON
15:32:20.201 total yield: H19= 524
15:32:20.203 H20= 27
15:32:20.204 H21= 69
15:32:20.205 Yesterday's yield: H22= 128
15:32:20.207 H23= 727
15:32:20.208 HSDS= 4

 

[solarhombre]

[...]
NTP works (not always - will try different NTP servers)
[...]

I have a bunch or esp8266/esp32 running tasmota or esphome. Some of them work best when their time is consistently accurate, and already stretch the limit of what the 1 or 2 CPU cores can do. After trying to add more stratum 1 servers and other different configs, the solution I implemented was to run an NTP server in my x86-64 firewall and dnat any NTP connection trying to reach external NTP servers to my local one. It's super fast, always works, and frees up precious cycles on the busiest esp8266s.

 

[rin67630]

...
wifi gets IP address
NTP works (not always - will try different NTP servers)
...

If NTP does not work, one should increase the delay in setup

  getNTP();
  delay(6000);
  getEpoch();  // writes the Epoch (Numbers of seconds till 1.1.1970...

It's not unusual that the NTP server responds slowly.
you can increase it to 15 seconds.

 

[rin67630]

For those following this project:
Initially, it was ESP8266 only.
I am currently rewirting the code to accept ESP32 modules as well.
And especially the TTGO-T1 board, which features a gorgeous color display:

It gets a boot report giving all important information like the IP address, whether the menu is serial or Telnet, the Thinger device and User...
on screen, which is important when you retrieve an already programmed device and don't know it's status to access it.
It has several display modes, which can be changed from the two on-board buttons.

I expect to deliver the code by next week. If you are interested, you may already order a TTGO-T1 module.
Additionally I provide the option to get a Black and White 128 x 64 Pixel display on either an ESP8266 or an ESP32.
This one is less pretty and cannot display as much information, but draws less current and is cheaper.

By the way: The ESP32 has also a remote mode and can display the values sent from a local ESP8266 module in your living room.

Expect nice things to come soon...

 

[rin67630]

Just a remark for those on Windows.
If you want a connection with Telnet, don't use the broken Windows' Telnet option, which does not handle CR/LF correctly.
Get WSL (the Windows' Subsystem Linux) and from there, use a working Telnet to communicate with your IoT devices.

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\Users\MiFi> WSL
mifi@MBP2014Windows:/mnt/c/Users/MiFi$ telnet 192.168.188.41
Trying 192.168.188.41...
Connected to 192.168.188.41.
Escape character is '^]'.

Waiting for input
1
Switching display to 1
2
Switching display to 2
3
Switching display to 3
B

 Battery Stats
 Hour   |  00   |  01   |  02   |  03   |  04   |  05   |  06   |  07   |  08   |  09   |  10   |  11   |
 Bat Ah | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
 Bat V  | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
 Hour   |  12   |  13   |  14   |  15   |  16   |  17   |  18   |  19   |  20   |  21   |  22   |  23   |
 Bat Ah | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
 Bat V  | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |

 

[SeaGal]

Just a remark for those on Windows.
If you want a connection with Telnet, don't use the broken Windows' Telnet option, which does not handle CR/LF correctly.
Get WSL (the Windows' Subsystem Linux) and from there, use a working Telnet to communicate with your IoT devices.

... or PuTTY

 

[wpns]

... or PuTTY

Another vote for Putty. Installing WSL just to get telnet is a lot of work. [Yes, did the WSL thing, went back to macOS, Raspberry Pi, Debian on old Mac Minis, etc. Life’s too short. Windows is for gaming and windows-only apps.]

 

[rin67630]

The remote ESP32 is programmed and I am providing 6 colour screens to display the information gathered and derived from either a Victron SCC or a shunt (vendor agnostic)

• Screen 1: Output
• Screen 2: Panel
• Screen 3: Overview
• Screen 4: Battery
• Screen 5: Coulombs (integrating current hour, last hour, Day)
• Screen 6: Day's harvest (Yellow= positive, Red= negative)
  Common information on all screens:
• top: V Panel, VBat, I Bat
• bottom: Time / Date
• right: Bar graph SOC Battery.
  
  You can switch screens over Telnet from your computer screen (here seen on the background), or using the hardware buttons of the ESP32 TTGO.
  Communication between the local ESP8266 and the home's ESP32 TTGO occurs over WiFi (not short-range Bluetooth)
  Basically, if you open the ports on your Router and you have a fixed IP, the remote display my be operated worldwide.
  The communication is not encrypted, but nothing harmful can be done:
  There is no OS involved, Telnet can only run the menu, the data is transmitted in floating point binary and does not contain anything valuable for others.
  
  Enjoy!

 

[rin67630]

Just a few lines of code more: With the parameter A0_IS_SIMUL, and using a stock Witty ESP8266 board, I have now a light-driven panel+SCC simulator. For me, it is convenient to get some life with halfway plausible values onto my dashboards and statistics in order to debug without the need to have a cumbersome hardware to carry with me.
I also have configured a new "Statistics" dashboard in Thinger.io to get a view on how yesterday did perform:

 

[rin67630]

Today I have concentrated on documentation.
I made an overview synoptic to facilitate the understanding of what's offered:

Currently, if you just want to sniff around without yet investing in hardware, buying a bare Witty ESP8266 board (3-4$) is all you need.

Next development: Next, I am trying to generically estimate the percentage of charge.
First, I am computing the quiescent cell voltage (battery voltage - internal resistance * battery current) divided by the number of cells.
Then I am using two chemistry specific tables, (one for charging, one for discharging), and am doing an interpolation mapping out of that.
It is far from being perfect, but it gives an approximate value.

 

[rin67630]

Now the project has got an extensive user manual explaining all steps to realize the project according to your needs:
User manual
you might want to directly download the PDF (30 pages) to read it off line.

 

[rin67630]

I have now posted a new version with following improvements:
- can be compiled for ESP8266 and ESP32 with the same code.
- streamlined configuration, no need to specify which ESP is used.
- option to determine the thinger.io device name from the ESP-MAC address (tu use the same compiled result on several devices)
- greatly improved WiFi initialisation It starts from stored WiFi, else from Credentials, else from SmartConfig.
- new menu command X to restart, the ESP forgetting WiFi credentials to move the ESP to a new network over SmartConfig.
- beta-version of a serial connection option to DROK DC-DC converters.
Enjoy !