diy solar

diy solar

JK-BMS-CAN with new Cut-Off Charging Logic (open-source)

Back with another question.

My 6000xp inverter seems to disable any loads ( and throws a low voltage notice in its logs ) when SOC hits 15% ( and the lowest cell voltage was at 3.16v (16s pack ))

I have 2.64v ( perhaps a little low ) set as 0% SOC in the JK BMS settings. Are there any yaml settings I can edit to push my batteries a little further here?
 
Back with another question.

My 6000xp inverter seems to disable any loads ( and throws a low voltage notice in its logs ) when SOC hits 15% ( and the lowest cell voltage was at 3.16v (16s pack ))

I have 2.64v ( perhaps a little low ) set as 0% SOC in the JK BMS settings. Are there any yaml settings I can edit to push my batteries a little further here?
Are you sure it's a BMS triggered disconnect and not the Inverter Thresholds for Shutdown / Alarm / Warning based on Battery Voltage ?

Cannot remember what I have on the Deye now that I enabled BMS communication, but before I for sure had a Shutdown Voltage, a Warning / Low-Battery Voltage as well as a Restart Voltage (that once recovered re-enables the Inverter to supply the Loads).
 
Are you sure it's a BMS triggered disconnect and not the Inverter Thresholds for Shutdown / Alarm / Warning based on Battery Voltage ?

Cannot remember what I have on the Deye now that I enabled BMS communication, but before I for sure had a Shutdown Voltage, a Warning / Low-Battery Voltage as well as a Restart Voltage (that once recovered re-enables the Inverter to supply the Loads).
My friend, you are completely right. ;)
 
@Sleeper85 , @MrPablo : Somewhat unrelated (and maybe obvious to everybody except me ...)

I tried switching a Test HomeAssistant Instance (basically an empty one where I just setup some MQTT entities logging) for test Purposes.

I converted the SQLite3 Database from this Test HomeAssistant Instance ({DATA_FOLDER}/home-assistant_v2.db) to PostgreSQL then finally to TimescaleDB-HA (PostgreSQL + TimescaleDB Extension + PostGIS Extension + TimescaleDB Toolkit) to see if that would Improve the Performances. This is on a KVM Virtual Machine with ZFS Pool/Dataset inside the Virtual Machine (AMD64 Host with Proxmox VE with ZFS Pool/Dataset also) running with ZFS and Compression set to LZ4.

The Migration was performed using a Script I developed. I lost basically 2 days dealing with ramdom Podman (Docker) Network Hostnames & DNS Failures that occurred on a random Basis due to some Network Configuration that for some Reason was dangling around in my temporary User Folder :( .

Anyways here it is in case anybody is interested:

The Script will run in Podman/Docker directly on the Host where your Docker Container Database is. The Advantage is that will NOT require any Authentication & Data Transfer over UNPROTECTED Network Connection (by default PostgreSQL is NOT using SSL, at least in PostgreSQL Docker Containers). The communication can therefore be done directly "across Containers", with no need to send data over unencrypted connections.

The Script takes care of:
- Migrate the SQLite3 Database (home-assistant_v2.db) to a (temporary) PostgreSQL using pgloader
- Fix Sequences on (temporary) PostgreSQL database using psql (otherwise you'll get lots of errors when HomeAssistant tries to write to the Database, complaining about non-Unique Keys, since the Sequences were NOT updated automatically)
- Backup the (temporary) PostgreSQL Database using pg_dump
- Initialize the (production) TimescaleDB-HA Database according to the Instructions on the TimescaleDB Website
- Import/Migrate (Restore) the (temporary) PostgreSQL Database into the (production) TimescaleDB-HA Database
- Finalize the (production) TimescaleDB-HA Database according to the Instructions on the TimescaleDB Website

Of course there is a bit of Configuration Required and you need to bring down your HomeAssistant Container before starting the actual migration (and prevent it from Starting again until the Migration is finished). On the other side you need to start the (temporary) Postgresql Database and the (production) TimescaleDB-HA Database before starting the Migration.

I plan to test it a bit better by copying the "Production" home-assistant_v2.db File to a Test Virtual Machine and try to do the Conversion with that. That will most likely spot some new Issues, since importing 6GB of Data will probably be a bit more challenging (RAM, CPU, time) than 2MB of an almost empty Database.


If Somebody is Interested in the ZFS (or Other File System) Compression Data Information:
Bash:
#!/bin/bash

# Define Folder / File
target=${1-""}
if [[ -z "$target" ]]
then
   read -p "Enter the Target File or Folder to Analyze: " target
fi

# Check if File or Folder or if doesn't Exist
if [[ -d "${target}" ]]
then
   echo "Analyzing Folder ${target}"
elif [[ -f "${target}" ]]
then
   echo "Analyzing File ${target}"
else
    echo "Target ${target} does NOT Exist. Aborting !"
    exit 9
fi

# Declare Function
get_size() {
   # Target File / Folder
   local ltarget="${1}"

   # The Arguments for "du" Command is passed by nameref
   declare -n largs="$2"  # Reference to output array

   # Run Command and Filter Output
   #local lresult=$(du ${largs[*]} "${ltarget}" | sed -E "s|^([a-zA-Z0-9_-]+)\s*?(.*)$|\1|g")                    # Value + Unit In one Block (e.g. "256M")
   #local lresult=$(du ${largs[*]} "${ltarget}" | sed -E "s|^([0-9]+)([a-zA-Z]*?)\s*?(.*)$|\1 \2|g")             # Value + Unit with Space Between them (e.g. "256 M")
   local lresult=$(du ${largs[*]} "${ltarget}" | sed -E "s|^([0-9]+)(\.?)([0-9]*?)([a-zA-Z]*?)\s*?(.*)$|\1\2\3 \4|g")  # Support for e.g. "5.6G" Integer Value . Decimal Value + Unit with Space Between them (e.g. "256 M")

   # Return Result
   echo $lresult
}

# Get Uncompressed Size
uncompressedOptionsRaw=("--summarize" "--apparent-size")
uncompressedOptionsHuman=("--summarize" "--apparent-size" "--human-readable")
uncompressedSizeRaw=$(get_size "${target}" uncompressedOptionsRaw)
uncompressedSizeHuman=$(get_size "${target}" uncompressedOptionsHuman)

# Get Compressed Size
compressedOptionsRaw=("--summarize")
compressedOptionsHuman=("--summarize" "--human-readable")
compressedSizeRaw=$(get_size "${target}" compressedOptionsRaw)
compressedSizeHuman=$(get_size "${target}" compressedOptionsHuman)

# Compute Compression Ratio
compressedRatioFormatted=$(awk -v u=${uncompressedSizeRaw} -v c=${compressedSizeRaw} 'BEGIN{printf("%.1f\n", c/u*100.0)}')

# Echo
echo "Uncompressed Size: ${uncompressedSizeHuman}B (${uncompressedSizeRaw} B)"
echo "Compressed Size: ${compressedSizeHuman}B (${compressedSizeRaw} B)"
echo "Compression Ratio: ${compressedRatioFormatted} %"

After approx. 12 hours of data Logging from a SINGLE JK BMS I get, for TimescaleDB-HA 16.2 with PostgreSQL 16.2 + PostGIS + TimescaleDB Toolkit under the Hood (which should be very high performance):
Bash:
root@TESTING:/tools# ./getSizeCompressionData.sh "/home/podman/containers/data/homeassistant-timescaledb/"
Analyzing Folder /home/podman/containers/data/homeassistant-timescaledb/
Uncompressed Size: 593 MB (606562 B)
Compressed Size: 250 MB (255859 B)
Compression Ratio: 42.2 %

My Production HomeAssistant Instance running in the Garage and Handling Solar (currently logging 2 x JK BMS + 3 x Inverters + 1 x Meteo Station) still running only the "default" SQLite3 Database home-assistant_v2.db but also on ZFS with LZ4 Compression:
Bash:
root@SOLAR:/tools# ./getSizeCompressionData.sh "/home/podman/data/homeassistant/"
Analyzing Folder /home/podman/data/homeassistant01/
Uncompressed Size: 5.7 GB (5894054 B)
Compressed Size: 3.2 GB (3293903 B)
Compression Ratio: 55.9 %

So I assume that the big File Size (approx. 0.5 GB / 12h / 1 BMS = 1GB / day / BMS) is due to the Update Frequency of the JK BMS (every 1 second, so that the controller for Automatic Charge Voltage / Automatic Charge Current can work Correctly).

But 1GB / BMS / day is A LOT of data without much use IMHO. I am also looking into using HomeAssistant "LTSS" addon to replace the Recorder Section (overall that will make HomeAssistant use 2 databases: 1 for LTSS [Long Term Data Storage] where ALL Data will be periodically dumped to and 1 for Normal Operation [only Store Data for a few Days])].

It seems like Home Assistant Recorder Purges Data older than 10 days by Default ...

But ... if the Value does NOT Change, I do not see why that value should be re-inserted into the Database.

I didn't see any "Global" HomeAssistant Configuration for the "recorder:" Section that can limit the Sample Rate / Frequency of Sensors. I saw some Posts suggesting the Definition of "Virtual" Sensors.

One Option is maybe to Specify a "delta" Filter for each JK BMS Sensor based on:

E.g. for a Cell Voltage specify Maybe "delta: 0.002" [V] so that any "Measurement Noise / Tolerances" of 2mV or so will be Skipped.
Or do the RAW Sensor Value as "Internal" and only Publish the Filtered Sensor Value to API/MQTT ?

Any other Ideas ?


EDIT 1: on second thought, maybe "delta" is not very good. If every "step" is less than 0.002 V (as in my example) it will NEVER log anything :( . Worst it will NEVER use that for the actual controller.

EDIT 2:
Maybe we therefore need to split EACH sensor into:
- ${sensorName} (as it is now but with internal: true)
- ${sensorName}_publish (as it is now but with internal: false and throttle: ${maximumUpdateFrequency}s

Any other ideas ?

EDIT 3:
*In theory* for a **text file** ZFS should be able to handle the compression of similar Data quite well.

But Home Assistant SQLite3 Database is **not** a text file, but it's more like a binary File. So I'm not so sure how effective ZFS Compression is in this case (I'm pretty sure that it also depends on Block Size, Ashift Value). After all ZFS Compression is NOT the same as ZFS De-Duplication Feature (the latter being VERY Memory Intensive and therefore VERY rarely used).

TimescaleDB .. Not Sure.

This is what can be found on the Internet quite Easily regarding PostgreSQL Tuning on ZFS:

And this very extensive Analysis which, among other stuff, suggests using recordsize=8K or recordsize=16K:

Apparently I'm using the "Defaults" of recordsize=128K :( .

The "official" OpenZFS Documentation suggests recordsize=32K/64K/128K:
 
Last edited:
The default behavior of the api: and wifi: components is a reboot every 15 minutes in the event of loss or no connection.

If you don't use Home Assistant the api: line must be configured like this to avoid reboots.

YAML:
api:
  reboot_timeout: 0s

Same for wifi:

YAML:
wifi:
  reboot_timeout: 0s
I just (by accident) "tested" this.

The ESP32 Rebooted within 15 seconds of losing connection to WiFi / API / MQTT. Not 15 minutes ...
 
Last edited:
The documentation says 15min. Could be another reason?
Uhm difficult to tell yesterday was a bit of a mess with something like 20 trips.

First checked network connectivity that is also sketchy inside the garage apparently.

And of course when I put back the Esp32 which I had to flash using usb, I connected it to the inverter but the USB cable that I used was the wrong one. Not connected to the AC (just a USB cable not connected to anything) 🙄. I tidied up quite a bit to avoid it happening again.

But that also makes no sense...BMS_Err-Stop is ENABLED. Why it attempted to start, then trip immediately, if the ESP32 was clearly not working?

I got lots of BMS_Communication_Err, F41_Parallel_Error and some F18_AC_Overcurrent. Even without any load, just the 3 inverters together... Once I got overcurrent without load and only one inverter comected. It was driving me crazy.

After connecting usb power it worked almost flawlessly. Weird that it attempted to start-stop-start etc though. What could that be?

At first I though the USB cable for the Esp32 was connected to the inverter output (off grid, hence a trip of the inverter would cause a trip of Esp32 and vice versa). But it wasn't connected at all...

I'm particularly worried about the AC overcurrent fault... I swear I heard a sound of a spark coming from one of the inverters when that message popped upthe first time 😭. There should be some overvoltage protection in place in that output filter, even in an event of an IGBT/MOSFET turn off 🙄.

And during those periods for some reason, even with bms enabled, one lifepo4 cell hit 3.69 V with a few spikes (not ultra long, but I am worried that it could habe been damaged).

EDIT 1: that's the reason I was afraid to raise OVP to 3.65 V @MrPablo. There doesn't seem to be so great headroom in case BMS stops talking to Inverter (and there is always a bit of delay for the inverter to detect the "Heartbeat" on the CANbus.
1713602555725.png

EDIT 2: And of course at the end of the day (besides the USB Cable Issue) I also discovered that I never added the ESPHome to Home Assistant :rolleyes: . Stupid me 🫣. I thought I did but actually I didn't. Damn ....
 
Last edited:
There should be no limit for a multi-bms bms Master Slaves version. (still needs to be developed).

I ask the question because if you wish you could test the multi-bms version (2x JK-BMS BLE) on the same Atom S3.

The first tests could begin in a few days.
Been away - just got back from holiday....

are we saying multi-BMS yaml code is out?

and for which ESP Atom-Lite or the normal ESP 32 ?
 
Guys - Excuse my ignorance
wanting to flash V1.17.5 from V1.17.5 and I am just finding out a lot of stuff has been placed into variables (this has no manual on use case or how to deploy the codes) well I have managed to input many of the variables into my secret but lost on the below

where it was :
esp32:
board: esp32doit-devkit-v1
framework:
type: esp-idf

is now

esp32:
board: {{board}}
variant: {{variant}}
framework:
type: esp-idf


whilst I know there is a good explanation for this change - I thought this project is aimed to make it user-friendly to dummies who do not know coding, so as many as possible can rip the benefits of the hard work put in, and it's not just for geeks to use.

Anyways it is failing at that line, what am I missing?

1713996478314.png

Can we try and keep the process simplified to avoid chasing others away from this project to other projects...
 
@chaosnature

The multi-bms code (several BMS connected on the same ESP32 as with PVbrain2) will soon be ready to test. I have been very busy on another project in recent weeks and still have to code the multi-bms alarm system.

We haven't changed the part of the code you cite, it's defaulted to esp32doit-devkit-v1 and you need to change it for Atom S3 Lite see the readme and config file below.


This will be simpler in the new packaged version.
 
@chaosnature

The multi-bms code (several BMS connected on the same ESP32 as with PVbrain2) will soon be ready to test. I have been very busy on another project in recent weeks and still have to code the multi-bms alarm system.

We haven't changed the part of the code you cite, it's defaulted to esp32doit-devkit-v1 and you need to change it for Atom S3 Lite see the readme and config file below.


This will be simpler in the new packaged version.
So will this be 2 x ESP32 or batteries will be connected to just one MAster ESP?

Do you have a Diagram for connectivity mapping out the GPIO between them etc?

thanks...
 
'
Anyone know if its possible to store and how to store more than one variable in the secret.yaml

i would like to have all my MAC define in the secrets if so..

thanks
 
Last edited:
Guys - Excuse my ignorance
wanting to flash V1.17.5 from V1.17.5 and I am just finding out a lot of stuff has been placed into variables (this has no manual on use case or how to deploy the codes) well I have managed to input many of the variables into my secret but lost on the below

where it was :


is now




whilst I know there is a good explanation for this change - I thought this project is aimed to make it user-friendly to dummies who do not know coding, so as many as possible can rip the benefits of the hard work put in, and it's not just for geeks to use.

Anyways it is failing at that line, what am I missing?

View attachment 211393

Can we try and keep the process simplified to avoid chasing others away from this project to other projects...


I have documented the process in the below link:

How to Flash Atom S3 with EspHome - HA

I struggled with flashing Esp32 S3 Atom lite with the project I was actively involved for the Rs485/CAN battery Comm conversion unlike fasling Esp32 dev kit chip, its not as straightforward. but i finally found a way around it. 1. Make sure the GPIO is correct or else you will end up with a...
diysolarforum.com
diysolarforum.com

Hope it helps other HA EspHome users.
 
I have documented the process in the below link:

How to Flash Atom S3 with EspHome - HA

I struggled with flashing Esp32 S3 Atom lite with the project I was actively involved for the Rs485/CAN battery Comm conversion unlike fasling Esp32 dev kit chip, its not as straightforward. but i finally found a way around it. 1. Make sure the GPIO is correct or else you will end up with a...
diysolarforum.com
diysolarforum.com

Hope it helps other HA EspHome users.

Thank you for taking the time to document your procedure.
 
Anyone upgrading to core 2024.5.0 or 1 hold off
i think there is a bug
broke my overview and some ESPhome components

1714957349622.png

i am trying to revert back as we speak to 2020.4.4


# Edit - Multiple bug reports

1714959085366.png
 
Last edited:
@Sleeper85

small favour.

Can you write a code and update the yaml to include Screen-Display for the Esp32 S3 model please - when you get the chance?

1715092228154.png1715092287117.png


Would be nice to have a display next to the battery
 
Last edited:
and by the way, I have tested the project with :

# Models - 3kW Hybrid 5G Inverter
(Solis) SOL-3K-RHI-48ES and

# Sofar
HYD 3600-ES (G2)

......on both the Esp32 Atom S3 and Devkit

Atom speed is great - the only pitfall is the flashing process.


you can add it to your log...
 
@Sleeper85

small favour.

Can you write a code and update the yaml to include Screen-Display for the Esp32 S3 model please - when you get the chance?

View attachment 213826View attachment 213828


Would be nice to have a display next to the battery

Not having an Atom S3, I cannot develop or test anything but there is already code available and tested by @arzaman. This will be part of the new packaged version.

 
Ok, so my equipments and batteries have arrived.

I see multibms is being developed but still based on BLE / older BMSes.

Anyway we could implement this on JK Inverter BMSes variants?:

This is the most efficient way to monitor my JK Inverter BMSes so far, only 1 ESP32 for all my 7 packs.

I dont know any coding, but I guess this logic can be used to implement on this project.

I really need your CAN implementation. Builtin CAN of this inverter is fine, but it doesnt have the features I need from your project, especially the offset. I really hate the 0.5V offset of Deye inverters. BMS wont trigger float and never reach absorption stage.

I'm willing to pay.
 
Ok, so my equipments and batteries have arrived.

I see multibms is being developed but still based on BLE / older BMSes.

Anyway we could implement this on JK Inverter BMSes variants?:

This is the most efficient way to monitor my JK Inverter BMSes so far, only 1 ESP32 for all my 7 packs.

I dont know any coding, but I guess this logic can be used to implement on this project.

I really need your CAN implementation. Builtin CAN of this inverter is fine, but it doesnt have the features I need from your project, especially the offset. I really hate the 0.5V offset of Deye inverters. BMS wont trigger float and never reach absorption stage.

I'm willing to pay.

I'm currently working to finalize the multi-bms version with everything connected on a single ESP32.
This is mainly intended for the PVbrain2 card but this should therefore work with 2 or 3 BMS and a classic ESP32 (to be tested).


Thank you for the links, indeed if the project you indicated works well for you then we could use it. You can contact me privately to discuss it.
 
I'm currently working to finalize the multi-bms version with everything connected on a single ESP32.
This is mainly intended for the PVbrain2 card but this should therefore work with 2 or 3 BMS and a classic ESP32 (to be tested).


Thank you for the links, indeed if the project you indicated works well for you then we could use it. You can contact me privately to discuss it.
Sent you DM.

Yes, that project is serving my monitoring thru home assistant very well.

So maybe, datas collected using that project, can be used to implement your CAN project.

Just like the other contributor that uses MQTT instead, but this one Im suggesting is a little bit more efficient in my opinion.
 
Working on JK BMS B1A24S15P + Deye sun 6k sg03lp1 + DIY Battery 16S

Problem (attention test), the pinout of the wires in the connector is different than in the diagram


Created cards Home Assistant Lovelace UI cards for a beautiful dashboard here.
The problem lies in the values of some parameters.
Are they described in the project?

upd. values parameter binary_sensor.esp32_lfp_wire_jk_bms_can_balancing_switch
 

Attachments

  • 1a.jpg
    1a.jpg
    109.6 KB · Views: 9
Last edited:

diy solar

diy solar
Back
Top