EG4 18KPV - What not to do in parallel

[sunshine_eggo]

Never in my wildest dreams did I imagine such an issue could occur due to a wrong comms cable...I don't feel so bad now "huffs Victron Ethernet cable bag"

My experience with Victron communication issues EVEN WITH THE CORRECT CABLES:



There I was thinking I would be able to just plug in the second Quattro, configure them in split phase and just keep motoring.

I pooped my pants a little when the INSTANT I plugged in the correct cable to the correct port, the whole system shut down. Paraphrasing: The operating inverter sensed a foreign and unwelcome interloper on the VE.Bus, and it shut itself down placing it in the programming mode.

Then I configured them to skip along together happily in split phase, they fired right up.

While some might object to the inconvenience and lack of redundancy, if EITHER inverter faults and goes offline, the other shuts down.

Sounds like a 50 year old company has taken a far more strict and conservative approach.

 

[automatikdonn]

@automatikdonn can you verify no other wiring was changed besides the parallel cable with RJ45 ends?

Reading @Quattrohead 's post it seems AC wiring was/is also incorrect.

Just that cable. It's in the video.

 

[Brucey]

My experience with Victron communication issues EVEN WITH THE CORRECT CABLES:

View attachment 218727

There I was thinking I would be able to just plug in the second Quattro, configure them in split phase and just keep motoring.

I pooped my pants a little when the INSTANT I plugged in the correct cable to the correct port, the whole system shut down. Paraphrasing: The operating inverter sensed a foreign and unwelcome interloper on the VE.Bus, and it shut itself down placing it in the programming mode.

Then I configured them to skip along together happily in split phase, they fired right up.

While some might object to the inconvenience and lack of redundancy, if EITHER inverter faults and goes offline, the other shuts down.

Sounds like a 50 year old company has taken a far more strict and conservative approach.

Really it's the only way that makes sense. To me the only "downside" is the need for a ve.bus to USB adaptor to communicate in a language the Victron device understands for more than superficial initial configuration. Personally seems like $68 bucks well spent.

 

[ricardocello]

I pooped my pants a little when the INSTANT I plugged in the correct cable to the correct port, the whole system shut down. Paraphrasing: The operating inverter sensed a foreign and unwelcome interloper on the VE.Bus, and it shut itself down placing it in the programming mode.

You aren't kidding. Victron's default is to shut everything down now! Nor do they let you snoop around on the VE.Bus. They demand that everything is working or No Soup for You.

 

[HRTKD]

and they still have the sense to cover their asses by saying only use approved UTP cables...

When I setup my Victron communication cables, I checked to see about making my own. There were too many posts on the Victron Community about DIY cables causing problems. I've made hundreds of Ethernet cables and even a few coax cables for Token Ring back in the day. Yes, I could have made my own and I would have been OK. But the Victron cables aren't that expensive, so I figured why temp Murphy.

 

[marionw]

The big question is this.
Communications between various pieces of solar equipment require both a hardware and software protocol. The hardware protocol is generally RS232, RS232-TTL, RS485 or CAN. The software protocol is generally ASCII Text (Victron Smart Shunt as example), Modbus RTU or some form of HexAscii (Pylontech, Growatt etc).

How could the "Master" inverter ever communicate correct data to the "Slave" inverter using a crossover cable. Assuming the parallel RJ45 jacks on both inverters are wired the same and the CAN pin pair (usually pins 4&5) between the two RJ45 jacks are not connected due to a crossover cable being used, how the heck did the "Slave" ever receive valid data from the "Master". In the absence of "Master" communications how does the "Slave" react.

Are we missing something here?

 

[ksmithaz1]

Then how would it know to enable parallel and also how would it know the other inverter is faulted?

I'd bet it carries more than a can signal. One of the other inverters I used had a 15 pin HD-sub (VGA). My guess is the sync wave is on there in analog. Not inclined to tap it to check, but now that this occurred I may get a wild hair. If it was pure CAN I think it should have either just worked or just failed, but CAN is a relatively slow bus. I think keeping up with a 60Hz signal would require too much digital sampling.

I keep trying to work in my head the wave on the scope. Looks like the units went into some sort of test mode or something the scope is showing a 1.67Khz high frequency wave. That's gonna look like DC to an AC appliance.

 

[wheisenburg]

The way his system is wired, the damage occured only from the grid input side. I think you missed the other thread that went into this in detail.

So basically if you had some loads on a "Grid Panel" down stream from your Grid disconnect, the inverter back fed that panel with the incorrect voltage when the wrong connection cable was used. So anything connected there could be damaged. In my home the old "Main Panel" became the new "Critical Loads" panel. I have a new "Grid Panel". It just has nothing in it other than some surge suppressors, but I can move things there if I need to.

I get that an incorrect cable could cause the inverters to not generate the AC power correctly. I don't get why the inverter would stay connected to the grid when this happened.

 

[Brucey]

When I setup my Victron communication cables, I checked to see about making my own. There were too many posts on the Victron Community about DIY cables causing problems. I've made hundreds of Ethernet cables and even a few coax cables for Token Ring back in the day. Yes, I could have made my own and I would have been OK. But the Victron cables aren't that expensive, so I figured why temp Murphy.

You can also make your own vedirect cables but the Victron ones have galvanic isolation.

 

[Brucey]

The big question is this.
Communications between various pieces of solar equipment require both a hardware and software protocol. The hardware protocol is generally RS232, RS232-TTL, RS485 or CAN. The software protocol is generally ASCII Text (Victron Smart Shunt as example), Modbus RTU or some form of HexAscii (Pylontech, Growatt etc).

How could the "Master" inverter ever communicate correct data to the "Slave" inverter using a crossover cable. Assuming the parallel RJ45 jacks on both inverters are wired the same and the CAN pin pair (usually pins 4&5) between the two RJ45 jacks are not connected due to a crossover cable being used, how the heck did the "Slave" ever receive valid data from the "Master". In the absence of "Master" communications how does the "Slave" react.

Are we missing something here?

The ve.direct protocol is known:

VE.Direct Protocol FAQ [Victron Energy]

www.victronenergy.com

The ve.bus protocol used between linked Victron inverters is proprietary and I'm not sure if there's ever been a breakdown of it.

 

[42OhmsPA]

Just to keep beating this thread to death, the crossover cable caused the 18kpv device(s) to behave in a manner that no one could predict.

What we haven't been told (and probably won't be told), is why the device is designed to do that at all, and what the pinout on those connectors are.

My far from an expert guess is it has something to do with the "parents" of this inverter being a single phase ~230V architecture capable of operating in parallel and 3 phase.

Thought this was nice in the EU12K manual.

 

[cs1234]

Just to keep beating this thread to death, the crossover cable caused the 18kpv device(s) to behave in a manner that no one could predict.

What we haven't been told (and probably won't be told), is why the device is designed to do that at all, and what the pinout on those connectors are.

It's an advanced feature, future proofing for when the US grid moves to a much higher voltage. We weren't supposed to discover the hidden pinout until the announcement.

 

[ksmithaz1]

I get that an incorrect cable could cause the inverters to not generate the AC power correctly. I don't get why the inverter would stay connected to the grid when this happened.

I've written too much of this crap. My programming gut is telling me the cable created a race condition somewhere in the firmware getting extremely conficting information once the grid dropped out, preventing the logic flow from progressing to the "Turn off the relay" state. Some dependent event that makes the relay shut down never occurs. Since this can be trivially duplicated, has likely been escalated at a pace never before seen, I'd guess this is already in the middle of testing in the lab.

This is a pretty big deal from a now that we know let's get this fixed, but in the grand scheme electricity is dangerous and when people make seemingly minor mistakes they can amplify. 240v can kill you just as easily as 400v.

 

[42OhmsPA]

It's an advanced feature, future proofing for when the US grid moves to a much higher voltage. We weren't supposed to discover the hidden pinout until the announcement.

A shocking easter egg

 

[automatikdonn]

So basically if you had some loads on a "Grid Panel" down stream from your Grid disconnect, the inverter back fed that panel with the incorrect voltage when the wrong connection cable was used. So anything connected there could be damaged. In my home the old "Main Panel" became the new "Critical Loads" panel. I have a new "Grid Panel". It just has nothing in it other than some surge suppressors, but I can move things there if I need to.

I get that an incorrect cable could cause the inverters to not generate the AC power correctly. I don't get why the inverter would stay connected to the grid when this happened.

I think a grid feedback loop was occuring one was supplying the other. They were "unintentionally islanding"

Still doesn't explain the massive voltage spike though.

 

[wheisenburg]

But surely they aren't passing AC voltage across a standard ethernet cable that might not even be in conduit or anything.

Not 120 volts AC. It could be more like a 5-12 volt AC signal. It might also just be a timing signal. Something like a pulse every time the wave crosses from minus to plus for each leg. In any case remember that these standards for how to synchronize inverters are old. So is the inverter communication protocol. When was the last time you heard the term baud rate? I'm not sure a digital communications protocol at 9600 baud would work very well for synchronizing inverters. If it were a digital protocol it would likely either work or not work. The fact that it goes haywire likely means that analog signal voltages are being crossed causing the hardware to work incorrectly.

 

[ricardocello]

One of the other inverters I used had a 15 pin HD-sub (VGA

Did you plug a VGA monitor into it to see the video?

 

[automatikdonn]

I've written too much of this crap. My programming gut is telling me the cable created a race condition somewhere in the firmware getting extremely conficting information once the grid dropped out, preventing the logic flow from progressing to the "Turn off the relay" state. Some dependent event that makes the relay shut down never occurs. Since this can be trivially duplicated, has likely been escalated at a pace never before seen, I'd guess this is already in the middle of testing in the lab.

This is a pretty big deal from a now that we know let's get this fixed, but in the grand scheme electricity is dangerous and when people make seemingly minor mistakes they can amplify. 240v can kill you just as easily as 400v.

How is the grid sense dependent on parallel communication? It's an honest question. I don't write code for inverters and the only real time systems I touch use motors.

 

[ricardocello]

I keep trying to work in my head the wave on the scope. Looks like the units went into some sort of test mode or something the scope is showing a 1.67Khz high frequency wave. That's gonna look like DC to an AC appliance

Good eye on the graticule. Honestly, when I saw that waveform, I pictured the poor homeowner’s VFD freaking out and dying a painful death.

 

[ksmithaz1]

Pretty much everything these days is micro-code. Some thread in the event loop has a trigger that says the grid down verify X (sanity check) do Y then drop the relay, should happen in micro-seconds. The verification creates an exception for an out-of-bounds condition (divide by zero whatever) which interrupts Y, generates another exception and restarts the event loop. The event loop see's the grid is down ...

Race conditions occur all over the place in software these days. Most low level stuff like this has auto-restart and watchdog functions to prevent a race but it usually takes time to find all the edge cases. There is probably a small micro-controller with some dedicated programmable ASICS handling all this. I don't do inverter design either, but the concepts don't really change much.

Juniper switches had a bug in the BSD kernel that would make the kernel lock up and stop responding (junos 18.x) after an undetermined amount of time. All the layer 3 would go down, the only way to reset it was with power. It would however continue to function as a layer 2 device as programmed, vlans, switching all working as expected, you just couldn't change anything at all or reboot it.

Welcome to the modern world of programmable everything.

 

[ksmithaz1]

I might mention ASIC's often have bugs and errata that are handled by exceptions on the management plane.

 

[Quattrohead]

Just that cable. It's in the video.

That is not what I saw in the pictures on the original thread, I'm a little perturbed that the original thread got deleted. Would you mind posting those pictures back up again of the connections and the scope waveform.

 

[marionw]

Can an OP delete an entire thread that others have posted to or does it require an administrator to delete the entire thread?

 

[wheisenburg]

preventing the logic flow from progressing to the "Turn off the relay" state

Sounds like a possibility. Unfortunately, software is very unpredictable when it encounters situations where it was never tested. I'm a software guy too, but more on the IT side.

 

[Will Prowse]

Can an OP delete an entire thread that others have posted to or does it require an administrator to delete the entire thread?

Requires administrator. Op can only edit their original post.