An Enphase Ensemble Installation

GXMnow

Photon Sorcerer
Joined
Jul 17, 2020
Messages
1,428
How hard would it be to swap that panel with one near it that is making proper power? Just see if that inverter comes back up and the inverter you took the panel from then goes to crap. Then you will know if it is the panel or the inverter. With the thermal image of the panel, it looks like it may be damaged. It is odd how it has a line dividing the hot side from a not so hot side. Is that heat from the panel, or is the inverter under it getting crazy hot?
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
... odd how it has a line dividing the hot side from a not so hot side....
The panel's overall temperature is fairly uniform, what you're seeing is the affect of distance;
items farther away appear cooler. The photo to the right is looking east, the previous was
looking west. I typically snap a few shots from different angles to double check that.

I think the panel is just hotter because the other panels are converting solar energy into
electricity and that one isn't, so all it can do is radiate. The right most photo is a pic from
last month. The affected panel is the left-most in the middle row.
1600341008103.png

Currently I'm under contract with the installer so I'm just going to sit back in the air conditioning ;) - if they do come out I'll watch and learn what they do. Considering the envoy reported it was okay I'm hopeful it'll work okay today thinking it was under-producing at the end of the day as it was hotter than the rest.

Here's a link to what a string outage and microfactures look like. I think you're right about being able to see the microinverters on some of the panels underneath. I'll have to take some shots next time I'm up there above and below to see what the temps are to confirm that.

Update: The panel is still behaving badly today. I'll keep you all posted as to what the deal is.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
The panel was running okay yesterday afternoon, looks like Enphase pushed a firmware update…I’ll keep watch.

Here’s an event chronology (skipping dups and secondary effects (e.g., DC to low during a reset)):

Sat Sep 19
11:13 AM EDT Starting download of parm0, part number 540-00142-r01-v04.20.03​
11:16 AM EDT Completed download of parm0, part number 540-00142-r01-v04.20.03​
<…microinverter reboot messages…>
11:20 AM EDT Commanded Reset​
11:23 AM EDT High Skip Rate <… not sure what this is, sounds like microinverter-grid sync issue?>
11:36 AM EDT Starting download of procload, part number 520-00082-r01-v04.20.03​
<…11:40 last time external software recorded 0 watts from one of the panels…>
11:41 AM EDT Completed download of procload, part number 520-00082-r01-v04.20.03​
11:44 AM EDT Commanded Reset​
11:48 AM EDT Grid Instability: Set <occurs on panel with problem>
11:51 AM EDT Grid Instability: Clear​
04:06 PM EDT Grid Instability: Set <occurs on different panel>​
04:11 PM EDT Grid Instability: Clear​

The system Envoy S/W: D6.0.86 (c77974), 9/3/20 build

This thread suggests the High Skip Rate might be heat related. This thread suggests it has to do with the microinverter synchronizing to the grid and associated with large power drops...sounds like the problem... but this event didn't occur on the problematic panel. Interesting... the panels are just starting to wake up and I'm seeing a bunch of Skip messages with this software load that weren't in yesterday's information.

The Grid Instability seems odd. If the actual grid were unstable I'd expect all the microinverters to send messages. I saw a redit thread where it might be a wiring issue, doesn't seem to be consistent with a single panel though.

The monitoring software currently ignores events, but I'm wondering if it wouldn't make sense to send alerts when a pattern develops. Possibly as I get smarter about what patterns to look for it'll become more evident.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Here's the rest API to get events with the installer login:


Of course, it must be converted to url acceptable-speak (e.g., spaces to %20). So, for example, to search for the first 2 entries from Sep 19th, it's:


  • start - record number to start at where 0 is the first entry
  • length - number of records to return
  • search- optional
    • value - SearchValue is a string to search for, can be a device serial number, date, or event message.
    • regex - boolean if the search string contains regex (although even simple regex didn't seem to work)
The results should look something like:
JSON:
{
    "iTotalRecords": 500,
    "recordsTotal": 500,
    "iTotalDisplayRecords": 379,
    "recordsFiltered": 379,
    "aaData": [
        [
            "51941",
            "Microinverter failed to report: Set",
            "12345",
            "pcu ",
            "Sat Sep 19, 2020 07:24 PM EDT"
        ],...
    ]
}

There's a undescore parameter which takes the epoch date as a long, but it doesn't seem to have any effect. The full set of parameters I've seen for it are below and in a spoiler so as to not make this post any longer than it already is. The rest of the information seems to be GUI information of some sort indicating which columns are sortable and which fields are searchable; possibly the regex only applies to specific fields.

/datatab/event_dt.rb?locale=en&draw=1&columns[0][data]=0&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=false&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=1&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=false&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=2&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=false&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=3&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=false&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=4&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=false&columns[4][search][value]=&columns[4][search][regex]=false&start=0&length=72&search[value]=&search[regex]=false&_=1600606781051

I've updated post #17
 
Last edited:

GXMnow

Photon Sorcerer
Joined
Jul 17, 2020
Messages
1,428
When I try this I just get

{"iTotalRecords": 500, "recordsTotal": 500, "iTotalDisplayRecords": 500, "recordsFiltered": 500, "aaData": [ ["67041","DC Power Too Low: Clear","121917064031","pcu ","Sun Sep 13, 2020 07:31 AM PDT"]] }

I should get more entries.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
You can get all the events on a specific device by using its serial number as the search string.

...I should get more entries....

What did you use for your query? If you copy/pasted the query in #161 and updated the date there may have only been one "Sep 19" record left in your database":
GET http://envoy.local/datatab/event_dt.rb?locale=en&start=0&length=2&search[value]=Sep 19&search[regex]=false

Be careful not to have any spaces in-between entries (convert spaces to %20), same for brackets, back-slashes, etc.

Try this: GET http://envoy.local/datatab/event_dt.rb?locale=en&start=0&length=50

It should retrieve the last 50 records. To get the next 50 you just add +50 to the start. So, the next 50 entries would be:

In your browser you should be able to click any of the links in this post... they work on chrome anyway. First one should return two entries (if there are any from yesterday), next one should return the last 50, last one should return 50 more.
 
Last edited:

GXMnow

Photon Sorcerer
Joined
Jul 17, 2020
Messages
1,428
I got back to last Friday so far.
On Friday evening at 6:02 PM I start seeing "Power Too Low: Set" 3 of the panels cleared and set again and by 6:31 PM it looks like all 16 panels went into "Power Too Low: Set"

But this is where it get's odd. 10 panels then popped up "DC Voltage Too Low: Set" followed by "High Skip Rate" The other 6 panels did not show those message. But they do not match up with the ones that locked up. 3 of those are ones that stayed working and 2 are ones I got back up by covering the panel. 5 of them are ones that locked out and didn't come back until the next morning sun up. Covering may have worked, but I did not climb up to those.

On Sat morning, they all cleared the fail to report by 6:45 AM, and all 16 show a "Power On Reset" but only 10 came up with a "DC Voltage Too Low: Clear" message. And of course, they were th same ones that had the DC volts too low message the night before, so that part makes sense, but what didn't the other 6 show that?

So then on moved on to Sat night. at 6:50 PM they started showing the "DC Voltage Too Low: Set" and "High Skip Rate" messages again, but this time all panels did it. I was hoping to see a pattern and see if maybe some were running different firmware, but that does not seem to be the case.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Set/clear is SNMP trap protocol, system does a "set" when the condition occurs and a "clear" when it goes away. Ideally for every "set" there's a "clear", otherwise it thinks there's some left-over issue.

I think those events are normal at dawn/dusk...it's just saying the sun is setting and the voltage is falling (Power Too low...) and then that the power's low so it can't sync up to the grid (High skip rate). You should see the same thing in the morning as the panels "wake up".
 

GXMnow

Photon Sorcerer
Joined
Jul 17, 2020
Messages
1,428
I just found it odd that some panels did the "High Skip Rate" while others didn't
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Rainy day, got about 20% of normal power collection and it looks like all the panels are working correctly.

Interestingly, in the events I had a few of these event-cycles:

1:40 PM "Grid Instability: Set"​
1:40 PM "No Grid Profile: Set"​
1:45 PM "Grid Instability: Clear"​
1:45 PM "No Grid Profile: Clear"​

I've heard of people having problems due to the grid profile...I wonder if their fix for instability is to reset the grid profile?
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Interesting set of unusual events last night:

One of the Encharge units (194) reports 5x set/clears of EnsembleGhzCommsSignalLow in the space of an hour. If it was the router they'd all report... so probably just one having an issue?

Then at 11:16 PM five of the IQ8s reports PLC Communication Established with E3 and one of them (196) has some problems:
11:16 PM "AC Voltage Out Of Range"​
11:16 PM "AC Frequency Out Of Range"​
11:16 PM "Power Generation turned off by command"​
11:16 PM "boot_reason_fw_unexpected_reset"​
11:16 PM "under-frequency"​

Finally, one of the other IQ8s (494) also reports an under-frequency. I don't monitor grid frequency or voltage, but I'm guessing the power company was doing some maintenance but only one of the IQ8s was close enough to the tolerance to have been tripped by it.

Then the same Encharge as before reports 7x more set/clears on EnsembleGhzCommsSignalLow over 5 hours. When it's daylight I'll check for ants or a loose wire (it was breezy last night). Doubt I see anything, as I recall those devices are all in the NEMA enclosure.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Only found one spider husk, as suspected everything looked okay -- but only because it wasn't easily accessible.

Came back in and the monitoring software saw two conditions on the IQ7 that had been malfunctioning earlier (see #151), one of them looks pretty scary - never seen it before:

548: envoy.cond_flags.pcu_chan.gridinstability​
548: envoy.cond_flags.pcu_ctrl.bricked

But the Envoy shows it's still producing power from a report 3 minutes old .... so okay or confused? The current device status is "envoy.global.ok", so at the very least I have a different definition of "bricked" than Enphase does. Let's see we can find it in the Envoy javascript. A lot of times they're useless (e.g. HighSkipRate was "High Skip Rate")... oohhh a winner!

bricked:"No Grid Profile" ... guess we know what happened there.​

Yup... the Event log shows that IQ7 getting a grid instability and a reset to the grid profile. The monitoring software must have have hit at the same instant.
 

GXMnow

Photon Sorcerer
Joined
Jul 17, 2020
Messages
1,428
That is a bit odd that it would only be one inverter. To me that sounds like it might be a hardware issues that they are trying to work around with a software patch. OR.. I wonder if they are actually trying different versions on some of the inverters in a larger system for real world field stability testing? Maybe being a beta tester is not good enough. They are using your system to alpha test a new version. I still have not heard back on what they found with mine. I do want to do another power fail test son, but I don't want to lock up inverters again. My Enphase stuff has all been rock solid except for when I did that AC couple test. I do not see any easy way to get the firmware version on the normal end user pages. Is there a simple command to read out the version data?
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Yesterday when I putzing with the Encharges I noticed the top screws aren't screwed in. Seems this little screw hole has to be aligned properly for that to happen....it slides left and right and has a set screw. So how the heck do you align that when the lid's on? Hmmmm....
1600955499285.png

Found a few links on the Enphase site that sounded interesting too:
 

GXMnow

Photon Sorcerer
Joined
Jul 17, 2020
Messages
1,428
I was poking around a bit in the Enphase Installer Toolkit app on my iPhone.
Once I connect to my Envoy, click on "micros" and you see all the serial numbers. Click on the down arrow at eah one and it gives more info. The last entry is "running image" All 16 of mine show "520-000820r01-v04.18.02" and Updated "12/04/2019 3:43 PM"
So that must be the firmware version, and mine were all remotely updated in December, almost 5 months after they were installed.

What do your's show for firmware version? Are any of them different? How recent have they been updated?
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
At 11:25 & 11:58 a number (not all) of the inverters logged Grid instability messages .... but it's low-light raining/pouring today.

...All 16 of mine show "520-000820r01-v04.18.02" and Updated "12/04/2019 3:43 PM"
....What do your's show for firmware version? Are any of them different? How recent have they been updated?
You can get that via the API with: http://envoy.local/inventory.json, although you have to convert the longs to epoch time
Mine are all the same at 520-00082-r01-v04.20.03 and the img_load_date is ~6 days ago. There's also a created date, but it looks more like the original install date. From post #160 the images were loaded starting 11:13 of the 9/19, so that jives to 6 days.

Here's a simplified entry:
Java:
            {
                "part_num": "800-00625-r02",
                "installed": "1579280590",      //  Friday, January 17, 2020 12:03:10 PM
                "last_rpt_date": "1600967242",  //  Thursday, September 24, 2020 1:07:22 PM
                "created_date": "1579280590",   //  Friday, January 17, 2020 12:03:10 PM
                "img_load_date": "1600439448",  //  Friday, September 18, 2020 10:30:48 AM
                "img_pnum_running": "520-00082-r01-v04.20.03",
            }
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Had a slew of 162 unfamiliar events around 4:30 today... mainly the encharge and enpower having communication failures and various related events. Finally realized they were being rebooted somehow and guessed it must be a firmware update to the Encharge and Enpower... and sure enough both have image dates from today and the new version numbers are:
Encharge: 1.2.2627_release/20.34
Enpower: 1.2.1833_release/20.34"

Here's a list of the event types in no particular order:
  • EnchargeSoCMaintenance
  • EnsembleDevicePingReceived
  • EnchargeAGFPropagationDone
  • EnsembleModGone
  • EnchargeZigbeeCommFailure
  • EnchargeGridModeFromEnvoyMMOnGrid
  • EnchargeGridModeFromEnvoyUnknown
  • Device is excluded from the aggregate SOC
  • SOC below reserved level
  • EnsembleCfgPropagationFailure
  • EnpowerZigbeeCommFailure

No idea what changed... but makes me wonder if the encharge and Enpower don't have their own directly accessible APIs.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
51 unexpected events in this morning's log. Looks like some sort of maintenance starting at 2:40, possibly what was added to the new rev. According to Enlighten, the grid didn't go out. There may have been some battery use around 3:30ish (hard to be sure, wasn't much if it did).

Here's the events in sequence stripped to just one of each event:
2:40 Islanding​
2:40 EnchargeIslanding​
2:41 EnchargeSoCMaintenance​
2:41 Multi Mode On-Grid​
2:41 Grid-Tied​
2:41 EnchargePCUFault​
2:41 EnchargePCUStateGridFault​
2:41 EnchargeOnGrid​
2:41 EnchargePCUStateThrottled​
2:41 EnchargePCUFullyOperational​
3:40 EnchargeSoCMaintenance​
6:23 Modem Disconnect Set: Ping/Data​
6:23 Modem Disconnect Clear:Modem connection is OK​
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,932
Location
Key Largo
Nine days past equinox.... practically a boring morning, only oddity is the modem resetting a couple of times at 12:48 and 4:03. Seems to be part of the new normal after the last patch.

12:48 AM EDT Modem Disconnect Set: Ping/Data connection in the network is not working
12:48 AM EDT Modem Disconnect Clear: Modem connection is OK
 
Top