Diagnosing a Failure to diagnose a failure

[svetz]

Fair warning before you start, this is the highly comedic and entertaining section despite this being a true tale of woe & sadness. It's offered as a bit of puzzle so you can test your diagnostic skill and expertise while having a good laugh at my expense.

My sister-in-law calls in the morning, the power is out. We had horrific storms overnight, lots and lots of lightning. She complains about how the system has switched to backup like 30 times this month. Mine has switched once. To our credit, neither of us blame it on climate change.

I go over, slowly driving through all the flooded areas, and flip off all the breakers, including the pool pump. Wait a few minutes, then turn on the router. After a few minutes in the cell phone Enlighten app I can see the batty is at the shutdown 10% level.

As I watch the battery start to charge I explain to her the usual yada yada about the grid being out and draining the battery. It's annoying with Enphase as there's no real indicator you're on backup (they do send a text message alert and an email. I also tell her about the IQ Load Controller, which will automatically manage power-hungry devices (e.g., pump, AC) turning it on when there's excess solar and off if the battery starts getting too low.

But everything seems to be running in off-grid mode happy as a clam. Just need the grid to come back up.

As I'm leaving I glance at her meter and see the display is live. So, I go back upstairs and tell her the grid's been restored and we look at Enlighten. It's not switching to the grid. I go back downstairs and pull the ladder (yes, the house is on stilts) and climb up to check all the breakers. It's hot and humid, but there's a wicked 20 mph breeze gusting to 32, so it doesn't feel that bad. They all look good so I flip them on/off just to be certain. No Dice.

I call the power company and ask them to run a check, with the new meter they can test that the meter is seeing 240V, and the test comes back clean.

Have you figured it out yet?

So, I called Enphase and explain that we're off-grid, the solar is charging the battery and we have power, but that the system did not switch back to the grid. Two hours later they tell me to call the installer and give me the case number. They've rebooted the system, downloaded the latest firmware, and changed the profiles.
No dice.

The installer can't come out until Thursday. I calmly explain how THE FREAKIN POWER IS OUT and they tell me to try the manual ATS cutover hidden behind the peel label, behind the plastic cover.

I do that, it immediately flips back. He says to try it again. This time it sticks.

Now there's no power. The LEDs on the power meter are still at the same value they were from earlier.

The installer suggests switching back so we'll at least have battery power. I do that... and it looks like power comes back on.

But it's weird as some devices are powered and some aren't.

Naturally, the phase with the router is one that isn't coming up so I can't see what the enphase is doing easily (I'm feeling too lazy to use the toolkit). So I call the installer back and explain one of the phases isn't working from backup now.

They send a guy out.

Hindsight is such a wonderful thing, but at the time it was a mystery to me.

He pulls the covers off and starts frowning as he's measuring voltage. Each phase has 120V to neutral, but it's 0V across L1 & L2.

Making sense yet? If you're completely lost, that's about how I felt.

So, he works his way back to the meter and the meter box is wonky enough he can measure the input voltage, 240V between L1 & L2 and 120V to neutral for both phases.

He measures at the PV disconnect box, and again L1 & L2 are 120V to neutral, but 0 between L1 & L2. He goes back to try and measure the output at the meter and when he jiggles the box there's an angry zzzzt sound.

Now it's making sense. The whole house surge suppression behind the meter is probably toast - remember all that lightening? He says if the power company can get here quick enough he'll wait. Sure enough, they're five miles away and happy to come out.

But, with the grid off there's no power... so I ask if he can switch the batteries back on. For some reason, flipping the little switch behind the plate behind the peel puts the system into a manual mode, and while it was forced on grid the Envoy had no power. Without power, it couldn't reactivate the batteries. This makes no sense to me at the time, but I can see the Envoy's lights are out. In hindsight, it makes sense the batteries had to disconnect, it's the function of the ATS after all.

Didn't realize it at the time, but the Enpower had switched back to grid while I wasn't looking (duh, the envoy didn't have power and couldn't turn them on). But it was only using 1 phase, the panels were out because 0 volts between L1 and L2 (L1 was actually intermittent, which is why we got 120V on L1-to-neutral, stupid wind). Because the panels were out, lost most of the solar day with no battery charging.

The power utility guys arrive and pull the meter. Turns out it's not their whole house surge suppression device, it looks just peachy. It's the lugs, you know... the ones that are owned by the homeowner that the power company can't touch?

They can't leave the power on to the house in that state, it's too dangerous. But, If we can get an electrician out quickly they'll wait.

I explain that I understand, but if they could just crimp the lug and jam it back together for a few minutes then the Enphase installer could get 240V to restart the Envoy. That way I can keep the fridge going overnight with batteries in case I can't find an electrician at 6:00 PM. Of course, at the time it didn't occur to me the panels would have been off most of the day so there wasn't any power, since I switched the switch behind the cover behind the peel.

So, now my sister-in-law has her brother-in-law, the installer, the power company all meandering around, and an electrician on the way.

The power company guys do put the meter back in and the installer gets the envoy restarted. But, the house still has no power when switched back to the battery. The installer thinks it's because the batteries got discharged after flipping the secret switch behind the peel and the door. That's when I realize duh, the panels have been off all day because L1-L2 voltage was zero.

Turns out, hers isn't the only house off the transformer, so the power company has to break her lines to make it safe for the electrician. Big bucket trucks do their thing as the sun slowly sinks in the west.

What? You're still reading this? Oh, you want to know how it all ended? Well, I don't know for sure yet. I left before the electrician arrived...but after the installer left.

The real problem was probably always the lug which I should have picked up in the first couple of lines with how frequently the system was shifting to battery power. The big mistake I made was assuming right off the bat there was a problem with Enphase. Had I tested the voltage from the grid I could have saved myself most of the day.

So the moral of this story is, don't answer when your in-laws call.

 

[svetz]

This morning I connect remotely through the Enlighten web app.

I can see the house has consumed power from the grid overnight, so seems like the power company and electrician did their thing. But, all the other circles are empty and half the microinverters and all of the batteries report errors. The Enlighten app isn't sure if it's on grid or not. So around 8 A.M. I call Enphase. Like yesterday it's a little outrageous I have to wait on hold for less than 30 seconds to talk to an actual person that speaks English.

Sadly, the agent on the far end can't get into the Envoy remotely and he wants me to push the two buttons to make it phone out so they can get access.

They say it like it's no big deal. And if it were at ground level it probably wouldn't be. But the installers installed it about 12 feet up, and it is essentially on the edge of the pool, so you can't get the ladder directly below it. But at least if I fall doing this I'll probably hit the pool. Probably.
Really makes me wonder hard how the installer installed it there. It's almost as bad as the Enpower which is installed directly over the pool pump and heater, making it equally gymnastically challenging to access.

I'm not physically there though, so I explain that my sister-in-law said yesterday, don't come over before 9 under pain of no coffee. He says no problem, that there are some fates worse than death. He'll call me back at 9. He'll call me back on my cellphone number, they know my number. Does that mean that, like the FBI, they have a file on me? God I love Enphase. Stuff is always going to happen, but priceless to get support when you need it. I can't imagine how tough it must be to work support remotely.

This morning I'm taking my lightweight ladder with me, yes... didn't I already say her house is on stilts? She's got this freaking heavy 14' fiberglass A-Frame ladder which she stores attached to the bottom of her house. Yesterday I had to get that monster down and back up each time I thought I was done. On the upside, I won't have to exercise today, yesterday was plenty.

So, with the ladder, I'm about to drive back through the flooded roads to get to her house so I can push two buttons. But it's Tuesday, the sun is shining, the birds are singing, the wind is still blowing (13 mph gusting to 22 mph), and I have a dentist appointment at noon. What could possibly go wrong?

 

[Quattrohead]

What could possibly go wrong?

Nothing, you live in Florida !!!!! It saved my life I am sure, up north was killing me.
Yeah south Florida had some rough weather for a couple days, we seem to miss it around the Sebastian Inlet area as it always goes around us.
I noticed that Titusville and Port St Lucie tend to get way more storms than my area.

 

[Supervstech]

But it's Tuesday, the sun is shining, the birds are singing, the wind is still blowing (13 mph gusting to 22 mph), and I have a dentist appointment at noon. What could possibly go wrong?

Did you just admit to a dental trip, and ask what else vould go wrongbin the same sentance?

Brave, brave soul...

 

[svetz]

The good news was the flooding had receded by half. Unfortunately, Enphase can't see anything, nothing we try is getting them in. Finally, they suggest calling the installer.

I go upstairs as the harbinger of crappy news, and my sister-in-law asks, could it be because the network is down? Couldn't have mentioned that earlier? The router survived lightning, only 1 of two phases, but apparently, it couldn't survive the power being restored.

The networks are gone, but the lights on the router are blinking away happily even if the bits are going into a strange dimension. So, one factory reset and no one remembers the old network password. So, new network names and new passwords. The LAN is now up, but no WAN. One call to Xfinity and going through endless messages and we get to a human. They decide to roll a truck...tomorrow. Well, time to visit the dentist. So I get a break and drive back through the receding waters.

While at the dentist I wonder about the RG85 connection, I mean, what else am I going to be thinking about? It's a screw-on instead of crimp. I go back and she's taking her guests to the airport, so I'm on my own. Did I mention earlier that she had two guests staying with her through the storm and power outage? They were real troopers taking everything in stride. No karen's there.

Before tackling the router, I decide to reset the NetGear Extender, it's only purpose in life is to forward the wifi to the Envoy. The extender only has to go through one wall and over about 4 feet. But it only gets about 1 bar on the Envoy. That's some wall.

For the life of me, I can't figure out how to program it locally (the NetGear not the wall). It's got Wifi and an RJ-45 jack. So I call NetGear tech support. He explains to me that yes, it has wifi but it's not a part of the local network. I tell him he's wrong. He tells me I'm wrong. I ask, do I need the internet to print a document from my PC over over WiFi. He says yes, I tell him I do not. I'm trying to be a better person but I may have raised my voice a bit before the line went dead.

So, time to work on the router! I pulled the connector off, twisted it back on and it worked! So I call Xfinity and tell them they don't have to roll the truck. But, turns out I'm not authorized to cancel the truck because I'm not the owner (who is in a car driving to the airport). This is just after the NetGear call so I was still a wee bit irritated. I tell them I don't have to be authorized, they can just look at the damn router remotely to see if it's working and cancel the truck on their own. They can't do that. Fine. I text my sister-in-law and she deals with it. Truck canceled.

Then I go back to the NetGear extender and I figure out what he was saying, the fricking thing has an interface that collects the needed information. But, instead of saving it in the firmware, it sends the username, password, address, and secret questions back to Netgear HQ, in return HQ remotely programs the extender. I've never bought any NetGear, and I'm now positive I never will.

So, still freaking annoyed I stomp out to change the IP address on the Envoy. You all know the four pages of steps. But it occurs to me as I'm on the ladder dangling over the pool that they added a change to Enlighten cell phone app, all you have to do is to turn off your cellphone's data stream, then connect to the Envoy as the WiFi access point. Then you bring up Enlighten, it detects no internet, and gives you local access. All I have to do is click a few buttons. E Z P Z.

So, risking it all as the last two calls haven't been friendly (I'm trying to be a better person, honest), I call Enphase and give them my case ID and ask if they can see the system. They say yes. I give a cry of relief that I can finally go home having done the "good" brother-in-law thing. They then go on to tell me that there were errors on the system, but that they've all cleared and the system is working perfectly. I thank them and gush over how great they are, and really they are great.

I get home, and the wife has baked treats waiting for me. Not from a box, but from scratch. This practically brings tears to my eyes.

Did you just admit to a dental trip, and ask what else vould go wrongbin the same sentance?

Brave, brave soul...

So about that... it turns out I have a hole in my head, not the usual ones; but something new. They didn't think it was a cavity as it seemed to be hard tooth and no signs of decay. It's new, not on the old X-Rays. It's just a hole They speculated alien abduction or CIA torture and mind-wipe. I had no idea they were into conspiracies.

They X-rayed it every which way to Sunday, so I suspect any surviving bacteria will glow in the dark tonight.

[what could go wrong?]
Nothing, you live in Florida !!!!!

Damn right!

 

[Supervstech]

A Hole you say...
In a tooth...
Not a cavity?
So... a cyanide capture capsule?

 

[svetz]

So... a [rupturted] cyanide capture capsule?

Must have been while I was sleeping then, probably the snoring expelled the actual cyanide. Who knew that was a finely honed Darwinian survival mechanism?