Features for the Perfect Monitoring Software for a Residential System

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
What features should a residential monitoring system have?

Here's a stab at it, but let me know what you think:
  • How much power the panels are producing
  • How much power is being exported to the grid
  • The state of the battery
  • Duration of battery at current consumption
  • Time until the Battery is recharged
  • Alert if there are any problems with a panel (e.g., under producing for conditions - that is it's sunny but not getting the expected power)
  • API to the smart panel
Link to latest version source: https://diysolarforum.com/threads/f...ware-for-a-residential-system.3944/post-54880

Other projects on the forums

Other Resources
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Alert: Panels not getting enough power

So, how can a monitoring program know if your panels are generating the right amount of power? Thinking out loud here as I google stuff, let me know your thoughts.

Using the Panels
There are multiple panels on the roof, so any panel in the same orientation getting significantly less power than the others probably has a problem.
But, if all the panels are soiled equally, you wouldn't know they're soiled.

PV Calculation & local weather
Knowing the date and panel location and orientation you can calculate the watts from the sun they get. But, this doesn't account for haze/fog/clouds. If you live in SouthWest Texas, @ghostwriter66 confirms you don't need to worry about this. ;-)

WS-2902A & Weather forecasts
In @jasonhc73's post How much "Sun" do you actually get? he
talks about a weather station that reports solar radiation. So you could get that
information from the device but you'd need to keep it's sensor from getting soiled.
You don't really need to buy one as folks like Jason publish their data so a program
could get it, but that depends on someone in the neighbor having one, sharing
their data, and the owners keeping the device tuned. Similarly, you could just
check weather.com for the local conditions (e.g., sunny, partly sunny
icons). The UV index is also available locally in most places hourly and it
probably can be related to the solar radiation.
Capture.PNG Capture.PNG

Other possibly crazier ideas...
SMARTS is an NREL program to predict the visibility. There's also a NASA model, but it didn't work for my location. You could get the color of the pixel over your house from satellite data. NOAH broadcasts weather information over RF. There's an astronomer's forecast for sky conditions too.

What will I use?
Depends on what ideas you might have, but right now I'm leaning towards a panel comparison for panel problems and for soiling modifying a calculated expected power output against some easily available and reliable web data (like the UV index) to validate the power outputs.
 
Last edited:

jasonhc73

Cat herder, and dog toy tosser.
Joined
Oct 1, 2019
Messages
1,640
Location
Wichita, Kansas
Alert: Panels not getting enough power

So, how can a monitoring program know if your panels are generating the right amount of power? Thinking out loud here as I google stuff, let me know your thoughts.

Using the Panels
There are multiple panels on the roof, so any panel in the same orientation getting significantly less power than the others probably has a problem.
But, if all the panels are soiled equally, you wouldn't know they're soiled.

PV Calculation & local weather
Knowing the date and panel location and orientation you can calculate the watts from the sun they get. But, this doesn't account for haze/fog/clouds. If you live in SouthWest Texas, @ghostwriter66 confirms you don't need to worry about this. ;-)

WS-2902A & Weather forecasts
In @jasonhc73's post How much "Sun" do you actually get? he
talks about a weather station that reports solar radiation. So you could get that
information from the device but you'd need to keep it's sensor from getting soiled.
You don't really need to buy one as folks like Jason publish their data so a program
could get it, but that depends on someone in the neighbor having one, sharing
their data, and the owners keeping the device tuned. Similarly, you could just
check weather.com for the local conditions (e.g., sunny, partly sunny
icons). The UV index is also available locally in most places hourly and it
probably can be related to the solar radiation.
View attachment 6555 View attachment 6559

Other possibly crazier ideas...
SMARTS is an NREL program to predict the visibility. There's also a NASA model, but it didn't work for my location. You could get the color of the pixel over your house from satellite data. NOAH broadcasts weather information over RF. There's an astronomer's forecast for sky conditions too.

What will I use?
Depends on what ideas you might have, but right now I'm leaning towards a panel comparison for panel problems and for soiling modifying a calculated expected power output against some easily available and reliable web data (like the UV index) to validate the power outputs.

You are likely within 5 maybe 10 miles of someone with a "personal weather station", in USA or EU.
Weathercloud.net
pwsweather.com
wunderground.com/dashboard/pws

These services all receive the data from my weather station.

Solar radiation is pretty much the same for the region. It's pretty safe that the sun is shining the same everywhere in the near 5 to 10 miles, or even 200-mile radius. I am only 1 mile from my nearest PWS, that has a Solar Radiation meter on it!
Near me
https://www.wunderground.com/dashboard/pws/KKSWICHI80
Mine


The primary reason I have a weather station to see the wind at my house.
Wind may be a viable option for residential generation.
One must understand your wind data from the airport is under very different conditions than that at your house. There are very few expensive weather stations available if you only are interested in finding the wind data at the site to potentially install a wind turbine.

Open the two links to the PWS on wunderground, notice the Solar Radiation is close enough to be the same, but the winds are quite different.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Wind varies greatly on both your elevation and the surrounding topography, a weather station is a great idea to see if it's worth it (if you're on a flat plain (e.g., Kansas ;-) I've heard SAM can be pretty good at predicting it too. Many people get pretty good results with a tower as surface wind isn't good, but just 90' up is plenty.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
...You are likely within 5 maybe 10 miles of someone with a "personal weather station", in USA or EU.
You could probably read a few local stations and take the average, possibly send an email to the lowest asking them to clean their sensor. 😜
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Not perfect by any means, but attached is the start of a java program that uses rtlamr to read an itron RF netmeter and makes a local call to the envoy to get the total watts produced via http://envoy.local/production.json. The attached program isn't meant for anyone to run to get useful data (it needs rtlamr/rtl_tcp in a lib subdirectory and to have the proper meter Endpoint IDs). But it is meant as a simple example of calling PowerShell and make a restful API call (although I haven't yet bothered with a json parser yet) via Java.

How it Works
From the envoy call I pull whLifetime:
{
"production": [
{
"type": "inverters",
"readingTime": 1583881411,
"wNow": 312,
"whLifetime": 1440898


From rtlamr I use the consumption from two of the 3 endpoint IDs it broadcasts:
{"Time":"2020-03-11T08:20:05.7600049-04:00",...{...,"EndpointID":xxxxxxx6,"Consumption":893...}}
{"Time":"2020-03-11T08:20:05.981719-04:00","...{...,"EndpointID":xxxxxxx8,"Consumption":1010,...}}


The Data
Currently, all the program does (unexcitingly) is to output the timestamp and those 3 values to a log file:

Code:
Date                              Delv   Recv  whLifetime
2020-03-10T19:40:39.2386502-04:00  885,  1010, 1440929
2020-03-11T09:29:25.7789322-04:00  893,  1011, 1442532

The Recv is what the system exports to the grid in kWh. The Delv is what is imported from the grid in kWh.

So far, that's far from the ideal system above. But, it's a start. By comparing two lines for a given span of time you can see how much was generated, how much was exported to the grid, how much was imported to the grid, and how much the house consumed.

For example, from the above two lines you can see the house consumed 8 kWh overnight, the system has already produced 1.6 kWh, and since the system has exported 1 kWh then house must have consumed ~0.6 kWh since the system has been producing. Pity the meter isn't in watts, for narrow time frames like this it's not to accurate, but over the course of a month it should be fine.

Current Problems
Only encountered one tricky bit in the code. Once the program terminated the SDR dongle stayed hot. Tried enabling/disabling the port. Resetting the USB port, then the same with the hub. But it had no effect.

I learned if I run rtl_tcp stand alone and end it with CTRL+C it cools back down as expected. So, figuring how to end the process with a CTRL+C from Java is the next hurdle (or altering the rtl_tcp code to take a shutdown signal). In the mean time, I've minimized the problem by killing rtl_tcp and restarting it with a low gain/frequency, then killing it again.

Update: Old source removed to not junk up Will's server. New source below!
 
Last edited:

Danny

Solar Addict
Joined
Sep 21, 2019
Messages
224
Not perfect by any means, but attached is the start of a java program that uses rtlamr to read an itron RF netmeter and makes a local call to the envoy to get the total watts produced via http://envoy.local/production.json. The attached program isn't meant for anyone to run to get useful data (it needs rtlamr/rtl_tcp in a lib subdirectory and to have the proper meter Endpoint IDs). But it is meant as a simple example of calling PowerShell and make a restful API call (although I haven't yet bothered with a json parser yet) via Java.

How it Works
From the envoy call I pull whLifetime:
{
"production": [
{
"type": "inverters",
"readingTime": 1583881411,
"wNow": 312,
"whLifetime": 1440898


From rtlamr I use the consumption from two of the 3 endpoint IDs it broadcasts:
{"Time":"2020-03-11T08:20:05.7600049-04:00",...{...,"EndpointID":xxxxxxx6,"Consumption":893...}}
{"Time":"2020-03-11T08:20:05.981719-04:00","...{...,"EndpointID":xxxxxxx8,"Consumption":1010,...}}


The Data
Currently, all the program does (unexcitingly) is to output the timestamp and those 3 values to a log file:

Code:
Date                              Recv  Delv   whLifetime

2020-03-10T19:40:39.2386502-04:00  885,  1010, 1440929

2020-03-11T09:29:25.7789322-04:00  893,  1011, 1442532

The Recv is what the system imports from the grid in kWh. The Delv is what is exported tothe grid in kWh.

So far, that's far from the ideal system above. But, it's a start. By comparing two lines for a given span of time you can see how much was generated, how much was exported to the grid, how much was imported to the grid, and how much the house consumed.

For example, from the above two lines you can see the house consumed 8 kWh overnight, the system has already produced 1.6 kWh, and since the system has exported 1 kWh then house must have consumed ~0.6 kWh since the system has been producing. Pity the meter isn't in watts, for narrow time frames like this it's not to accurate, but over the course of a month it should be fine.

Current Problems
Only encountered one tricky bit in the code. Once the program terminated the SDR dongle stayed hot. Tried enabling/disabling the port. Resetting the USB port, then the same with the hub. But it had no effect.

I learned if I run rtl_tcp stand alone and end it with CTRL+C it cools back down as expected. So, figuring how to end the process with a CTRL+C from Java is the next hurdle (or altering the rtl_tcp code to take a shutdown signal). In the mean time, I've minimized the problem by killing rtl_tcp and restarting it with a low gain/frequency, then killing it again.
I followed you to this thread. Thanks for your work.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Played with getting the ^C to work... it's seems there are ways to do it, but it involves writing a windows app best I can tell with GenerateConsoleCtrlEvent. But, as I was starting to do that, it occurred to me there was an easier way for now... just run a self terminating test version of rtl_tcp that comes bundled in the distributable. Basically, instead of running ./rtl_tcp.exe -g 1 -f 10000 at the, change it to rtl_test -t. Since it terminates normally it's suspends the SDR allowing it to stay cool.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
According to this article, there's an improved version of rtl_tcp for the raspberry pi. Looks like there are a number of versions, I'll have to look inside the version with rtlamr to see what it has... (it's written in go rather than c).
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Alerts

SMS or email are the two obvious choices. I'm not liking SMS as it depends on using a service, although I did see a cute android app that let's your cell phone get via wifi the message and send it as a text. But, that's extra battery drain.

That leaves email... while that's also a service it's one I already have. The problem there is everyone is security conscious these days so it would need encryption and that's program bloat (e.g., javax.mail package).

Powershell has a way of sending email too. Hate to keep leveraging Powershell... but it's already installed so that does minimize the bloat. So, the only difficult thing there is the credentials which have to originally be input via a GUI window (that is there's no unencrypted text version of the password hanging about). From there it can be exported/imported and reused. It would look something like this:

Code:
$c = Get-Credential
$c | Export-CliXml -Path 'cred.xml'
$credential = Import-CliXml -Path 'cred.xml'
Send-MailMessage -from 'Me <user@domain.com>' -To 'Me <user@domain.com>' -Subject 'Test mail' -Body "this is the email body with imported creds"  -SmtpServer 'xxx.com' -port xxx  -UseSsl -Credential $credential

What do you think? Is there a better way to do alerts?
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Alexa might be a good way to do alerts too.

Attached is the latest version. It's the same as the prior version with these changes:
  • the various classes have been put into their own files
  • Wrote a simple JSON parser class (couldn't see bulking up with one of the available parsers (e.g. GSON))
  • Added a class to send emails
  • split configuration data out into data files
  • The main program now checks to see if any of the microinverters have a fault and if so sends an email.
  • Put Enphase specific stuff into a new class
Next up will be to see if any of the panels are misbehaving. I'll get the panel data and make sure that for each panel the current watts is within a tolerance of all the rest. If not (panel going bad, shade, etc) it'll send an alert. Might even put some comments in... :unsure:

Reminder: This isn't meant as a finished program...rather just a stepping stone and some example java source for anyone interested.

Update: old source removed so Will doesn't run out of space... new source below...
 
Last edited:

boondox

Chief Engineer, RedNeckTech Industries
Joined
Mar 1, 2020
Messages
655
Do you have kwH in and out? I'd want to have today, this week, last month, year to date.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Right now it just outputs the 3 values (meter in/out in kWh and solar produced in watt-hours) and generates an alert if an inverter has a fault when you run it (see post #6). But there is a data collector/aggregation for rtlamr here.
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
So, setting up an Alexa alert looks easy...but you need to have a working internet (access to the Enphase API just requires a working intranet).
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Rofl! Alerts work! Got a bunch of last night:

Code:
123456789000: [envoy.cond_flags.pcu_chan.dcvoltagetoolow, envoy.cond_flags.pcu_ctrl.dc-pwr-low, envoy.cond_flags.obs_strs.failure]
123456789001: [envoy.cond_flags.pcu_ctrl.dc-pwr-low, envoy.cond_flags.obs_strs.failure]
   .
   .
   .
So, need to not count these as failures when it's night time....
 

boondox

Chief Engineer, RedNeckTech Industries
Joined
Mar 1, 2020
Messages
655
That gets a little more tricky. I helped a friend build a tracking solar array 10 years or so ago. We tried several different sensor systems and programming techniques. We had a number of issues leading to the array "hunting" or stalling. In the end what we did was have a GPS disciplined RTC (real time clock) and made a look up table for time and date and set the panel position that way. Although it was fun to watch it move when clouds went by. :)
 

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
How to find an underperforming panel

From http://envoy.local/api/v1/production/inverters you get:
JSON:
{    "serialNumber": "1111",   "lastReportDate": 1584225631,    "devType": 1,    "lastReportWatts": 56,    "maxReportWatts": 296  },
{    "serialNumber": "1112",   "lastReportDate": 1584225638,    "devType": 1,    "lastReportWatts": 20,    "maxReportWatts": 297  },...
Where each row represents inverter output for the day.

So, to find panels not matching up against the rest all you need do is to run the query, find the max/min of maxReportWatts, then find any panels reporting the maxReportWatts that are outside the range specified and alert on them. My panels do have an east/west component, but roof is overall fairly flat, so my numbers are generally close. You could widen the range or just ignore that query in the morning (by late afternoon everything should have hit it's peak). Right now I'm running at 5%, but that might get adjusted as I get some operational experience with it. Worse case, you could pool the different locations by serial numbers and just process them one at a time.

Not quite easy as pi(), turns out that query has to be authenticated, and the Envoy uses Digest authentication. I know, I could have bloated up and sucked in Apache or one of the other big libs...but hey, wrote my own authenticator from the RFCs.

Other changes:
  • Sprinkled lightly with some comments
  • as noted above, added alerts for panels not producing within the margins of their brothers
  • fixed the bug mentioned in #15.
  • Created a utility class and moved a few general functions into it rather than keep copies.
So, while this tells about one panel outside the bounds, it doesn't alert you if the entire system is degraded (e.g., heavy soiling). So, that's probably next to look into (see posts start at #2 for previous discussion that).

Update: delete the old source attachment, a new version replaces it that was added at the end of the thread on 3/19.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
Consider a panel shaded late in the afternoon, such a panel would have already hit it's max power...so clearly the algorithm above (#17) is flawed in that case. To do this properly, will possibly need to isolate panels by roof geometry and use lastReportWatts.
 
Last edited:

svetz

Works in theory! Practice? That's something else
Joined
Sep 20, 2019
Messages
4,906
Location
Key Largo
How to tell if the panels are producing power in accordance with the local conditions?

There are a lot of "weather" APIs out there to get local conditions, but most are "forecasts" rather than actual measured conditions. NOAA and Wunderground are the two exceptions I've found. Wunderground has an API, but it requires a token that you can only get if you have a station. NOAA is more heavily concentrated in coastal regions.

Not ideal, because unlike an API it may change frequently, but Wunderground does serve html pages showing the data that can be parsed.

The watts each panel produced should be a function of their efficiency, the Pmax temperature coefficient, and the local solar radiation level. So, how well does it work in practice? See https://diysolarforum.com/threads/solar-panel-size-efficiency-calculations.5312/
 

boondox

Chief Engineer, RedNeckTech Industries
Joined
Mar 1, 2020
Messages
655
Real solar radiation sensors aren't cheap. There are plenty of DIY options out there, light dependent resistors, LEDs with the top filed off, temperature sensors in a jar, all kinds of ways. As you would expect, there are pros and cons to each. Luckily you are not attempting a tracking array which increases the complexity. I do think that a local sensor will be required, especially if you want more than total failure detection. If you are trying to monitor long term panel degradation that will be much harder.

These are the reasons why in the tracking array I mentioned that we decided to go for a date/time basis for orienting the panels. We found the dance of the planet much more reliable and easy to work with than sensors. :)
 
Top