PowerView Pro data download

[GregTR]

I just got a Sol-Ark 15K installed and while I'm not hating the PoverView page (I am OK with my data being stored in China) I'm disappointed that I can't download the data. Is there a way to get the raw measurements back so I can correlate it with my Emporia measured values?

Alternatively, can I have both Solar Assistant and PowerView Pro both log data at the same time? I might invest in the Solar Assistant if that is the only way to obtain the raw data but I want to retain the other site too as it seems to be fine/looks OK.

 

[629658]

I’m in the exact same boat. I have written Solar Assistant but no response yet. Sol Ark has limited data to a single day now which is complete BS

 

[GregTR]

Just an update, I did purchase a Solar Assistant and I do have ability to download my data to play with it in Excel. It works on unison with PowerView and it even allows connection via CAN-BUS to my Homegrid battery.

I'm pretty happy with this setup, there are some minor things on my wish list for Solar Assistant, customizable charts, setting export data frequency, data backup from Solar Assistant SD to another device/cloud. It also has some issues with Sol-Ark 15K that the developer promised to address in the next software update.

 

[Ted Katsouras]

Just an update, I did purchase a Solar Assistant and I do have ability to download my data to play with it in Excel. It works on unison with PowerView and it even allows connection via CAN-BUS to my Homegrid battery.

I'm pretty happy with this setup, there are some minor things on my wish list for Solar Assistant, customizable charts, setting export data frequency, data backup from Solar Assistant SD to another device/cloud. It also has some issues with Sol-Ark 15K that the developer promised to address in the next software update.

What are the steps to download SA data to Excel?

 

[GregTR]

Go to charts, select the drop-down in a given chart and select (i) info and download a CSV file. Pretty straight forward.

Depending on the time range in view the resolution of the data will be more or less granular. Most you can get is every 10 seconds. In day view it's 1 minute intervals.

 

[robby]

I just got a Sol-Ark 15K installed and while I'm not hating the PoverView page (I am OK with my data being stored in China) I'm disappointed that I can't download the data. Is there a way to get the raw measurements back so I can correlate it with my Emporia measured values?

Alternatively, can I have both Solar Assistant and PowerView Pro both log data at the same time? I might invest in the Solar Assistant if that is the only way to obtain the raw data but I want to retain the other site too as it seems to be fine/looks OK.

Up to about 5 months ago you could download the last 3 days worth of powerview data. I complained to Sol-Ark about it being limited to 3 days and the Tech sounded surprised. He then tried it on his end with a company Inverter and he was also limited to 3 days.
I reported the issue and got an email that they had contacted Linter about the problem and that it should be fixed.
An update came in for PowerView after that just killed the whole thing and it has not worked since then,

BTW I have emailed Linter and they got back to me on several occasions really quickly. I report the problems and they say "Oh yes we know about it and it will be fixed in the next update".
Of course it is never fixed and they just keep giving me these pleasant replies that it will be fixed soon. Got fed up with it and stopped emailing them.

At this point I am just waiting for the AWS migration to be completed and then see what happens.

I love the Powerview Web App but the export thing seems to be a no go.

 

[GregTR]

Looks like they use the same grafana library and other Deye backend providers allow export of the data. It's good to learn that PoverView used to as well, at least on a limited fashion.

Now that I invested the money in Solar Assistant I'm a lot less concerned about it I just want to make sure the data doesn't get lost either by AWS migration or by virtue of a corrupted SD card database.

 

[robby]

Guess what, since you mentioned it and I noticed they did an update today I just tried it again and it is working but only for a single day of data.
Under Operation Data you just put in the serial number and pick a date and then export. It creates an XLSX file.

 

[robby]

Looks like they use the same grafana library and other Deye backend providers allow export of the data. It's good to learn that PoverView used to as well, at least on a limited fashion.

Now that I invested the money in Solar Assistant I'm a lot less concerned about it I just want to make sure the data doesn't get lost either by AWS migration or by virtue of a corrupted SD card database.

I think there is almost zero chance of a long term data loss during the AWS migration but the SD card is a serious concern.
I lost a few months of data with Solar Assistant when the Database got corrupted. He really needs to add a backup routine that can back up to either a USB stick or a drive on the network.

 

[GregTR]

Guess what, since you mentioned it and I noticed they did an update today I just tried it again and it is working but only for a single day of data.
Under Operation Data you just put in the serial number and pick a date and then export. It creates an XLSX file.

Yeah, that wasn't working just a week ago. A day's worth of data at 5 minute intervals is better than no data.

I got shell access to my Solar Assistant so I'll be backing up the database myself going forward until a backup/restore feature gets implemented for it.

 

[robby]

PowerView really collects a lot of data sets, I think it’s about 70 parameters.
It might be worth it to create a script to make it easier to graph selected parameters.

 

[GregTR]

I finally got around finishing a script for Solar Assistant to do hourly backups.

Requirements:
- USB stick, can be fat32 formatted for easy Windows read.
- Access to the Solar Assistant file system either via SSH or mounting the SD card on a linux machine

Add the following code into a file on /etc/cron.hourly/<filename>, I named mine sa_backup. Make sure the file is executable.

It will automatically make an hourly backup and save it on the USB drive. You can pull the stick pretty much any time and it will just skip the following backup until you put it back in. The backups seem to happen at the 17th minute of every hour on mine (you can verify by looking at your /etc/crontab file when the hourly call is made)

I also log the backup status to both ~solar-assistant/backup.log and the same file on the USB stick.

If you want to temporarily stop the backups, just place a nobackup file in solar-assistant's home directory by running touch ~solar-assistant/nobackup on the raspberry PI. I would put this file there before I went over to the device to pull the USB stick in case I would happen to be doing it at the 17 minute mark.

#!/usr/bin/bash
cleanup_tasks() {
    if [ "$2" -lt "1" ]; then
        /usr/bin/echo "$timestamp SA Backup Completed - $1" >> /home/solar-assistant/backup.log
#        /usr/bin/find /mnt/usb/SA_Backup/* -type d -mindepth 1 -maxdepth 1 -mtime +30 -exec /usr/bin/rm -rf {} \;
    else
        /usr/bin/echo "$timestamp SA Backup Failed - $1" >> /home/solar-assistant/backup.log
    fi
    if [ "$2" -lt "2" ]; then
        /usr/bin/cp /home/solar-assistant/backup.log /mnt/usb/SA_Backup/backup.log > /dev/null 2>&1
    elif [ "$2" -eq "2" ]; then
        /usr/bin/cp /home/solar-assistant/backup.log /mnt/usb/solar_assistant_backup.log > /dev/null 2>&1
    fi
    cd /mnt
    /usr/bin/sleep 5
    /usr/bin/umount /mnt/usb > /dev/null 2>&1
    exit 0
}
timestamp=$(date +"%Y%m%d_%H%M")

if [ ! -f "/home/solar-assistant/nobackup" ]; then
    if [ ! -d "/mnt/usb" ]; then
        /usr/bin/mkdir /mnt/usb || cleanup_tasks "/mnt/usb creation failed" 3
        /usr/bin/chmod 755 /mnt/usb || cleanup_tasks "chmod /mnt/usb failed" 3
    fi
    /usr/bin/mount /dev/sda1 /mnt/usb > /dev/null 2>&1 || cleanup_tasks "Mount Failed" 3
    if [ ! -d "/mnt/usb/SA_Backup" ]; then
        /usr/bin/mkdir /mnt/usb/SA_Backup || cleanup_tasks "/mnt/usb/SA_Backup creation failed" 2
        /usr/bin/chmod 755 /mnt/usb/SA_Backup || cleanup_tasks "chmod /mnt/usb/SA_Backup failed" 2
    fi
    cd /mnt/usb/SA_Backup > /dev/null 2>&1 || cleanup_tasks "SA_Backup Directory Not Found" 2
    /usr/bin/influxd backup -portable /mnt/usb/SA_Backup/$timestamp/ > /dev/null 2>&1 || cleanup_task "InfluxDB Failed" 1
    /usr/bin/tar -czf /mnt/usb/SA_Backup/$timestamp/grafana_db.tar.gz -C /var/lib/grafana/ grafana.db || cleanup_task "grafana tar Faield" 1
    if [ -f "/home/solar-assistant/backup.tar.gz" ]; then
            /usr/bin/rm -f /home/solar-assistant/backup.tar.gz
    fi
    /usr/bin/tar -czf /home/solar-assistant/backup.tar.gz -C /mnt/usb/SA_Backup/$timestamp/ . || cleanup_task "package tar Failed" 1
    cleanup_tasks "Success" 0
else
    /usr/bin/echo "$timestamp Backup Skipped" >> /home/solar-assistant/nobackup
    /usr/bin/echo "$timestamp SA Backup Skipped" >> /home/solar-assistant/backup.log
fi

 

[GregTR]

So just a quick update: it looks like the good folks at Solar Assistant tried to clamp down access in very interesting ways but in their work they made a critical mistake that also broke cron.

They tried to clamp down access and they tried to disable access with password or any other means besides their own private key. They obfuscated the allow list of keys, they make sure that sshd has login with password turned off and they also ensured that only one key is allowed in the file and it has to be their key. If you do anything "hokey" like adding another key, replacing the key, moving the source of the keys for sshd to another file or enabling passworded logins PAM will deny access.

BUT they also mucked with the /etc/pam.d/common-auth file and added a one liner.

auth    [success=die default=ignore]    pam_localuser.so

This disables any account access for accounts in /etc/passwd. Root is in that file and everything that is in /etc/cron.d is run as root currently on the system. Since /etc/pam.d/cron has a line to include "common-auth" it also denies access to any local account, therefore cron fails. It's a really weird fail because root can run "crontab -[ler]", add its own commands etc. but anything that runs from the cron daemon fails that check during execution with a very non-descript error in the log.

Oct  1 15:04:27 solar-assistant cron[390]: (CRON) INFO (pidfile fd = 3)
Oct  1 15:04:27 solar-assistant cron[390]: (CRON) INFO (Running @reboot jobs)
Oct  1 15:17:01 solar-assistant cron[390]: Permission denied
Oct  1 16:17:01 solar-assistant cron[390]: Permission denied
Oct  1 17:17:01 solar-assistant cron[390]: Permission denied
Oct  1 18:17:01 solar-assistant cron[390]: Permission denied

After I removed that one line from the common-auth (which essentially puts it back to its default state, I ran a diff on all common-* files after I reset and this was the only change) cron works again:

Oct 12 08:17:01 solar-assistant CRON[21739]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 09:17:01 solar-assistant CRON[27582]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 10:17:01 solar-assistant CRON[1207]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 11:17:01 solar-assistant CRON[7209]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 12:17:01 solar-assistant CRON[13225]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)