Using Victron VRM CSV files to measure AC Consumption

ricardocello · Jun 20, 2024

In this thread

how much inverting power would you need to run your whole house 100% of the time

Peak AC Consumption? That’s a job for Victron VRM. I never thought to look, but since 1 Jan 2024 it was almost 30kW!

diysolarforum.com

I decided to use VRM to show my AC consumption and determine the peak power used in the house (400A) service (close to 30 kW).
While the chart is useful, I want real statistics.

Turns out if you ask VRM to download CSV files for your system from the beginning of the year, it sends you an email with a download link later.
The amount of data is staggering!
I downloaded a zip file from the link, and they broke the CSV into three pieces, each piece well over 100 Mbytes.

Excel and Numbers both bombed out trying to open them, so I wrote a Python script.
Here are the results for the three files:

Code:

$ python3 parse_vrm.py 0_HomeESS_log_20240101-0000_to_20240301-0000.csv
Samples:           88860
L1 Consumption (kW):    [Mean 1.855] [Min 0.315] [Max 14.963]
L2 Consumption (kW):    [Mean 1.744] [Min 0.132] [Max 14.603]
Total Consumption (kW): [Mean 3.599] [Min 0.525] [Max 29.561]

$ python3 parse_vrm.py 0_HomeESS_log_20240301-0000_to_20240430-0100.csv
Samples:           89120
L1 Consumption (kW):    [Mean 1.813] [Min 0.000] [Max 12.197]
L2 Consumption (kW):    [Mean 1.694] [Min 0.000] [Max 12.822]
Total Consumption (kW): [Mean 3.507] [Min 0.000] [Max 25.019]

$ python3 parse_vrm.py 0_HomeESS_log_20240430-0100_to_20240619-2133.csv
Samples:           70279
L1 Consumption (kW):    [Mean 1.958] [Min 0.000] [Max 10.574]
L2 Consumption (kW):    [Mean 1.918] [Min 0.000] [Max 10.381]
Total Consumption (kW): [Mean 3.876] [Min 0.000] [Max 20.955]

@1201

Brucey · Jun 20, 2024

ricardocello said:
In this thread

how much inverting power would you need to run your whole house 100% of the time

Peak AC Consumption? That’s a job for Victron VRM. I never thought to look, but since 1 Jan 2024 it was almost 30kW!

diysolarforum.com

I decided to use VRM to show my AC consumption and determine the peak power used in the house (400A) service (close to 30 kW).
While the chart is useful, I want real statistics.

Turns out if you ask VRM to download CSV files for your system from the beginning of the year, it sends you an email with a download link later.
The amount of data is staggering!
I downloaded a zip file from the link, and they broke the CSV into three pieces, each piece well over 100 Mbytes.

Excel and Numbers both bombed out trying to open them, so I wrote a Python script.
Here are the results for the three files:

Code:

$ python3 parse_vrm.py 0_HomeESS_log_20240101-0000_to_20240301-0000.csv Samples: 88860 L1 Consumption (kW): [Mean 1.855] [Min 0.315] [Max 14.963] L2 Consumption (kW): [Mean 1.744] [Min 0.132] [Max 14.603] Total Consumption (kW): [Mean 3.599] [Min 0.525] [Max 29.561] $ python3 parse_vrm.py 0_HomeESS_log_20240301-0000_to_20240430-0100.csv Samples: 89120 L1 Consumption (kW): [Mean 1.813] [Min 0.000] [Max 12.197] L2 Consumption (kW): [Mean 1.694] [Min 0.000] [Max 12.822] Total Consumption (kW): [Mean 3.507] [Min 0.000] [Max 25.019] $ python3 parse_vrm.py 0_HomeESS_log_20240430-0100_to_20240619-2133.csv Samples: 70279 L1 Consumption (kW): [Mean 1.958] [Min 0.000] [Max 10.574] L2 Consumption (kW): [Mean 1.918] [Min 0.000] [Max 10.381] Total Consumption (kW): [Mean 3.876] [Min 0.000] [Max 20.955]

@1201

Very cool!

ricardocello · Jun 20, 2024

Brucey said:
Very cool!

Here's the code if anyone wants it. Not proud of it, released into Public Domain (as is).

Code:

# This is released to Public Domain as is.

import sys
import csv


class VRMLogFile:

    def __init__(self, filename):
        self.filename = filename
        self.csvfile = open(filename, newline='')

        self.reader = csv.reader(self.csvfile, delimiter=',', quotechar='"')
        self.source = next(self.reader)
        self.description = next(self.reader)
        self.units = next(self.reader)
        self.header = list(zip(self.source, self.description, self.units))

    def show_parameters(self):
        for p in self.header:
            print(p)

    def find_parameter(self, source, description):
        for index, v in enumerate(self.header):
            if source in v[0] and description in v[1]:
                return index
        return -1

    def next_data(self):
        return next(self.reader)

    def show_consumption(self):
        l1_consumption_watts_index = self.find_parameter('System overview [0]', 'AC Consumption L1')
        l2_consumption_watts_index = self.find_parameter('System overview [0]', 'AC Consumption L2')
        if l1_consumption_watts_index < 0 or l2_consumption_watts_index < 0:
            return

        count = 0

        l1_sum = 0.0
        l2_sum = 0.0
        total_sum = 0.0

        l1_max = 0.0
        l2_max = 0.0
        total_max = 0.0

        l1_min = 1000000.0
        l2_min = 1000000.0
        total_min = 1000000.0

        try:
            while True:
                d = vrm_log.next_data()
                l1_str = d[l1_consumption_watts_index]
                l2_str = d[l2_consumption_watts_index]
                if not l1_str or not l2_str:
                    continue

                l1 = float(l1_str) / 1000.0
                l2 = float(l2_str) / 1000.0
                total = l1 + l2

                l1_sum += l1
                l2_sum += l2
                total_sum += total

                l1_max = max(l1_max, l1)
                l2_max = max(l2_max, l2)
                total_max = max(total_max, total)

                l1_min = min(l1_min, l1)
                l2_min = min(l2_min, l2)
                total_min = min(total_min, total)

                count += 1
        except StopIteration:
            pass

        l1_mean = l1_sum / count
        l2_mean = l2_sum / count
        total_mean = total_sum / count

        print(f'Samples:           {count}')
        print(f'L1 Consumption (kW):    [Mean {l1_mean:.3f}] [Min {l1_min:.3f}] [Max {l1_max:.3f}]')
        print(f'L2 Consumption (kW):    [Mean {l2_mean:.3f}] [Min {l2_min:.3f}] [Max {l2_max:.3f}]')
        print(f'Total Consumption (kW): [Mean {total_mean:.3f}] [Min {total_min:.3f}] [Max {total_max:.3f}]')


if __name__ == '__main__':
    file = sys.argv[1]    # .csv filename

    vrm_log = VRMLogFile(file)
    vrm_log.show_parameters()
    vrm_log.show_consumption()

ricardocello · Jun 20, 2024

Also, this is what VRM for my setup is recording (it's very comprehensive).
Others will be different depending on configuration.

Part 1 (split due to forum limitation of 10000 characters)

Code:

('timestamp', 'America/New_York (-04:00)', '')
('Gateway [0]', 'VRM Log time offset', 's')
('Gateway [0]', 'ESS Scheduled Charging', '')
('Gateway [0]', 'Grid setpoint', 'W')
('Gateway [0]', 'Active SOC Limit', '%')
('Gateway [0]', 'Relay 1 state', '')
('Gateway [0]', 'CCGX Relay 2 state', '')
('Gateway [0]', 'Relay 1 function', '')
('Gateway [0]', 'Relay 2 function', '')
('Gateway [0]', 'Whether system accepts Dynamic ESS instructions.', '')
('Gateway [0]', 'Dynamic ESS battery capacity', '%.2F')
('Gateway [0]', 'Actual working state of Dynamic ESS', '')
('Gateway [0]', 'Dynamic ESS error code', '')
('Gateway [0]', 'DESS Grid/battery energy exporting restrictions', '')
('Gateway [0]', 'Target SOC for a scheduled charge slot', '%')
('Gateway [0]', 'Solar Irradiance', 'W/m²')
('Gateway [0]', 'Solar Irradiance Forecast', 'W/m²')
('Gateway [0]', 'A place for the DESS algorithm to store an efficiency figure', '')
('VE.Bus System [276]', 'Phase rotation', '')
('VE.Bus System [276]', 'Input voltage phase 1', 'V')
('VE.Bus System [276]', 'VE.Bus custom name', '')
('VE.Bus System [276]', 'Input voltage phase 2', 'V')
('VE.Bus System [276]', 'Input current phase 1', 'A')
('VE.Bus System [276]', 'Input current phase 2', 'A')
('VE.Bus System [276]', 'Input frequency 1', 'Hz')
('VE.Bus System [276]', 'Input frequency 2', 'Hz')
('VE.Bus System [276]', 'Input power 1', 'W')
('VE.Bus System [276]', 'Input power 2', 'W')
('VE.Bus System [276]', 'Output voltage phase 1', 'V')
('VE.Bus System [276]', 'Output voltage phase 2', 'V')
('VE.Bus System [276]', 'Output current phase 1', 'A')
('VE.Bus System [276]', 'Output current phase 2', 'A')
('VE.Bus System [276]', 'Output frequency', 'Hz')
('VE.Bus System [276]', 'Output power 1', 'W')
('VE.Bus System [276]', 'Output power 2', 'W')
('VE.Bus System [276]', 'Voltage', 'V')
('VE.Bus System [276]', 'Current', 'A')
('VE.Bus System [276]', 'Battery temperature', 'C')
('VE.Bus System [276]', 'Active input', '')
('VE.Bus System [276]', 'Active input current limit', 'A')
('VE.Bus System [276]', 'AC Input 1 Current Limit', 'A')
('VE.Bus System [276]', 'AC Input 2 Current Limit', 'A')
('VE.Bus System [276]', 'AC Input 1 Current Limit Adjustable', '')
('VE.Bus System [276]', 'AC Input 1 Current Limit Adjustable', '')
('VE.Bus System [276]', 'VE.Bus state', '')
('VE.Bus System [276]', 'VE.Bus Error', '')
('VE.Bus System [276]', 'Switch Position', '')
('VE.Bus System [276]', 'Temperature', '')
('VE.Bus System [276]', 'Low battery', '')
('VE.Bus System [276]', 'Overload', '')
('VE.Bus System [276]', 'Temperatur sensor alarm', '')
('VE.Bus System [276]', 'Voltage sensor alarm', '')
('VE.Bus System [276]', 'High DC Ripple', '')
('VE.Bus System [276]', 'Temperature L1', '')
('VE.Bus System [276]', 'Low battery L1', '')
('VE.Bus System [276]', 'Overload L1', '')
('VE.Bus System [276]', 'High DC Ripple L1', '')
('VE.Bus System [276]', 'Temperature L2', '')
('VE.Bus System [276]', 'Low battery L2', '')
('VE.Bus System [276]', 'Overload L2', '')
('VE.Bus System [276]', 'High DC Ripple L2', '')
('VE.Bus System [276]', 'Temperature L3', '')
('VE.Bus System [276]', 'Low battery L3', '')
('VE.Bus System [276]', 'Overload L3', '')
('VE.Bus System [276]', 'High DC Ripple L3', '')
('VE.Bus System [276]', 'Charge state', '')
('VE.Bus System [276]', 'BMS lost', '')
('VE.Bus System [276]', 'Unequal output distribution in parallel system: L1', '')
('VE.Bus System [276]', 'Unequal output distribution in parallel system: L2', '')
('VE.Bus System [276]', 'Unequal output distribution in parallel system: L3', '')
('VE.Bus System [276]', 'Unequal mains distribution in parallel system: L1', '')
('VE.Bus System [276]', 'Unequal mains distribution in parallel system: L2', '')
('VE.Bus System [276]', 'Unequal mains distribution in parallel system: L3', '')
('VE.Bus System [276]', 'Ignore AC input', '')
('VE.Bus System [276]', 'Ignore AC input 2', '')
('VE.Bus System [276]', 'Serial number of 1st device', '')
('VE.Bus System [276]', 'Serial number of 2nd device', '')

ricardocello · Jun 20, 2024

Part 2 (split due to forum limitation of 10000 characters)

Code:

('Battery Monitor [279]', 'Voltage', 'V')
('Battery Monitor [279]', 'Starter battery voltage', 'V')
('Battery Monitor [279]', 'Current', 'A')
('Battery Monitor [279]', 'Consumed Amphours', 'Ah')
('Battery Monitor [279]', 'State of charge', '%')
('Battery Monitor [279]', 'Time to go', 'h')
('Battery Monitor [279]', 'Low voltage alarm', '')
('Battery Monitor [279]', 'High voltage alarm', '')
('Battery Monitor [279]', 'Low starter-voltage alarm', '')
('Battery Monitor [279]', 'High starter-voltage alarm', '')
('Battery Monitor [279]', 'Low state-of-charge alarm', '')
('Battery Monitor [279]', 'Low battery temperature alarm', '')
('Battery Monitor [279]', 'High battery temperature alarm', '')
('Battery Monitor [279]', 'Mid-voltage alarm', '')
('Battery Monitor [279]', 'Deepest discharge', 'Ah')
('Battery Monitor [279]', 'Last discharge', 'Ah')
('Battery Monitor [279]', 'Average discharge', 'Ah')
('Battery Monitor [279]', 'Charge cycles', '')
('Battery Monitor [279]', 'Full discharges', '')
('Battery Monitor [279]', 'Total Ah drawn', 'Ah')
('Battery Monitor [279]', 'Minimum voltage', 'V')
('Battery Monitor [279]', 'Maximum voltage', 'V')
('Battery Monitor [279]', 'Time since last full charge', '')
('Battery Monitor [279]', 'Automatic syncs', '')
('Battery Monitor [279]', 'Low voltage alarms', '')
('Battery Monitor [279]', 'High voltage alarms', '')
('Battery Monitor [279]', 'Minimum starter voltage', 'V')
('Battery Monitor [279]', 'Maximum starter voltage', 'V')
('Battery Monitor [279]', 'Discharged Energy', 'kWh')
('Battery Monitor [279]', 'Charged Energy', 'kWh')
('Solar Charger [0]', 'Voltage', 'V')
('Solar Charger [0]', 'Current', 'A')
('Solar Charger [0]', 'Battery watts', 'W')
('Solar Charger [0]', 'Battery temperature', 'C')
('Solar Charger [0]', 'Load state', '')
('Solar Charger [0]', 'Charger on/off', '')
('Solar Charger [0]', 'Charge state', '')
('Solar Charger [0]', 'PV voltage', 'V')
('Solar Charger [0]', 'PV power', '')
('Solar Charger [0]', 'MPPT State', '')
('Solar Charger [0]', 'Relay on the charger', '')
('Solar Charger [0]', 'Low batt. voltage alarm', '')
('Solar Charger [0]', 'High batt. voltage alarm', '')
('Solar Charger [0]', 'Yield today', 'kWh')
('Solar Charger [0]', 'Maximum charge power today', 'W')
('Solar Charger [0]', 'Yield yesterday', 'kWh')
('Solar Charger [0]', 'Maximum charge power yesterday', 'W')
('Solar Charger [0]', 'Error code', '')
('Solar Charger [0]', 'User yield', 'kWh')
('Solar Charger [0]', 'PV Yield Solar Charger Forecast', 'Wh')
('Solar Charger [1]', 'Voltage', 'V')
('Solar Charger [1]', 'Current', 'A')
('Solar Charger [1]', 'Battery watts', 'W')
('Solar Charger [1]', 'Battery temperature', 'C')
('Solar Charger [1]', 'Load state', '')
('Solar Charger [1]', 'Charger on/off', '')
('Solar Charger [1]', 'Charge state', '')
('Solar Charger [1]', 'PV voltage', 'V')
('Solar Charger [1]', 'PV power', '')
('Solar Charger [1]', 'Solar charge current limit', 'A')
('Solar Charger [1]', 'MPPT State', '')
('Solar Charger [1]', 'Relay on the charger', '')
('Solar Charger [1]', 'Low batt. voltage alarm', '')
('Solar Charger [1]', 'High batt. voltage alarm', '')
('Solar Charger [1]', 'Yield today', 'kWh')
('Solar Charger [1]', 'Maximum charge power today', 'W')
('Solar Charger [1]', 'Yield yesterday', 'kWh')
('Solar Charger [1]', 'Maximum charge power yesterday', 'W')
('Solar Charger [1]', 'Error code', '')
('Solar Charger [1]', 'User yield', 'kWh')
('Solar Charger [1]', 'PV Yield Solar Charger Forecast', 'Wh')
('Solar Charger [278]', 'PV Yield Solar Charger Forecast', 'Wh')
('System overview [0]', '#1 Low SOC; discharge disabled', '')
('System overview [0]', '#2 BatteryLife is active', '')
('System overview [0]', '#3 Charge disabled by BMS', '')
('System overview [0]', '#4 Discharge disabled by BMS', '')
('System overview [0]', '#5 Slow charge is active', '')
('System overview [0]', '#6 Charge disabled by user setting', '')
('System overview [0]', '#7 Discharge disabled by user setting', '')
('System overview [0]', 'PV - DC-coupled', 'W')
('System overview [0]', 'AC Consumption L1', 'W')
('System overview [0]', 'AC Consumption L2', 'W')
('System overview [0]', 'AC Consumption L3', 'W')
('System overview [0]', 'Grid L1', 'W')
('System overview [0]', 'Grid L2', 'W')
('System overview [0]', 'Grid L3', 'W')
('System overview [0]', 'DC System', 'W')
('System overview [0]', 'Voltage', 'V')
('System overview [0]', 'Current', 'A')
('System overview [0]', 'VE.Bus charge current', 'A')
('System overview [0]', 'Battery Power', 'W')
('System overview [0]', 'VE.Bus charge power', 'W')
('System overview [0]', 'Battery SOC', '%')
('System overview [0]', 'Battery state', '')
('System overview [0]', 'Battery Consumed Amphours', 'Ah')
('System overview [0]', 'Battery Time to Go', 'h')
('System overview [0]', 'DVCC Multiple batteries alarm', '')
('System overview [0]', 'GX Error #48 DVCC with incompatible firmware', '')
('System overview [0]', 'No grid meter alarm', '')
('System overview [0]', 'AC-Input', '')
('System overview [0]', 'Consumption Forecast', 'Wh')
('System overview [276]', 'Grid alarm', '')
('Grid meter [30]', 'Grid L1 - Power', 'W')
('Grid meter [30]', 'Grid L2 - Power', 'W')
('Grid meter [30]', 'Grid L3 - Power', '')
('Grid meter [30]', 'Grid meter voltage L1', 'V')
('Grid meter [30]', 'Grid meter current L1', 'A')
('Grid meter [30]', 'Grid meter voltage L2', 'V')
('Grid meter [30]', 'Grid meter current L2', 'A')
('Grid meter [30]', 'Grid meter voltage L3', 'V')
('Grid meter [30]', 'Grid meter current L3', 'A')
('Grid meter [31]', 'Grid L1 - Power', 'W')
('Grid meter [31]', 'Grid L2 - Power', 'W')
('Grid meter [31]', 'Grid meter voltage L1', 'V')
('Grid meter [31]', 'Grid meter current L1', 'A')
('Grid meter [31]', 'Grid meter voltage L2', 'V')
('Grid meter [31]', 'Grid meter current L2', 'A')
('Grid meter [32]', 'Grid L1 - Power', 'W')
('Grid meter [32]', 'Grid L2 - Power', 'W')
('Grid meter [32]', 'Grid L3 - Power', '')
('Grid meter [32]', 'Grid meter voltage L1', 'V')
('Grid meter [32]', 'Grid meter current L1', 'A')
('Grid meter [32]', 'Grid meter voltage L2', 'V')
('Grid meter [32]', 'Grid meter current L2', 'A')
('Grid meter [32]', 'Grid meter voltage L3', 'V')
('Grid meter [32]', 'Grid meter current L3', 'A')
('Temperature sensor [24]', 'Temperature', 'C')
('Temperature sensor [24]', 'Temperature status', '')
('Temperature sensor [25]', 'Temperature', 'C')
('Temperature sensor [25]', 'Temperature status', '')
('Temperature sensor [26]', 'Temperature', 'C')
('Temperature sensor [26]', 'Temperature status', '')
('DC Source [278]', 'Battery Voltage', '')
('DC Source [278]', 'Generated Current', '')
('DC Source [278]', 'Total energy produced', 'kWh')
('DC Source [278]', 'Low voltage alarm', '')
('DC Source [278]', 'High voltage alarm', '')
('DC Source [278]', 'Low auxiliary voltage alarm', '')
('DC Source [278]', 'High auxiliary voltage alarm', '')
('DC Source [278]', 'Low temperature alarm', '')
('DC Source [278]', 'High temperature alarm', '')

1201 · Jun 20, 2024

Very nice. Is there a way to store this locally so you don't have to get a huge file from victron?

ricardocello · Jun 20, 2024

1201 said:
Very nice. Is there a way to store this locally so you don't have to get a huge file from victron?

That's a really great question, I'll ask over on Victron community.

1201 · Jun 20, 2024

ricardocello said:
That's a really great question, I'll ask over on Victron community.

I found this on Reddit. Dont know how accurate it is.

Victron uses MQTT to ship data to VRM. Victron Connect is an app that views the data from VRM.

You can "peel off a copy" of that mqtt data by configuring a local MQTT broker and subscribe to all the data. From there, you map the data you're interested in storing into a time series database.

The basic flow is \[MQTT data source (GX device\] \] -> \[broker subscription\] -> \[data map or pump service\] -> \[time series database like influx, graphite, elastic...\] -> \[visualization like grafana, kibana, etc\]

ricardocello · Jun 20, 2024

1201 said:
I found this on Reddit. Dont know how accurate it is.

Victron uses MQTT to ship data to VRM. Victron Connect is an app that views the data from VRM.

You can "peel off a copy" of that mqtt data by configuring a local MQTT broker and subscribe to all the data. From there, you map the data you're interested in storing into a time series database.

The basic flow is \[MQTT data source (GX device\] \] -> \[broker subscription\] -> \[data map or pump service\] -> \[time series database like influx, graphite, elastic...\] -> \[visualization like grafana, kibana, etc\]

Oh, that's certainly doable. I'll let you know how it goes.

featherlite · Jun 20, 2024

I am flabbergasted by the fact that you guys know how to "write code" and do what you do.

ricardocello · Jun 20, 2024

featherlite said:
I am flabbergasted by the fact that you guys know how to "write code" and do what you do.

One of the great things about this forum is the broad level of knowledge and practical experience across multiple disciplines.
I’ve learned a lot from the people here, and am thankful for the vibrant discussions.

Using Victron VRM CSV files to measure AC Consumption

ricardocello

Watching and Learning

how much inverting power would you need to run your whole house 100% of the time

Brucey

Solar Wizard

how much inverting power would you need to run your whole house 100% of the time

ricardocello

Watching and Learning

ricardocello

Watching and Learning

ricardocello

Watching and Learning

1201

Solar Wizard

ricardocello

Watching and Learning

1201

Solar Wizard

ricardocello

Watching and Learning

featherlite

Solar Enthusiast

ricardocello

Watching and Learning