Protecting EG4 18kPV from Remote Attacks

[Quattrohead]

One word that should horrify everyone "SCADA"
These systems run most of the infrastructure in the world and are very old and insecure, this is where infrastructure attack is going to happen.
Mind you there is no harm in doing what you can for yourself as well if you have the know-how.

 

[ChrisG]

After a recent EG4 18kPV installation I was curious about how its cloud monitoring system was designed. To my surprise, there is no encryption whatsoever, and raw RS485 MODBUS commands are being sent directly across the public internet.

This means a MITM attacker has trivial control over sensitive inverter parameters, such as battery thresholds and grid-interactive features. As just one example, the parameters used for IEEE 1547 interaction could be misconfigured as part of a larger effort to destabilize the grid. Finally, the entire inverter firmware appears to be updatable via this route, possibly opening up an even wider attack surface area beyond just the documented MODBUS parameters.

There were recent rumors of solar inverters being attacked remotely (also discussed on this forum), but they appear to have been dispelled after investigators followed-up:

• https://today.lorientlejour.com/art...systems-explode-during-wednesdays-attack.html
• https://diysolarforum.com/threads/l...ms-exploded-in-several-areas-of-beirut.91076/

Regardless, it remains prudent to reduce the external attack surface of these devices, as they are often connected to potent energy storage systems. One way to mitigate this would be to completely disconnect the inverter from the Internet, but that would mean having to roll our own local monitoring.

Instead, I used the raw plaintext protocol to my advantage and wrote some lightweight router firewall rules that allow a handful of vetted innocent requests through, while blocking all other mutation requests or otherwise undocumented features. This lets us leverage the existing EG4 cloud monitoring and inverter as-is with no extra hardware requirements, while also protecting our equipment. Here's the firewall rules, instructions, and background:

• https://github.com/jsharkey/lxp-firewall

When attempting to modify inverter settings in the cloud or app I now get "Timeout" or "Unknown error" messages, and then it takes a few minutes for RS485 tunnel to be reestablished and automatic statistics to begin flowing again. There's also a narrow allowance that allows remotely modifying the date/time to adjust for clock drift and applying DST changes.

Hopefully others find this useful as they consider how to best protect their equipment.

Trying to follow your no encryption statement. Are you saying the API that our dongles send data to at EG4 AWS systems are not encrypted or tokenized? If so this is something that home network vlans, opensense, etc can fix. Do you have wireshark or some other tool evidence of it?

If that is indeed the case one could easily grab a device ID from the many screenshots here and flood the API with garbage data for specific user/device.

 

[pshoe]

Aren't many of these things using the Solarman platform? Isn't the dongle a commodity purchased from a 3rd party? Its possible that other companies have exactly the same concern. Deye, Sol-Ark, Sunsynk...

As noted you could vlan the device to block it from phoning home and then run something like solar-assistant.

 

[shazam]

One more vote for not attaching things to the grid. I never put the dongle on the SA15k for internet. My IotaWatt and Solar Assistant sit on the LAN and are blocked at the router. I use a VPN on the router to hit the LAN for monitoring that stuff and my security cameras, which are also blocked. Ya do what ya can. I did IT work for a lotta years....if somebody wants you bad enough, they'll getcha!

Jim

 

[TM48]

The Luxpower dongles have three access methods. Bluetooth, local web server and the Wifi connection to your network. The connection to your network has always been encrypted at least from the firmware version V2.02 I started with. The local Wifi web server was not. It was literally a wide open connection into your inverter. This was finally fixed with version v2.06. V2.04 gave you the ability to upgrade the dongle firmware. Unfortunately the blue tooth is still open. Another gripe on BT is the need to allow locations to use. I would like the ability to disable BT completely.
I normally use Solar Assistant and Home Assistant and block the internet from my IOT lan. Both HA and SA will phone home continuously if you let them. BTW if you're using opnsense as your router note if you haven't already that it does not remove sessions automatically if you merely disable the rules allowing access to the internet. You have to manually destroy the established sessions.

 

[Oldphile]

Since Sol-Ark migrated to US servers with the new MySolArk App. my system has disconnected from the grid twice due to changed grid settings. Once in June and again in October. The migration began in June.

 

[jsharkey]

I wish I understood 1/2 of what you are saying. Is there a way to protect the inverters without knowing how to write code? I am not familiar with writing code but would like to be able to protect my investment if possible.

Sorry if it's a bit obtuse, these instructions are aimed at folks who are already running their own router firmware. OpenWrt is a popular choice as it's open source, and can be flashed onto hundreds of different models of popular devices:

• https://openwrt.org/toh/start

Running open-source router firmware like OpenWrt is generally the first step towards protecting your overall network against attacks, since closed-source router firmware from the manufacturer is often outdated with known vulnerabilities or even hidden backdoors. Here's a good overview:

• https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-270a

Once you have an OpenWrt router, the steps on the GitHub README.md should be pretty straightforward to apply. If you have another non-OpenWrt router that supports Linux nftables you should be able to adapt the rules to it quite easily.

 

[jsharkey]

Good post. I've had the same concerns and even posted about it, but didn't garner a lot of interest. You may want to also set outbound rules. If the firmware is already compromised it can establish an outbound connection.

A diagram would help as it's not clear how you connect the router to the wireless dongle.

I've considered bridging a connection across NICs on my windows box and using the built-in firewall. I live in Linux, but Windows might be easier for most people.

In my case the OpenWrt router is the router/NAT for the entire site, with POE powered Wi-Fi access points and VLANs. The EG4 uses the included Wi-Fi dongle to connect to an "insecure" IoT VLAN SSID that I'm running, which already has increased scrutiny and isolation from other devices.

Regarding "outbound" rules, I believe the forwarding table is consulted for traffic being routed both to/from the Internet, so it should be catching both inbound and outbound packets. Here's the nftables flowchart:

• https://wiki.nftables.org/wiki-nfta...lter_hooks_into_Linux_networking_packet_flows

I'm not sure if Windows (or other OS) firewalls supports DPI rules, which is what we're doing to inspect the MODBUS packets.

 

[jsharkey]

Are any of you smart folks willing to write a detailed step by step instruction manual? I have a Protecti Vault that the internet recommended to install opnsense for a firewall. I was going to follow the Youtube video on setting it up but never thought of how to incorporate the inverters.

OPNsense and pfSense are good choices for a firewall, but as far as I know the "pf" firewall that they're likely using from FreeBSD doesn't offer a way to do deep packet inspection (DPI) which is what I'm doing to inspect the MODBUS contents going to/from the inverter.

• https://man.freebsd.org/cgi/man.cgi?query=pfctl

 

[jsharkey]

LOL, this isn’t a troll/didn’t take meds today? I guess you posted the receipts on GitHub.

I am also kind of shocked that 1741SB/1547 certification doesn’t include some basic security audit.

Wow.

@EG4TechSolutionsTeam why was this security design deemed acceptable?

The closest "rule" that would apply would be NEC 2023 110.3(A)(8) [pasted below], but I doubt an AHJ would be digging this deep unless they were really paranoid or looking for technicalities to reject. The "life safety" aspect of this rule is probably clearly defined by NFPA 101 "Life Safety Code", and probably doesn't apply to typical residential installations.

110.3 Examination, Identification, Installation, Use, and Listing (Product Certification) of Equipment.
(A) Examination. In judging equipment, considerations such as the following shall be evaluated:
(8) Cybersecurity for network-connected life safety equipment to address its ability to withstand unauthorized updates and malicious attacks while continuing to perform its intended safety functionality

Informational Note No. 3: See the ANSI/ISA 62443 series of standards for industrial automation and control systems, the UL 2900 series of standards for software cybersecurity for network-connectable products, and UL 5500, Standard for Remote Software Updates, which are standards that provide frameworks to mitigate current and future security cybersecurity vulnerabilities and address software integrity in systems of electrical equipment.

 

[jsharkey]

There’s a tradeoff here between functionality (remote access, EG4/SS tech support, remote firmware upgrades, data logging) and the danger of someone mucking about with your system, or worst case everyone’s system.

One potential mitigating factor is that EG4 has AWS servers in the US so it’s possible SS is firewalled from any Chinese govt malfeasance.

Note that it’s possible for firmware to start acting wonky (that’s a technical term) after a certain amount of time which will cause you to reconnect to the servers for tech support and firmware updates, which essentially breaches your firewall. Not saying they are doing this, just that they could, and some of the worst malware is from nation-state actors. All kinds of stuff is possible, including using this foothold in your network to poke around and infect other computers on your LAN.

On the other tentacle, the keys to your castle are your inverter serial number, with that anyone can do anything. I had an SS tech poking at my system, making changes, doing stuff, and my inverter wasn’t doing what he expected. Then he read back the wrong serial number and we both realized he was poking at someone else’s inverter!

Personally I’m going to trust EG4/SS for now, but block Internet access once I get everything stable. Then I can use Solar Assistant (note: same problems as above) in local mode and my own monitoring code (completely different problems ).

Who to trust? No-one knows.

One general note here is that even though the hard-coded IP address is hosted by AWS in the USA, it appears to be a generic monitoring solution, possibly offered as a "white label" service similar to how the 18kPV appears very similar to a LuxPower LXP-LB-EU 12K.

In fact, notice how the EG4 web portal looks pretty much identical to this "generic" one:

• https://solarcloudsystem.com/WManage/web/login

And if you click "Visit demo station" there, it shows several LXP devices.

Thus, I'm inclined to treat any EG4 branded cloud services with caution.

 

[jsharkey]

Trying to follow your no encryption statement. Are you saying the API that our dongles send data to at EG4 AWS systems are not encrypted or tokenized? If so this is something that home network vlans, opensense, etc can fix. Do you have wireshark or some other tool evidence of it?

If that is indeed the case one could easily grab a device ID from the many screenshots here and flood the API with garbage data for specific user/device.

Yes, their "API" is simply a TCP keep-alive'd socket that shuttles raw MODBUS commands back/forth in cleartext across the Internet. The tests.py includes all the packet flavors I captured from a local tcpdump and decoded with Wireshark. (I redacted my serial numbers from the packets when writing the tests.)

I agree a VLAN or blanket firewall rule would work to secure the devices too, but that would mean you lose all remote monitoring. The deep packet inspection firewall rules here offer a balance between security while still enabling remote monitoring.

I'm also an avid Home Assistant user, but don't have the time at the moment to configure something like lxp-bridge. I can also imagine that it'd be easier to process any potential warranty claims if EG4 has all the historical data.

 

[Daddy Tanuki]

One more vote for not attaching things to the grid. I never put the dongle on the SA15k for internet. My IotaWatt and Solar Assistant sit on the LAN and are blocked at the router. I use a VPN on the router to hit the LAN for monitoring that stuff and my security cameras, which are also blocked. Ya do what ya can. I did IT work for a lotta years....if somebody wants you bad enough, they'll getcha!

Jim

just do not put your stuff online to begin with. all of my gear has the ability to go online i just do not connect them.

 

[wpns]

Yes, their "API" is simply a TCP keep-alive'd socket that shuttles raw MODBUS commands back/forth in cleartext across the Internet.

Grrrr, how can people still do this in this day and age? And how would you have answered if this wasn't a rhetorical question? I mean, c'mon EG4/SS, get some software professionals in there!

 

[Nobodybusiness]

I wish I understood 1/2 of what you are saying. Is there a way to protect the inverters without knowing how to write code? I am not familiar with writing code but would like to be able to protect my investment if possible.

Easiest way would be to get a Firewall.

Deny all incoming communications to everything.
Setup outbound only ports for monitoring.

Or

Just monitor it locally and not connect to internet.

 

[ksmithaz1]

Yes, their "API" is simply a TCP keep-alive'd socket that shuttles raw MODBUS commands back/forth in cleartext across the Internet. The tests.py includes all the packet flavors I captured from a local tcpdump and decoded with Wireshark. (I redacted my serial numbers from the packets when writing the tests.)

If you camp on port 8000 it spews modbus. However, the device establishes communication to a known endpoint. This would have to be spoofed or redirected or MIM'ed to do something nefarious. Not impossible, but unlikely. It's not like random actors on the internet can throw packets at your inverter. Again, if it's a concern, unplug it. They should really have done something with a key and client cert but whatever.

My concern is a screw-up at the server. All the encryption in the world is not going to help if some dufus that works for them types the wrong thing and blows up my inverter.

 

[Daddy Tanuki]

Grrrr, how can people still do this in this day and age? And how would you have answered if this wasn't a rhetorical question? I mean, c'mon EG4/SS, get some software professionals in there!

with all the boneheaded maneuvers that this company has done, this is just another minor blip as you can at least run it without plugging it into the net to begin with.

 

[TM48]

The problem really isn't EG4/SS it's Lux Power. If you look at EG4.s monitoring web site you see it's identical to Lux Power's except for colors. The security issues stem from Lux Power. I don't use the EG4 monitoring web site for the above reasons. Solar Assistant works for me in local only mode as it will phone home too if you let it.

 

[Quattrohead]

Do some port mapping in your firewall, I redirected my standard ports way up into the 50,000 area.
I started doing this hundreds of years ago in windows, redirected the RDP on 3389 to x3389 etc

 

[TechRider]

Port mapping, etc isn't really going to work if you're trying to use their monitoring. You really have 2 choices.
1. Use some sort of deep packet inspection rules like the OP says.
2. Disconnect the thing from the internet. (IE: Physically or firewall rules.

The fear for a man in the middle attack is real. Yes, nation states have better things to do than attack all of us one at a time. If there is a grid attack (false flag or otherwise) you can guarantee things like this will be targets as well.

 

[zanydroid]

How much can a cyber ransom gang get from owning an American prepper who is 90% dependent on their inverter? Couple $K per shot?

 

[42OhmsPA]

The problem really isn't EG4/SS it's Lux Power. If you look at EG4.s monitoring web site you see it's identical to Lux Power's except for colors. The security issues stem from Lux Power. I don't use the EG4 monitoring web site for the above reasons. Solar Assistant works for me in local only mode as it will phone home too if you let it.

Lux may be the root cause of the problem but EG4 chose to partner with them knowing, or not, that security was an issue.

 

[TM48]

Lux may be the root cause of the problem but EG4 chose to partner with them knowing, or not, that security was an issue.

Your point is pointless. The simple solution is don't use their web site. Not hard as there are other solutions. No cloud based service is secure.

 

[zanydroid]

There are cloud based services for varying levels of audit and compliance requirements -- plenty of different well-defined enterprise / public company / medical provider / government thresholds

You can choose to pick one of those or not, as a private individual... but I doubt most private individuals are more special than all of those.

In any case, this protocol is pretty terrible even by early 2000s standards.

 

[Quattrohead]

There are cloud based services for varying levels of audit and compliance requirements -- plenty of different well-defined enterprise / public company / medical provider / government thresholds

Name me any company that has not leaked data in the past 10 years. T-Mobile and Adobe don't count LOL