I’d be interested to hear your opinions / ideas on the following:
We have an MP2-3000, a Cerbo and a 30 kWh battery here as an ESS system. The primary objective is to store the electricity generated by the PV during the day.
It’s very tempting to use all the features such as Node-RED, Shelly and possibly even our own scripts. Everything works perfectly as long as there’s a LAN / WAN connection.
We’ve just had another two-day internet outage — the Victron system hung up.
Yes, it was my fault: I had Node-RED writing to a Volkszähler server on the web that was no longer reachable. At some point the watchdog kicked in, followed by constant rebooting, with old firmware even being installed in some cases.
I’m changing this now and will check the internet connection in the script, test the system and simulate an internet outage.
But is there any way, in the event of an error, to tell it to forget all the Node-RED scripts and so on and just act as a dumb inverter?
I’m not concerned about seamless switchover; there’s nothing important connected to AC-out, just an emergency power socket.
I’m already considering keeping a second Cerbo ready to swap in, with only the basic installation on it …
There’s nothing more frustrating than the power going out, thinking, “No problem, we have the battery,” only for it to fail because of the software …
After a power failure i found that i had no local monitoring because the intenet failed too.
Not on my side but on the DSL connection point. So i started to pump all my data points into Influxdb. I then visualize everything via Grafana.
Except for VRM very few of my systems have internet access at all. Every device I buy is
Selected by how it operates without cloud. Even AI inference servers are running locally.
the whole Victron setup works without internet access or cloud access, as long as you don’t add or program anything that requires a data connection to the outside world.
Shellys and the Android GX Display also run on the local LAN and don’t need the internet.
The Cerbo stores the log data temporarily and automatically uploads it to the VRM Portal when it comes back online.
If the router is powered on but has no internet connection, it should continue to work as before.
The alternative is an LTE stick in the router or something similar.
That’s the sort of thing it does, for example – but that’s not what this is about; it does work, after all, and makes sense in an ESS situation. Just not when running on backup power.
Even this setting is questionable and can cause problems if the internet goes down:
But in the event of a power cut, various Shellys of mine will no longer be reachable either.
We even have Starlink as a backup here, including a Ubiquiti Cloud Gateway that switches over automatically — nevertheless, after the outage, two ESPEasy/Tasmota devices had lost their settings. Perhaps they didn’t like something about the changed DNS; I’m still trying to work it out.
I’ve wondered the same thing, which is why I have everything either locally on the Wi-Fi or connected directly by cable.
In case the power goes out, the network infrastructure is connected to AC_Out1.
I think that should be enough, but I haven’t tested it yet…
I mainly use Node-RED for visualisation. A BKW might not be detected then, but that would only be a small part of my system, so it wouldn’t be a problem.
what do you mean? The logging? Yes, but that’s no comparison—it’s like a Polo versus a Rolls-Royce
yes, that’s why I’m seriously considering a second Cerbo—you can make everything work perfectly today, then tomorrow you make one careless change and end up setting a trap for yourself
I had problems with the watchdog, for example—so I quickly put together a Node-RED script to log the uptime—but quick fixes like that can cause massive problems when the internet goes down
I don’t want that, or rather, it can be switched over manually
After every reboot following a firmware update and so on, the power briefly goes out and everything restarts
The solution would be to add a small UPS, for which the battery could then be replaced every three years
It really is a European thing to have loads in front of the inverter. Here, with a poor grid, everything is a critical load. I stopped counting after 1000 grid losses, never lost an internet connection nor access.
If nodered code caused resources to run out and trigger the watchdog, then either the code needs optimisation or you need a more powerful device, or to split nodered to a dedicated host.
The remote console is enough for almost everything anyway
For critical systems, I always install a wired Victron display in the plant room to visualise the operating data and, where necessary, make configuration changes.
When nothing else works, that does…
It always depends on what you expect from it; for the vast majority, that’s perfectly adequate.
If you want more, you can also log all sorts of things with a local server, for example using Grafana.
Then you’re independent of the internet.
You don’t have to install every firmware update…
If the system is running, then it’s running, so you don’t need to install updates—or you might do it once a year.
Yeah, my requirements are pretty high there Volkszähler was around long before Victron — Victron needs to integrate with it too.
The local server — sure, you can do that — but what for? It also has nothing to do with the MP2 and an internet outage. I don’t want Victron Venus to crash just because the server is down either.
Absolutely — but I really don’t want to start disabling random NodeRED scripts and looking for the problems during a power or internet outage.
Venus doesn’t crash for an internet outage. It will reboot if you configure it to.
It’s your code causing the issue, not Victron.
There is a reason any modification or flow is unsupported.
Rather architect your environment better. It’s not difficult to do.
Your reply comes across as rather condescending and dismissive. You tell me that my code is the problem and that I should “architect the environment better,” without actually answering the question I asked.
Making the system fail-safe was exactly the purpose of my post. I even mentioned that I would be willing to invest in a second Cerbo GX if that were a sensible solution.
I am not claiming that Venus OS crashes simply because the internet connection fails. I am asking how the system should be designed so that it continues operating safely if the internet connection fails.
So instead of simply telling me that I am doing it wrong: What specific architecture would you recommend?
If you believe my current approach is flawed, please explain the concrete failure scenario and suggest a supported alternative. Otherwise, your reply sounds more like know-it-all criticism than an attempt to help.
Did you actually understand the purpose of my question?
It is simply not possible to trigger a watchdog without having customised the system, by your own admission.
This load is not factored into the GX performance.
Public nodes vary in quality, as does any flow. Blaming the manufacturer for your own choices seems a bit unreasonable.
Any dev cycle includes testing for failure scenarios.
To hit a watchdog you are consuming resources:
Have you tracked the load your nodered flows have placed on the GX?
Have you considered raising the watchdog.
I apologise if you find this condescending, but the reality is responsibility for this does not lay with Victon.
I have numerous flows on my own system and if I drop internet access it will reboot in 1 hour, as it has been configured to do so.
Instead of arguing with an unpopular view, maybe explain why it is a venus issue that custom flows and non-victron nodes, pushed cpu resources high enough to trigger a threshold set on 10’s of thousands of gx devices?
Of course I have customised the system. That is the entire point: I need it to meet my specific requirements. I have never denied that, nor have I blamed Victron for the consequences.
I have already acknowledged that my Node-RED flow caused the problem.
My question is how to design the system so that a failed flow or an unreachable external server cannot compromise the backup-power function.
Monitoring the load and reviewing the watchdog settings may be part of the solution, but simply raising the watchdog limits does not make the system fail-safe.
I am not asking who was responsible for the previous failure. I am asking how to prevent it from happening again.
Fair enough.
The easiest is to raise the watchdog threshold, there are a few topics on this.
You can’t easily identify which flow is the issue.
It takes trial and error.
Chart the cpu usage - use top or another method to monitor usage.
Disabling flows, particularly the internet requiring ones and gauge the reduction in resources.
Simulate an internet loss and see how this changes.
It unfortunately needs some legwork but it isn’t too much effort.
Apologies again if I came across as blunt, that was not the intention.
(An ekrano or the next gen cerbo has double the horsepower, that is a reasonable plan B if all else fails)
However, this is again mainly about diagnosing my previous failure. My intention is to have a more general discussion about fail-safe system design.
Raising the watchdog threshold is not a fail-safe strategy. It only allows an overloaded or unresponsive system to continue for longer.
The fundamental question is: How should optional functions such as Node-RED flows be separated or supervised so that their failure cannot affect the essential ESS and backup-power functions?
For example, should non-essential Node-RED tasks be moved to a separate device? Are resource limits, automatic flow shutdown, or another form of isolation possible?
I am looking for a general architecture that fails safely, not merely a way to identify the particular flow that caused my last crash.
