MultiPlus-II GX 48/5000/70-50 Rebooting the GX computer

Yesterday, my customer had a very unpleasant situation with the equipment, namely, with an outdoor air temperature of more than +31, and a weight of 2800 kW, the GX internal computer began to lose communication with the MPPT controller and periodically reboot, and this happened until the temperature dropped to +29! Node-RED is running on GX. According to all the manuals and the declared characteristics of the product, it should work under such modes, but in fact it turns out that this is not the case! Please clarify the situation, the customer is very angry and dissatisfied, but wants to replace all the equipment with a Chinese Deye!

The GX under heavy CPU load will reboot. This depends on how it has been used, with mods and flows able to cause this.

You can chart the VRM dbus roundtrip time advanced widget, it is a good guide of CPU load.

If this is the cause, then mods and flows need to be improved to reduce load.

We run in way warmer temps here without issue, so you will need to dig into that the GX is being asked to do.

A question for the developers, why add the Node-RED functionality if it cannot work properly on the GX built-in computer at ambient temperatures of +31 and above? This is not a normal situation! What should I do with the customer’s equipment that does not interest him without Node-RED functionality?

Nodered works perfectly, thousands in use. However what you choose to make nodered do and how, really matters.

The intention is to use Victron nodes to customise the system, but there are numerous other nodes that can be independently loaded and then used in integrations, the quality of which can vary.

Typically it is poor flow design that is the issue - doing too much inefficiently, too often.

Many of us have plenty of flows constantly running, using little in the way of resource, me included, and up to 37C.

For really heavy use cases there are more powerful variants of the GX as well as the option to run it separately on dedicated hardware.

The internal GX has half the RAM of external devices and limited comms. If someone intends to be a power user an external GX is a better choice than the internal unit.

It is also worth considering if additional drivers and modules are loaded in the base OS - another common issue.

Support statement

Node-RED and SignalK are stable and reliable features in Venus OS Large for GX devices. There are tens of thousands of systems running Node-RED or SignalK in the field. However, Victron does not provide support for systems which make use of these extended features to implement advanced solutions.

The specs you are quoting are also that of the inverter itself, whereas you are describing an issue with the internal GX controller which is a separate, but integrated device with its own limitations. A device that will restart on comms loss (if configured) or if the internal watchdog detects a CPU overload - something that is independent of temperature.

in my opinion - the watchdog settings are to low

- → Victron: why did you decide to have them that low?

My system - i confess some changes like dbus-serialbattery, a lot of NodeRED became unusable, many reboots

I changed the watchdog limits - it works perfectly now

But with each Update these settings are overwritten, system reboots in an unstable state, gets back to an older Firmware

Doing an Update to another Beta-Version - i have to stop the boot process immediately, change the watchdog-settings, otherwise the system will crash

The problem is not the to poor Hardware, but the watchdog settings

Easy & simple solution, recommend :wink:

I do not use third-party Node-RED libraries, the code is optimised for minimal use of processor resources! If some equipment with the declared characteristics is designed and installed, then it must correspond to these characteristics, and not by the fact it turns out that the equipment does not work properly! I am very disappointed with the approach to the business of victron engineers and managers, which essentially goes from a reliable brand to cheap Chinese disposable products!

Hello @Val

Accepting that presently there is a problem with this installation, shall we work towards finding the least disruptive solution for your customer.

The suggestions here have been that the resource use of nodeRED is a likely cause of the reboots.

Is it possible that you or the customer could temporarily disable the flows to test that theory?

If the intention is to find help, then you will need to provide details about your configuration and installation, at present there is little to go on - no logs, no charts (previously requested), no photos.

If the purpose is just to rant, then there is little this community can do with that - we aren’t a support site, nor is this an ask Victron space, so it’s just ranting at a group of volunteers and enthusiasts who try to help, which won’t encourage much engagement.

If you need formal support, the support link is in our guidelines.

These systems are successfully installed in confined areas and mobile and marine applications, so if you want to get to the bottom of the issue help us to help you and try leave the frustration at the door on the way in.

As already shared, the image of inverter temp derating has zero to do with the GX controller, it in no way applies to that specific device’s operational envelope.

When all nodes and functions are turned off, at high temperatures, the reboot stops, now the outside temperature has dropped to +27, I turned on the nodes and functions again, the reboot does not occur, so the conclusion, the GX computer reboots inside under load and relatively high ambient temperatures, and this generally does not correspond to the declared product specification! Yes, of course, I can screw in additional fans from the outside, but this is not normal for such a brand!

Description of the power supply switching function between the battery (battery, mode ⁠2⁠) and the mains (mode ⁠3⁠) based on telemetry, solar generation forecast, tariffs and safety. Node-RED context. There, data on voltage, SoC (charge level), battery temperature, PV (solar panel) generation, lock timers, manual mode status, and anti-spam metres are accumulated there. Telemetry processing

If the incoming message is not a control button, the gauges are updated:

Battery temperature (⁠batTemp⁠), charge level (⁠soc⁠), battery voltage (⁠volt⁠).

Network voltage (⁠gridVolt⁠), output power output (⁠outPower⁠).

Current power of solar panels (⁠pvPower⁠), daily generation forecast (⁠pvForecastKwh⁠) and network loss status (⁠gridLost⁠).

  1. Automatic decision-making algorithm

Dynamic battery discharge threshold (⁠LOW_SOC_THRESHOLD⁠):

Default: 40%.

If the forecast of solar generation is high (\ge 10 kWh): decreases to 35% (network savings).

If the forecast is low (\le 3 kWh): increased to 50% (reserve preservation).

If the SoC indicator is not available, the estimate is performed by voltage (over 50 V corresponds to ~60%, otherwise 45%).

Defence triggers (switching to the Network — mode ⁠3⁠):

Battery voltage drawdown: 48.2 V voltage is held for more than 15 seconds.

Battery critical temperature: \le 2^\circ\text{C} or \ge 45^\circ\text{C}.

Power overload: output power > 4000 W for more than 10 seconds. At the same time, the lock is turned on to return to the battery for at least 5 minutes (or until the load drops below 3200 W).

Low SoC: The charge level has fallen below the calculated dynamic threshold.

Night rate:

From 23:00 to 07:00, if the solar forecast is low (\le 3 kWh) and the battery charge < 75%, the system switches to the Network.

Network accident:

If the mains voltage exceeds the safe range (below 180 V or above 259 V), the system forcibly switches to the battery (mode ⁠2⁠).

Anti-spam Defence:

Mode change commands are sent only if more than 8 seconds have passed since the last shipment (except for forced reset in “Auto” mode).

  1. Formation of indication and status

An informative text string is formed with data on the state of the battery, network, load and generation.

The visual status of the Node-RED node with colour coding is set:

Green: normal operation from the battery or recovery.

Blue: battery operation or manual mode.

Yellow: network warning.

Orange: transition to the network due to a discharge or blackout.

Purple: active night tariff.

Red: critical events (emergency network voltage, battery sagg, overload, lack of network when the battery is discharged).

Outside ambient temperature is one thing, but the embedded GX module inside the MP2 is likely at a higher temperature.

My MP2 system has an external CerboGX. I specified this rather than the MP2GX variant for ease of replacement in case of future failure of either component. But it seems there are other benefits too - more memory and less exposure to problematical high temps.

In your place, and having an obligation to make the system more resilient for a customer , I’d be inclined to install an external device to either just run the nodeRED flows or to take the place of the internal GX.

Then when all is working, take the matter up with the manufacturer to inform their future product design.

Remember that operational specs are based on a stock, standard OS, unmodified system, installed to manufacturer specs.

It in no way considers the effects of customisations or the large image - the features of which are clearly documented as unsupported - so you cannot base expectations on the default specs if you do not have a vanilla installation.

Adding customisations increases resource usage and temperature and narrows the operational envelope.

From the above it appears to be fine unmodified.

In hot climates we have to upsize designs to allow for performance in hot weather - here we comfortably sit in the high thirties.

An internal GX to run the large image in this environment will struggle without additional cooling.

A better design decision would have been to use an external GX that is not only faster but more able to dissipate heat.

@Val what I suggest is to open a topic on our nodered category. There are plenty of people who can help look at the flows and the necessary debug to help optimise and get around this behaviour.

The watchdog, if it is being triggered, can be raised.

First, check the system logs in /var/log - it will contain entries for watchdog reboots if this is the cause.

Post this advanced widget, it is a barometer for overload - the GX by default is optimised to keep a vebus system running:

Via ssh you can also run “top” which will show your CPU utilisation.

If these metrics confirm load as the main factor, there is always some adjustments that can be made to flows to lower the load and restore stability.

(The forum allows you export the json from nodered and paste in with tags so it displays correctly)

An engineering error in the design of this equipment is that the GX computer is installed inside the inverter in the upper part close to the transformer that emits a lot of heat, a small CPU cooling radiator, the lack of active cooling of the computer and processor!

Isn’t it so that the internal GX module even has the stronger CPU ?

The new models are a quad core, but half the ram.

The ekrano and Nucleo are the fastest as far as I am aware.

Cerbo still dual-core, for now, but it would cool easier. Depends exactly what the issue is - temp vs horsepower or memory.