Cerbo GX MK3: system reboots when opening or updating Signal K apps while Node-RED is running

[Slapperia]

I just got a Cerbo GX MK3 to replace my MK2, which wasn’t powerful enough to run Node-RED and Signal K at the same time.

Setup:

• Venus OS Large image, updated to v3.80
• Multiplus, shunt, XS and smartSolar connected
• Enabled all tank, digital and temperature inputs
• Enabled NMEA 2000 and imported my Node-RED flows (identical to the MK2)
• Enabled Signal K

Problem:
Signal K runs normally, but the whole system restarts as soon as I open an app in the Signal K admin UI or try to update one.

Node-RED runs fine and I see no issues even if Signak-K server is running.

If I disable Node-RED, I can open and update Signal K apps without any problem.

Is anyone else running Node-RED and Signal K together on the MK3?

Does it work for you, or do you see the same behavior? Any suggestions on what to do are welcome

[nickdb (Nick)]

Unfortunately this is a known issue with the mk3, a side effect of memory pressure caused by the compiler when installing apps and the 64-bit OS.
You can verify the issue by ssh’ing to the cerbo, then grep /var/log/messages for “memory”, you should find the watchdog triggering a reboot due to memory exhaustion.

There is no solution currently, the workaround is to disable things you don’t need (nodered for example).
You can try disable non-critical services temporarily to load apps:

svc -d /service/start-gui
svc -d /service/dbus-shelly
svc -d /service/dbus-mqtt-integrations

To properly stop it happening, enable a 512MB swap file (this has no effect on performance and the flash drive is large enough).
This change will be lost on OS updates and is only to be used if you see this memory related reboot.

SSH access is needed.

dd if=/dev/zero of=/data/swapvol bs=1M count=512
chmod 600 /data/swapvol
mkswap /data/swapvol
swapon /data/swapvol

The above creates the swap volume and mounts it.
Now set swap priority:

echo 90 > /proc/sys/vm/swappiness

Now add the entries to the system config files so they persist a reboot:
Using vi or nano add the following to /etc/fstab:

/data/swapvol none swap defaults 0 0

Add to /etc/sysctl.conf
vm.swappiness=90

Reboot to verify the changes are in effect.
cat /proc/sys/vm/swappiness
cat /etc/fstab
swapon -s

You can check swap/memory in use with:
free -h mem

This is a temporary workaround, only use if absolutely necessary.

[Slapperia]

Thanks Nick, that confirms what I’m seeing (Venus OS Large v3.80, Node-RED + Signal K running).I bought the MK3 specifically to run Node-RED and Signal K together, so I’m disappointed to hear this is a known issue.

@Victron team: is this being worked on, and is a fix planned for a future Venus OS release? Thanks!

[nickdb (Nick)]

It is being worked on, by known, I mean I know about it as an early mk3 tester and having discovered the issue and working with the team to resolve. It is not something that is guaranteed to be found, and depends on individual apps. Yes, it is being actively worked on.

[Slapperia]

Thank you. This was very helpful information.

[nickdb (Nick)]

Hi, this should be addressed in the upcoming 3.81 and subsequent betas of 3.90.

[AukedeG (A de G)]

Subject: Cerbo GX MK3 (v3.90-beta5): watchdog restarts with ±150 MB free memory – min-memory = 2500 pages appears to be calculated using 64 kB pages

Installation

• Cerbo GX MK3 (hostname am62xx, 4 cores, 957936 kB RAM, no swap)
• Venus OS v3.90-beta5 (Large), migrated from a Cerbo GX MK2 on 29-09-2026
• Node-RED with a fairly large flow (±1880 nodes), 3x Multi RS on VE.Can, 2x JK-BMS (serialbattery), EVCS NS, MQTT integration in use

Observation
Since migrating from the Cerbo GX MK2 to the MK3, the GX restarts about once a day. Every restart in /data/log/messages* is a memory shutdown by the watchdog (times in UTC):

With help from Claude.ai, I collected the following data:

1 Oct 12:02:28 memory available 141420 kB is less than 2500 pages
2 Oct 03:42:39 memory available 145580 kB is less than 2500 pages
2 Oct 11:05:50 memory available 139788 kB is less than 2500 pages
3 Oct 17:35:03 memory available 157080 kB is less than 2500 pages
4 Oct 10:49:05 memory available 151368 kB is less than 2500 pages
5 Oct 06:48:25 memory available 150040 kB is less than 2500 pages
(each followed by: shutting down the system because of error 12 = ‘Cannot allocate memory’)

/etc/watchdog.conf:
watchdog-device = /dev/watchdog
log-dir = /var/volatile/log/watchdog
min-memory = 2500
max-load-5 = 10
max-load-15 = 6
repair-binary = /usr/sbin/store_watchdog_error.sh
test-prescaler = 60
retry-timeout = 0

getconf PAGESIZE returns 4096.

Analysis
With 4 kB pages, min-memory = 2500 would amount to ±10 MB. In practice, shutdown occurs with 140–157 MB of free memory. That corresponds to 2500 x 64 kB = 160000 kB (±156 MB). My suspicion is that the watchdog converts min-memory using the architecture’s fixed page size (64 kB on arm64, EXEC_PAGESIZE) rather than the actual page size (4 kB). On the MK2 (32-bit ARM), the same setting effectively means ±10 MB. That would explain why these memory-related restarts only began after switching to the MK3.

On a 1 GB device, this leaves very little usable headroom: under normal use, MemAvailable is 150–200 MB. So the entire GX (ESS, BMS, EV charger control) is restarted while there is still ±150 MB free. With retry-timeout = 0, there is no second check.

Largest processes (VmRSS): node-red 260–283 MB, venus-gui-v2 72 MB, dbus_shelly.py 53 MB, vrmlogger.py 47 MB, dbus-mqtt-integrations 45 MB, mqtt-rpc.py 40 MB.

Temporary measures

• Stopped dbus-shelly with svc -d (no Shelly devices here; there is no option in the menu to disable the integration): ±45 MB freed
• Limited the Node-RED heap with NODE_OPTIONS=–max-old-space-size=256 (the default limit was 516 MB)
  Available memory is now ±190 MB, only about ±35 MB above the effective watchdog threshold.

Questions

1. Is a min-memory threshold of ±156 MB intended on the MK3 / am62x, or should min-memory be converted using the actual page size of 4 kB?
2. Is there a supported way to adjust the watchdog’s memory threshold that will survive a firmware update?
3. Is there a supported way to disable the Shelly integration when there are no Shelly devices?

I’m happy to provide more logs or test a solution if that would be helpful.

[nickdb (Nick)]

Known issue, always pays to first try search for existing topics.

[nickdb (Nick)]

3.81 is released, it contains the fix.