Cerbo GX complete network failure

hello,

why did both network connections fail here? After a restart, everything worked again, so it can’t be a network or hardware problem!

I have now updated from 3.72 to 3.79 and hope that this has fixed the problem.

Otherwise, to be on the safe side, I would have to enable restarting on loss of connection to the VRM. I only really use that on the Venus-GX, which is normally connected via Wi-Fi, and there were problems reconnecting when the base was restarted.

cerbo-1.zip (4.7 KB)

bye

The attached log does not look like two independent hardware/network failures. The common trigger appears to be the network stack:

  • 12:14:04 and 12:14:41: brcmf_psm_watchdog_notify: the PSM watchdog was triggered (Wi-Fi driver)
  • the online check then fails
  • connmand subsequently reports: service ferdinand stuck in failed state, restarting
  • when connmand restarts, both wifi0 and eth0 are briefly taken down and reinitialised

The fact that Ethernet also disappears is therefore more likely a consequence of connmand restarting than a simultaneous fault affecting both interfaces. Updating from 3.72 to 3.79 is a sensible first step. I would only enable the automatic restart when VRM is lost as a fallback at most; it will not fix the underlying cause.

If it happens again with 3.79: save the logs again immediately afterwards and attach them here or send them to Victron Support, including exactly these messages and timestamps. That will make it possible to check whether the Wi-Fi watchdog is still triggering the connmand restart.

hello,

I suspect there’s definitely a software error here, and that the LAN went down first, then it tried to use the WLAN, which also didn’t work because of what may have been the same bug, and eventually it gave up and waited until I unplugged it.

This isn’t the first time something like this has happened. There has been a bug in the firmware since version 3.70 that disables the LAN interface, meaning you have to restart it. The fact that the WLAN was also affected this time is new, though.

The trigger for the LAN outage was apparently some packet that a Fritz!Box sent into the network. Two Fritz!Boxes in combination caused a complete network outage, or made the network unusable. One box: no problem. Two boxes: no functioning network anymore. They’re all connected to an isolated VLAN now, which isn’t causing any problems either. The cause of the problem with the Fritz!Boxes could have been one or more of the new Shellies. Some things just don’t get along on a network!

The only thing that helps is quarantine and not reconfiguring anything, which is the popular recommendation from the support hotlines!

In any case, everything is working at the moment. I’ve also installed monitoring now, but I haven’t tested it definitively yet.

bye