Cerbo GX CAN data drying up in Signal K

Question

question

Paul Reeve asked Apr 22, '22

Cerbo GX CAN data drying up in Signal K

I have a Cerbo GX running Signal K.

The connection (provider) interface in Signal K is set to the defaults: i.e. n2k-on-ve.can-socket using canboatjs on can0.

The system is mostly stable, but from time-to-time the CAN data stream in Signal K dries up. When this happens the Signal K dashboard reports 0 deltas per second against the usual 40 or so. Enabling logging on the CAN connection yields no useful information: the log just stops with no evidence of a causative event. Rebooting Signal K restores normal operation ... until the next time.

When the Signal K connection has died, CAN devices in Venus (like my fluid sensors) remain active, so the physical CAN interface and associated N2K data feed is obviously still available, but not being presented to / processed by Signal K.

Has anyone else noticed this issue? Any ideas on debugging from both the Venus OS and Signal K perspective would be helpful.

Paul

cerbo gx VE.Can SignalK

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Modifications Space

Node Red Space

Answer 1 · 2022-04-22T16:20:05Z

Stefanie (Victron Energy Staff) answered · Apr 22, 2022 at 04:20 PM

@Paul Reeve

What is the firmware version you're running?

I think having logging enabled from the data connection menu may not show anything useful. I tried this myself for other reasons with very poor result.

Did you try logging from one of the canboatjs filters in the server log menu? They usually give much better and meaningful results. Just be aware that some of them produce huge output ;-)
But with only 40 deltas/s you should be fine.

FYI: I'm moving the thread to the modification section. It's usually easier to get someone with the relevant knowledge to deal with your topic there.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 2 · 2022-04-23T06:37:16Z

Paul Reeve answered · Apr 23, 2022 at 06:37 AM

Thank you for your input @Stefanie .

I am running Firmware 2.80~11-large-20

I have enabled various debug tokens for canboatjs over the past few weeks, but never managed to have anything useful enabled at the time of a lock-up. I did catch one glitch with one of the output side canboatjs debug filters: canboatjs stopped producing debug output (maybe it died?) but there was nothing obviously amiss in the log up to that point.

I am fairly confident that the issue isn't a capacity one: as you say @Stefanie the 40 deltas/s load generated by my domestic bus isn't demanding. If I enable my navigation bus then I reach around 90 deltas/s but the rate of CAN dry-up remains constant.

I am thinking of cloning my input N2K data stream to NAS and implementing a simple watchdog in Signal K that will record when the CAN bus locks up. I should then be able to replay a snippet of the clone in Signal K and see if I can replicate the failure. I wonder if anyone has taken this approach before and with what success?

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Stefanie (Victron Energy Staff) ♦♦ commented · Apr 23, 2022 at 06:58 AM

Hi @Paul Reeve,

Two things you could check and update is firmware 2.82-large~30 and then make also sure to update the signalk-venus-plugin to latest version. There have been quite some changes recently.
Then see if it fix the problem.

If it's really coming from one of your N2K devices, it's rather hard to track down unless you have only a few devices connected to the bus, which would make it easy to disconnect them one by one and see if it makes a difference. But with a sporadic error this is much more difficult and probably lasts forever. A watchdog sounds like a very good idea.

FWIW, I've never experienced a lockup and it's the first time I hear about a problem there. On my installation there are up to 430 deltas/s with 360 paths. I once had a YD device with a faulty firmware which slowed down everything on the Signalk/Venus side, but it never locked up. Tracking down that issue was a nightmare.

0 ·

Answer 3 · 2022-04-23T08:30:50Z

Paul Reeve answered · Apr 23, 2022 at 08:30 AM

Thanks again @Stefanie

Before switching to the Cerbo GX I had a small industrial PC with an Actisense interface running the latest version of Signal K. The problem was evident on that system too and had been since my very first install of Signal K in 2018. Your FWIW is actually helpful in that it reinforces my belief that the problem is triggered by (possibly corrupt) data hitting canboatjs and not being properly disposed of.

I will try the update to latest versions (I've been avoiding it because of the many recent changes to Signal K that you allude to) and proceed with the watchdog.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

question