Mqtt broker seems to disappear for hours... checking Dmesg I see the following

Modifications
, , ,
1

I’ve been attempting to track generator runtimes (remotely) via mqqt. I found I was missing some. Eventually, I came to run mosquito_sub directly on the Cerbo. It too “missed” data. I’m running 3.66 Large

Looking around a bit I found these in the Dmesg buffer. Any ideas?

[208478.479267] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[208478.486705] CPU: 0 UID: 0 PID: 1602 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[208478.496247] Tainted: [O]=OOT_MODULE
[208478.499860] Hardware name: Allwinner sun7i (A20) Family
[208478.505250] PC is at 0x2dd60
[208478.508309] LR is at 0x23218
[208478.511389] pc : [<0002dd60>] lr : [<00023218>] psr: 20000010
[208478.517892] sp : beabba78 ip : 000000e1 fp : 00000001
[208478.523237] r10: b6eca8b4 r9 : 00f1c1c8 r8 : 00000004
[208478.528691] r7 : 000573e4 r6 : 00000081 r5 : 00000081 r4 : 00f80d84
[208478.535559] r3 : beabbae8 r2 : 00000040 r1 : 00373935 r0 : 00f646d0
[208478.542241] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[208478.549779] Control: 10c5387d Table: 4239406a DAC: 00000055
[208478.555808] Call trace:
[246082.467258] sunxi-mmc 1c12000.mmc: data error, sending stop command
[246082.476478] brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -110
[246082.482810] brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
[246082.493584] sunxi-mmc 1c12000.mmc: data error, sending stop command
[398954.563420] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[398954.570560] CPU: 1 UID: 0 PID: 23347 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[398954.580013] Tainted: [O]=OOT_MODULE
[398954.583667] Hardware name: Allwinner sun7i (A20) Family
[398954.589002] PC is at 0x502f314c
[398954.592252] LR is at 0x2e2a8
[398954.595285] pc : [<502f314c>] lr : [<0002e2a8>] psr: 20000010
[398954.601659] sp : bec19ae8 ip : 00000001 fp : 00000001
[398954.607056] r10: b6ee38b4 r9 : 00000000 r8 : 00000000
[398954.612388] r7 : 00058780 r6 : 01ff0000 r5 : 01d96d84 r4 : 01d8a278
[398954.619059] r3 : 502f314c r2 : 00000000 r1 : 01d8a278 r0 : 00000000
[398954.625716] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[398954.633081] Control: 10c5387d Table: 430c806a DAC: 00000055
[398954.638926] Call trace:
[490313.803028] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[490313.812982] CPU: 0 UID: 0 PID: 13599 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[490313.822505] Tainted: [O]=OOT_MODULE
[490313.826242] Hardware name: Allwinner sun7i (A20) Family
[490313.831635] PC is at 0x2dd60
[490313.834770] LR is at 0x7
[490313.837487] pc : [<0002dd60>] lr : [<00000007>] psr: 20080010
[490313.843910] sp : bef7ca78 ip : 00000000 fp : 00000001
[490313.849303] r10: b6f128b4 r9 : 00000000 r8 : 00000004
[490313.854704] r7 : 000573e4 r6 : 00035a28 r5 : 00000081 r4 : 0052cd04
[490313.861365] r3 : bef7cae8 r2 : 004bd008 r1 : 00000069 r0 : 00000000
[490313.868064] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[490313.875461] Control: 10c5387d Table: 4297806a DAC: 00000055
[490313.881301] Call trace:
[701346.863369] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[701346.873475] CPU: 0 UID: 0 PID: 10228 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[701346.883117] Tainted: [O]=OOT_MODULE
[701346.886726] Hardware name: Allwinner sun7i (A20) Family
[701346.892138] PC is at 0x2e274
[701346.895371] LR is at 0x1
[701346.898120] pc : [<0002e274>] lr : [<00000001>] psr: 20010010
[701346.904651] sp : bedefae8 ip : 00000001 fp : 00000001
[701346.910246] r10: b6f308b4 r9 : 00000000 r8 : 00000000
[701346.915696] r7 : 00058780 r6 : 01ff0000 r5 : 756c6156 r4 : 00522d58
[701346.922476] r3 : 405dd0a3 r2 : 00000001 r1 : 00000000 r0 : 00000001
[701346.929285] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[701346.936911] Control: 10c5387d Table: 422d406a DAC: 00000055
[701346.943227] Call trace:
[756983.465567] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[756983.475373] CPU: 1 UID: 0 PID: 4723 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[756983.484837] Tainted: [O]=OOT_MODULE
[756983.488534] Hardware name: Allwinner sun7i (A20) Family
[756983.493879] PC is at 0x2e274
[756983.497001] LR is at 0x153b8
[756983.500013] pc : [<0002e274>] lr : [<000153b8>] psr: 20010010
[756983.506475] sp : beb38ae8 ip : 000000e1 fp : 00000001
[756983.511812] r10: b6eee8b4 r9 : 00000000 r8 : 00000000
[756983.517204] r7 : 00058780 r6 : 01ff0000 r5 : 6772656e r4 : 009d3a98
[756983.523850] r3 : 6374656b r2 : 00000000 r1 : beb38b00 r0 : 00009966
[756983.530572] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[756983.538008] Control: 10c5387d Table: 41dd406a DAC: 00000055
[756983.543906] Call trace:
[834145.760023] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[834145.770118] CPU: 0 UID: 0 PID: 27274 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[834145.779659] Tainted: [O]=OOT_MODULE
[834145.783350] Hardware name: Allwinner sun7i (A20) Family
[834145.788807] PC is at 0x2dd60
[834145.791929] LR is at 0x2dda8
[834145.794937] pc : [<0002dd60>] lr : [<0002dda8>] psr: 200b0010
[834145.801403] sp : bea67a78 ip : 000000e1 fp : 00000001
[834145.806856] r10: b6f7f8b4 r9 : 00dfa1c8 r8 : 00000004
[834145.812279] r7 : 000573e4 r6 : 00035a28 r5 : 00000081 r4 : 00e50324
[834145.819043] r3 : bea67ae8 r2 : 00000000 r1 : 00000049 r0 : 00000081
[834145.825774] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[834145.833241] Control: 10c5387d Table: 422f406a DAC: 00000055
[834145.839221] Call trace:
[922487.791722] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[922487.801509] CPU: 1 UID: 0 PID: 11699 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[922487.810893] Tainted: [O]=OOT_MODULE
[922487.814504] Hardware name: Allwinner sun7i (A20) Family
[922487.819834] PC is at 0x2e274
[922487.822870] LR is at 0x153b8
[922487.825880] pc : [<0002e274>] lr : [<000153b8>] psr: 20070010
[922487.832296] sp : bee8aae8 ip : 000000e1 fp : 00000001
[922487.837621] r10: b6f1b8b4 r9 : 00000000 r8 : 00000000
[922487.842993] r7 : 00058780 r6 : 01ff0000 r5 : 00000020 r4 : 0049b330
[922487.849622] r3 : 00000010 r2 : 00000000 r1 : bee8ab00 r0 : 00009966
[922487.856288] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[922487.863662] Control: 10c5387d Table: 4317c06a DAC: 00000055
[922487.869509] Call trace:
[1095466.947683] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[1095466.957628] CPU: 1 UID: 0 PID: 22559 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[1095466.967184] Tainted: [O]=OOT_MODULE
[1095466.970944] Hardware name: Allwinner sun7i (A20) Family
[1095466.976372] PC is at 0x2e274
[1095466.979460] LR is at 0x2de18
[1095466.982609] pc : [<0002e274>] lr : [<0002de18>] psr: 20070010
[1095466.989081] sp : beb58ae8 ip : b6fb8118 fp : 00000001
[1095466.994632] r10: b6f788b4 r9 : 00000000 r8 : 00000000
[1095467.000089] r7 : 00058780 r6 : 01ff0000 r5 : 000000a9 r4 : 00af4158
[1095467.006828] r3 : 00b06748 r2 : 0000001c r1 : 000000ef r0 : 00000000
[1095467.013585] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[1095467.021034] Control: 10c5387d Table: 4229006a DAC: 00000055
[1095467.026965] Call trace:
[1272795.034123] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[1272795.044248] CPU: 0 UID: 0 PID: 16401 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[1272795.053890] Tainted: [O]=OOT_MODULE
[1272795.057586] Hardware name: Allwinner sun7i (A20) Family
[1272795.063199] PC is at 0x77657460
[1272795.066635] LR is at 0x2e2a8
[1272795.069795] pc : [<77657460>] lr : [<0002e2a8>] psr: 20010030
[1272795.076267] sp : beab5ae8 ip : 000000e1 fp : 00000001
[1272795.081776] r10: b6e9c8b4 r9 : 00000000 r8 : 00000000
[1272795.087287] r7 : 00058780 r6 : 01ff0000 r5 : 0105165c r4 : 01039bc8
[1272795.094145] r3 : 77657461 r2 : 00000000 r1 : 01039bc8 r0 : 00000047
[1272795.100962] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA Thumb Segment user
[1272795.108561] Control: 10c5387d Table: 429ec06a DAC: 00000055
[1272795.114738] Call trace:
[1345604.017519] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[1345604.027742] CPU: 0 UID: 0 PID: 6996 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[1345604.037306] Tainted: [O]=OOT_MODULE
[1345604.041126] Hardware name: Allwinner sun7i (A20) Family
[1345604.046560] PC is at 0x0
[1345604.049377] LR is at 0x2e2a8
[1345604.052473] pc : [<00000000>] lr : [<0002e2a8>] psr: 20070030
[1345604.059026] sp : beb8dae8 ip : b6f3f118 fp : 00000001
[1345604.064461] r10: b6eff8b4 r9 : 00000000 r8 : 00000000
[1345604.070064] r7 : 00058780 r6 : 01ff0000 r5 : 01907694 r4 : 01911bd0
[1345604.076808] r3 : 00000001 r2 : 00000000 r1 : 01911bd0 r0 : 018fef00
[1345604.083631] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA Thumb Segment user
[1345604.091372] Control: 10c5387d Table: 41fbc06a DAC: 00000055
[1345604.097312] Call trace:
[1414742.808266] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[1414742.818376] CPU: 1 UID: 0 PID: 9293 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[1414742.827823] Tainted: [O]=OOT_MODULE
[1414742.831593] Hardware name: Allwinner sun7i (A20) Family
[1414742.837083] PC is at 0x0
[1414742.839879] LR is at 0x2e2a8
[1414742.842967] pc : [<00000000>] lr : [<0002e2a8>] psr: 20070030
[1414742.849477] sp : bea07ae8 ip : b6eec118 fp : 00000001
[1414742.854909] r10: b6eac8b4 r9 : 00000000 r8 : 00000000
[1414742.860389] r7 : 00058780 r6 : 01ff0000 r5 : 01c6e3ac r4 : 01c6d878
[1414742.867216] r3 : 00000001 r2 : 00000000 r1 : 01c6d878 r0 : 01c11aa0
[1414742.874124] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA Thumb Segment user
[1414742.881835] Control: 10c5387d Table: 41fbc06a DAC: 00000055
[1414742.887804] Call trace:
[1468128.002344] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[1468128.012484] CPU: 0 UID: 0 PID: 19367 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[1468128.022154] Tainted: [O]=OOT_MODULE
[1468128.025849] Hardware name: Allwinner sun7i (A20) Family
[1468128.031543] PC is at 0x2e274
[1468128.034650] LR is at 0x2de18
[1468128.037789] pc : [<0002e274>] lr : [<0002de18>] psr: 20070010
[1468128.044557] sp : bebefae8 ip : b6f9c118 fp : 00000001
[1468128.050130] r10: b6f5c8b4 r9 : 00000000 r8 : 00000000
[1468128.055581] r7 : 00058780 r6 : 01ff0000 r5 : 00000081 r4 : 010a6958
[1468128.062534] r3 : 0109d168 r2 : 0000001c r1 : 000000ef r0 : 00000000
[1468128.069447] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[1468128.077025] Control: 10c5387d Table: 4322406a DAC: 00000055
[1468128.083277] Call trace:
[1484198.012900] venus-platform: venus-platform: potentially unexpected fatal signal 6.
[1484198.023476] CPU: 1 UID: 0 PID: 868 Comm: venus-platform Tainted: G O 6.12.23-venus-5 #1
[1484198.033108] Tainted: [O]=OOT_MODULE
[1484198.036786] Hardware name: Allwinner sun7i (A20) Family
[1484198.042260] PC is at 0xb64ccd34
[1484198.045601] LR is at 0xb64ccd28
[1484198.048964] pc : lr : psr: 600f0010
[1484198.055448] sp : beaa4508 ip : b6471ec8 fp : 00000002
[1484198.060935] r10: 00004000 r9 : 00000000 r8 : 00000006
[1484198.066365] r7 : 0000010c r6 : b54fc920 r5 : 000e8288 r4 : 00000364
[1484198.073248] r3 : b614c380 r2 : 00000006 r1 : 00000364 r0 : 00000000
[1484198.080086] Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM Segment user
[1484198.087522] Control: 10c5387d Table: 42ad006a DAC: 00000055
[1484198.093517] Call trace:
[1484199.748078] slcan serial1-0 can1: failed to send close command ‘C\r’
[1484200.868079] slcan serial1-0 can1: failed to send bitrate command ‘C\rS6\r’
[1542632.432109] vecan-dbus: vecan-dbus: potentially unexpected fatal signal 11.
[1542632.439446] CPU: 1 UID: 0 PID: 31515 Comm: vecan-dbus Tainted: G O 6.12.23-venus-5 #1
[1542632.448988] Tainted: [O]=OOT_MODULE
[1542632.452681] Hardware name: Allwinner sun7i (A20) Family
[1542632.458237] PC is at 0x0
[1542632.461047] LR is at 0x2e2a8
[1542632.464135] pc : [<00000000>] lr : [<0002e2a8>] psr: 20070030
[1542632.470697] sp : be93aae8 ip : b6fb9118 fp : 00000001
[1542632.476127] r10: b6f798b4 r9 : 00000000 r8 : 00000000
[1542632.481624] r7 : 00058780 r6 : 01ff0000 r5 : 00c5d3b4 r4 : 00c6c0d0
[1542632.488445] r3 : 00000001 r2 : 00000000 r1 : 00c6c0d0 r0 : 00c5e8b8
[1542632.495184] Flags: nzCv IRQs on FIQs on Mode USER_32 ISA Thumb Segment user
[1542632.502822] Control: 10c5387d Table: 430c006a DAC: 00000055

2

Have you more info? (system - connected USB devices)

Do you use MQTT Explorer to see the topics

Can you run ps |grep mq

image
image829×146 4.56 KB

Errors /data/log/

3

Fred,

root@einstein:~# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux 6.12.23-venus-5 musb-hcd MUSB HDRC host driver
Bus 001 Device 002: ID 0bda:d723 Realtek 802.11n WLAN Adapter
Bus 002 Device 001: ID 1d6b:0002 Linux 6.12.23-venus-5 ehci_hcd EHCI Host Controller
Bus 003 Device 001: ID 1d6b:0001 Linux 6.12.23-venus-5 ohci_hcd Generic Platform OHCI controller
Bus 004 Device 001: ID 1d6b:0001 Linux 6.12.23-venus-5 ohci_hcd Generic Platform OHCI controller
Bus 005 Device 001: ID 1d6b:0002 Linux 6.12.23-venus-5 ehci_hcd EHCI Host Controller

MQTT explorer was used much earlier ( a few weeks ago, ) but, not during these recent events

root@einstein:~# ps | grep mq
1696 root 1764 S supervise mqtt-rpc
1698 root 1764 S supervise flashmq
1701 root 47000 S {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
1708 root 1908 S multilog t s25000 n4 /var/log/flashmq
1713 root 1776 S multilog t s25000 n4 /var/log/mqtt-rpc
1841 root 46076 S /usr/bin/flashmq
9436 root 2700 S grep mq

No recent errors in /data/log/*

I made some some changes this morning and will post in an update in just a few seconds

4

I’ve learned a lot and have made some changes since my original post.

In hopes of getting VicronEnergy involved, I updated to 3.67 Normal (from 3.66 Large). I’m now running this command:

mosquitto_sub -v -d -i cerbo -h 192.168.10.74 -p 1883 --keepalive 300 -t “N/c0619ab6680d/generator/0/TodayRuntime” -t “N/c0619ab6680d/system/0/Relay/0/State” -t “N/c0619ab6680d/heartbeat” -t “N/c0619ab6680d/system/0/Dc/Battery/Current” -t “N/c0619ab6680d/system/0/Dc/Battery/Voltage” -t N/c0619ab6680d/generator/0/TodayRuntimewasher -t “N/c0619ab6680d/solarcharger/279/Yield/Power” -F “%I %t %p”

On the cerbo itself.

I experienced a short (35-40 minute) gap just a couple hours after this 3.67 Normal upgrade

Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/heartbeat’, … (20 bytes))
2025-12-07T20:48:25+0000 N/c0619ab6680d/heartbeat {“value”:1765140505}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Dc/Battery/Current’, … (13 bytes))
2025-12-07T20:48:26+0000 N/c0619ab6680d/system/0/Dc/Battery/Current {“value”:0.0}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Dc/Battery/Voltage’, … (28 bytes))
2025-12-07T20:48:26+0000 N/c0619ab6680d/system/0/Dc/Battery/Voltage {“value”:12.449999809265137}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/solarcharger/279/Yield/Power’, … (14 bytes))
2025-12-07T20:48:26+0000 N/c0619ab6680d/solarcharger/279/Yield/Power {“value”:62.0}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/solarcharger/279/Yield/Power’, … (27 bytes))
2025-12-07T20:48:27+0000 N/c0619ab6680d/solarcharger/279/Yield/Power {“value”:63.88999938964844}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Dc/Battery/Current’, … (29 bytes))
2025-12-07T20:48:28+0000 N/c0619ab6680d/system/0/Dc/Battery/Current {“value”:-0.4000000059604645}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Dc/Battery/Voltage’, … (26 bytes))
2025-12-07T20:48:28+0000 N/c0619ab6680d/system/0/Dc/Battery/Voltage {“value”:12.4399995803833}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/solarcharger/279/Yield/Power’, … (14 bytes))
2025-12-07T20:48:28+0000 N/c0619ab6680d/solarcharger/279/Yield/Power {“value”:63.0}
Client cerbo sending PINGREQ
Client cerbo received PINGRESP
Client cerbo sending PINGREQ
Client cerbo received PINGRESP
Client cerbo sending PINGREQ
Client cerbo received PINGRESP
Client cerbo sending PINGREQ
Client cerbo received PINGRESP
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/solarcharger/279/Yield/Power’, … (14 bytes))
2025-12-07T21:09:15+0000 N/c0619ab6680d/solarcharger/279/Yield/Power {“value”:42.0}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/generator/0/TodayRuntime’, … (14 bytes))
2025-12-07T21:09:15+0000 N/c0619ab6680d/generator/0/TodayRuntime {“value”:6003}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Dc/Battery/Voltage’, … (28 bytes))
2025-12-07T21:09:15+0000 N/c0619ab6680d/system/0/Dc/Battery/Voltage {“value”:12.359999656677246}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Dc/Battery/Current’, … (28 bytes))
2025-12-07T21:09:15+0000 N/c0619ab6680d/system/0/Dc/Battery/Current {“value”:-1.600000023841858}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/system/0/Relay/0/State’, … (11 bytes))
2025-12-07T21:09:15+0000 N/c0619ab6680d/system/0/Relay/0/State {“value”:0}
Client cerbo received PUBLISH (d0, q0, r0, m0, ‘N/c0619ab6680d/heartbeat’, … (20 bytes))
2025-12-07T21:09:15+0000 N/c0619ab6680d/heartbeat {“value”:1765141755}

This occurred while the generator was running as evidenced by relay state == 0 shortly after we recovered.

Generator is set to automatic at <12.1V for 30 seconds. It’s a two-wire affair driving a 100A battery changer into 10 12V 103Ah Agm batteries.

I suspect a much more serious outage in the wee hours of the morning as that has been typical.

Looking forward to ideas/suggestions/changes/tests

  • Jim
5

One more thing to add. The Cerbo is alive during these “outages” as evidenced by the generator run time increasing

accurately (settings>devices>genset>runtime*>Daily>today’s date

6

Are you running LINUX KERNEL 6.12.23?