MQTT values periodically corrupted (2.9X and 3.00beta)

Question

question

jago asked Mar 6, '23

MQTT values periodically corrupted (2.9X and 3.00beta)

Hello - I am using a Multiplus GX with a pylontech battery stack.

I am collection the process data via the internal mqtt-broker from the gx unit - this values are used in different systems (influx, nodered, grafana, home assistant). My problem is, that the battery values (current, voltage, power) are sometimes corrupted (like a overflow). First I thought, that my data processing is doing "somthing wrong" - but I have broken it down to the internal broker and i can see (e.g. with mqtt-explorer) that the .json data from the multiplus seems to be wrong...

Attached you can see a screenshot of the described phenomenon.

It happens first with the 2.9X version of the venusOS - actually I have installed the 3.00-21 as beta to see if the behavior changes - it does not change.

I have although activated the VRM-Data (which was deactivated in the past) to see if I have the same strange data - no: the data in the VRM seems to be ok. I haven never ever get some warnings or alerts from the system, so I thing the internal values "inside" the multiplus are ok - but the mqtt data is definitely corrupted.

Perhaps an interesting bug - If anyone has an idea, what to do or to check: feel free to comment my question ;)

Multiplus-II Node-RED MQTT home assistant

vic-mp2-mqtterror-current-01.png (161.5 KiB)

3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae commented · Mar 10, 2023 at 05:38 PM

I only started with victron GX on v2.91 and I've never seen it different from this....these are float numbers/values, so why care...the subscriber can take care of rounding/marshalling these, if need be.

I've never had a problem using these values in my Node-red flows and calculations.

0 ·

jago hominidae commented · Mar 13, 2023 at 11:05 AM

thanks for your response - if you take a look at the screenshot (History data) you can see, that the reported payload is {"value": -1996.800... } - it is not a mater of rounding, these values are physically wrong... There has never ever been a current of 2000 Amps ;)

And: there is no alert or warning tripped, so in my opinion, internally the values are ok, but there is a bug in the reporting module publishing the mqtt data to the broker

0 ·

hominidae jago commented · Mar 13, 2023 at 01:33 PM

Ah, yes...didn't catch that on your screenshot.

That's a valid point, then.

However, how frequent are these "glitches" ?

I am on v2.92 on my CerboGX and have never seen such alternating/inkonsistent numbers.

I am logging these into my own local influxDB as they come in and again, no glitches yet since last 5 month uptime, when the system has been first installed.

0 ·

Modifications Space

Node Red Space

Answer 1 · 2023-03-13T17:55:49Z

jago answered · Mar 13, 2023 at 05:55 PM

thanks for your response - if you take a look at the screenshot (History data) you can see, that the reported payload is {"value": -1996.800... } - it is not a mater of rounding, these values are physically wrong... There has never ever been a current of 2000 Amps ;)

And: there is no alert or warning tripped, so in my opinion, internally the values are ok, but there is a bug in the reporting module publishing the mqtt data to the broker

49788-vic-mp2-mqtterror-current-01-detail.png (180.1 KiB)

3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae commented · Mar 14, 2023 at 08:14 AM

as alreaday said above, I did not see/catch these kind of off/odd-values in my system. I am logging Battery Amps locally in my influxDB and only see values that are inside the valid range of my system setup (running a Cerbox GX with v2.92)

How frequent are you seeing these glitches?

0 ·

jago hominidae commented · Mar 22, 2023 at 07:37 AM

I would guess one or two times a day...

I have checked the can-bus error counter: no lost packages.

I have removed the termination resistor - no change

0 ·

mp2gx-can-pylontech-noerror.png (30.9 KiB)

hominidae jago commented · Mar 22, 2023 at 11:57 AM

strange.

This is from my system, from over the last 7 days.

No glitches...all well within possible real value ranges.

0 ·

1679486226130.png (169.7 KiB)

Answer 2 · 2023-03-22T16:02:15Z

jago answered · Mar 22, 2023 at 04:02 PM

can you check the firmware version of the pylontech batteries?

I have added one new US2000C block to my system and probably this could be an issue...

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 3 · 2023-03-22T19:34:53Z

hominidae answered · Mar 22, 2023 at 07:34 PM

no, I can't...it is said, that in case of an upgrade/change, the newest module (highest S/N and therefore probably the newest FW) must be made master in the stack.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 4 · 2023-03-28T15:49:20Z

jago answered · Mar 28, 2023 at 03:49 PM

I have checked the firmware versions of the US2000C - every module is up to date (Jan 2023 and Sep 2022 - based on the hardware revision all modules have the latest possible versions: US_C_Nt_V1.7 and US_C_V2.8)

The system operates flawless - there are no errors (temp or over current) logged and the data logged to the VRM-Portal has no glitches - to me, this behavior can only be explained by an internal software bug of the GX module...

I am not alone ;)

I have found a post with similar "glitches" on pylontech batteries:

https://community.victronenergy.com/questions/54257/pylontech-und-ccgx.html?childToView=195792#answer-195792

But: read the content carefully: it is not a matter of linking the batteries with the correct cabling or having a poor CANbus quality - be aware, that the overall performance of the running system is OK: no errors, no faults, no current or temperature warnings, no logs - just wrong values communicated by the GX module (it does not matter if you use mqtt or modbus)...

I hope someone with connections to victron can take a look to this - or give us a feedback how to debug the internal data.

kind regards ;)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 5 · 2023-03-28T21:01:54Z

mvader (Victron Energy) answered · Mar 28, 2023 at 09:01 PM

Hi, just a note that I’m following this conversation. I am from Victron.

But I have no idea where the bug would be.

The code that takes data from our internal databus (D-Bus); and publishes that to MQTT is here:

https://github.com/victronenergy/dbus-mqtt">https://github.com/victronenergy/dbus-mqtt

Fairly simple code.

I don’t remember ever seen such weird values for DC current in VRM.

Is it always the current that has an issue?

Can you show screenshots of graphs that show how often it happens?

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 6 · 2023-03-29T12:01:28Z

jago answered · Mar 29, 2023 at 12:01 PM

Hi, I have checked my DBs and I was surprised, that my retention policy was set to 7days only on my influxV2 instance - all the data are "collected" by telegraf directly from the internal mqtt broker of the victronGX module. It looks like this:

as you can see: all the values coming from the topic "N/XXX/battery/512/Dc/0/..." seems to be temporarily corrupted - the green SoC seems to be ok, but:

.../Current ...Voltage .../Temperature .../Power

have the same glitches.

20230329-victronbroker-512-dc-7d-values.png (182.2 KiB)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 7 · 2023-03-29T12:06:46Z

jago answered · Mar 29, 2023 at 12:06 PM

I compared the values from .../Voltage and the CellMin and CellMax voltage values from the N/XXX/battery/512/System/MaxCellVoltage and ...MinCellVoltage and copied following screenshot:

The database of my home assistant server although stores the values from the GX-Broker directly:

thanks for keeping an eye on this issue, kind regards ;)

20230329-victronbroker-512-volt-7d-values.png (179.2 KiB)

20230329-haos-march2023-values.png (99.6 KiB)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 8 · 2024-03-26T08:48:27Z

Stefan Boldt answered · Mar 26 at 08:48 AM

I am running firmware 3.30 as large version and I have the same glitches all over the mqtt values. Here an example:

Value is reported as 1.083,3 kWh and suddenly it goes to 915,03 kWh and back to normal.

I have the suspicion that it has to do with restarting HA (not reboot). Almost everytime I restart HA, my Node-Red instance on the Cerbo (not the HA Addon) crashes and restarts again. Not sure if this two problems are somehow related.

1711442606816.png (43.4 KiB)

1711442658647.png (42.6 KiB)

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hominidae commented · Mar 28 at 03:09 PM

...are you running an mqtt-bridge of some sort between the GX and HA or another external broker?

Looks like a glitch with retained values, when the HA side restarts, probably?

0 ·

question