How to get internal state / debug information from Multiplus-II?

Question

question

kenzo asked Dec 7, '23

How to get internal state / debug information from Multiplus-II?

Unfortunately, this isn't the first time for the that the firmware doesn't behave as expected and needs some "enforcement" to do what it's supposed to; however, there don't seem to be good ways of debugging the state of the system when there's a problem (like enabling debug logs via MQTT).

The system is configured with a min SOC of 60% and hasn't seen a charge in some time (it's Winter over here and we had good amounts of snow):

Today is the first day in ~2 weeks with enough sun to charge the battery, which it didn't start until I did some convincing via ESS mode (going from "Optimized (without BatteryLife)" to "Keep batteries charged" and back which is the only thing I did).

The problem I'm facing seems to have started some time ago:

As you can see, the system indeed kept the battery at ~60% at a max until it started not doing that. While I noticed, I didn't pay much attention and waited for the sun to come back; I'm pretty sure it usually charged from the grid when SOC went below the threshold, however it didn't for the last two weeks.
As you can see, none of the settings limit charge:

As you can see in VRM, once there was surcharge coming from the roof, the system started feeding into the grid, with 0 mA going into the battery. Once I changed the configuration and had the battery charged from grid, I immediately reconfigured back to "Optimized (without BatteryLife)" which apparently made the MP-IIs change their internal state and continue charging the battery as expected:

I would love to support Victron in debugging an issue like this for the benefit of all customers since I'm pretty certain that this is indeed a bug (edit: unfortunately, strike-through isn't supported - I don't think this is true anymore); however, without getting more information about the actual state inside the fabric I am unable to - all settings in the remote console are fine and do not indicate a problem.

Is there any way of getting debug output from the system?
My last support request for a similar issue was closed since I'm using NodeRed which isn't "officially" supported, so I'm not even trying to go that route now ...
Thanks!

Multiplus-II charger

1701947249646.png (28.0 KiB)

1701947459420.png (32.3 KiB)

1701947633031.png (25.2 KiB)

1701947667120.png (18.2 KiB)

1701947772546.png (28.5 KiB)

1701947799470.png (20.3 KiB)

1701947857558.png (26.3 KiB)

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

kenzo commented · Dec 10, 2023 at 09:07 PM

Wondering if I got it completely wrong, and the MPs just needed a certain threshold to hit until they would charging (and I just happened to look at a time where that hadn't happened yet): Yesterday, it started charging when -600W went into the grid, so I'm not sure if there's a setting for exactly that threshold, however a debug log showing the internal state could make this available to the interested user as well ... :)

0 ·

1702242271677.png (56.3 KiB)

Modifications Space

Node Red Space

Answer 1 · 2023-12-07T11:29:14Z

kevgermany answered · Dec 07, 2023 at 11:29 AM

Have passed on your offer, thanks

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 2 · 2023-12-07T11:30:58Z

nickdb answered · Dec 07, 2023 at 11:30 AM

Your battery is requesting a 58.4V charge voltage but you have forced the system to limit it to 56.5V.

That is quite a difference.

Why are you limiting it?

What is the battery voltage at the time?

If it is 56V, then it is down to you using DVCC to prevent proper charging (this feature is not intended for this).

Try remove the DVCC charge limit and see if it resolves.

Is the remote console showing an ESS# code or what is the state under the inverter say?

What battery is this?

If you have made modifications (node red etc), disable them, go back to the basic setup and see if it all works.

7

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

kenzo commented · Dec 07, 2023 at 01:03 PM

> Why are you limiting it?

Because my Seplos BMS is not good at doing it, and I want to retain control of how for up cell voltage goes with more levels of freedom than the BMS gives me - I like to refrain from going up high in voltage for numerous reasons.

> What is the battery voltage at the time?

I appreciate you asking, see below - however, as you could see from the other graphs and my explanation above, the battery immediately started charging once I went from "Optimized (without BatteryLife)" to "Keep batteries charged" [and back], so I'm pretty confident this hasn't got to do with battery voltage. Otherwise, why is it still charging now?

> Is the remote console showing an ESS# code or what is the state under the inverter say?

Yes, it showed ESS code #1 (low SOC), state "bulk".

> If you have made modifications (node red etc), disable them, go back to the basic setup and see if it all works.

See above, I did not change a single configuration, just went from "Optimized (without BatteryLife)" to "Keep batteries charged" and back which made things work immediately.

0 ·

1701953831004.png (28.0 KiB)

nickdb ♦♦ kenzo commented · Dec 07, 2023 at 01:38 PM

To add. Your offset is almost 2V beneath CVL. If the BMS SOC accuracy is based on the higher voltage, chances are your SOC is wrong. Rather let it do what it is wanting to do and see if your issues resolve.

You would be better off reconfiguring the BMS parameters than forcing it lower on the GX, something that will only cause issues.

That parameter was designed for Pylontech (and later other batteries) to help with severe imbalances on new batteries. It was not intended to be used this way on an ongoing basis.

You can also disconnect the BMS altogether and rather program the chargers with the required voltage limits so it charges the old fashioned way.

1 ·

nickdb ♦♦ kenzo commented · Dec 07, 2023 at 01:21 PM

Keep charged adds a voltage offset and forces grid charging, so if Voltage or PV was a problem it would start to charge. With ESS #1 PV won't be used for loads, only to charge, so if the battery does not want to charge, extra PV is lost. With keep charged it will be allowed to cover loads so production will ramp up. At that SOC it won't use grid either to charge, only for loads. It would need to get to 63% before discharge and PV for loads is enabled again.

I would check your settings carefully and measure the voltages at the battery and chargers to make sure you don't have loss somewhere.

Right now it just appears your battery does not want to charge, or the chargers are at the charge voltage and are throttling.

I would still remove that DVCC setting to test.

In any case, with an unsupported battery and adjustments, you aren't going to get any official support. Maybe search the forum for Seplos, it does pop up from time to time, often with issues.

This is a managed battery, so the system will follow what it is told by the BMS or overridden by manual GX settings.

Is the Seplos configured as the battery monitor in the GX system settings?

0 ·

kenzo nickdb ♦♦ commented · Dec 07, 2023 at 01:42 PM

> so if the battery does not want to charge
What other value do you want to see that the BMS would use to tell the charger to it does not want to charge? I am pretty certain that I provided all possible ones.

> With keep charged it will be allowed to cover loads so production will ramp up.
This is an AC coupled system and there was no frequency shifting in place, so production wasn't limited.

> Right now it just appears your battery does not want to charge, or the chargers are at the charge voltage and are throttling.

Can you provide any reason based on a value that I provided or could get that makes you believe that?

> I would still remove that DVCC setting to test.

What am I supposed to test, though? The system started charging after I went from "Optimized (without BatteryLife)" to "Keep batteries charged" and back, and it's been in this condition without issues since.
That's exactly why I was asking about debug output, it might take a while for this condition to occur again (if ever), so replicating in a lab might be pretty difficult (except if they already have an idea based on code/explanation).

> This is a managed battery, so the system will follow what it is told by the BMS or overridden by manual GX settings.

I provided all settings in my original post, please explain what setting you believe may be at play here.

> Is the Seplos configured as the battery monitor in the GX system settings?

It is! As an example, the battery voltage recorded in VRM aligns perfectly between what the BMS measures and what's measured by the MP-IIs.

0 ·

nickdb ♦♦ kenzo commented · Dec 07, 2023 at 02:10 PM

What versions are you running?

You failed to mention what your PV configuration was, or indeed much about the system. It makes it difficult to help without all the information.

Failure to charge properly typically comes down to two things, battery issues or config/installation.

Perhaps post your inverter config file so we can have a look.

How much a battery charges depends on the voltage it is charged at.

If there is loss or it is being limited will have a direct bearing on whether it charges or not, and how much.

There is nothing available to you to help debug it further, short of using bms utilities to see the battery data at the time.

1 ·

nickdb ♦♦ kenzo commented · Dec 07, 2023 at 02:23 PM

It is not impossible that you have run into an interoperability issue.

Software keeps changing and testing is required to make sure everything works reliably and supports new capabilities.

Generally, for supported batteries issues are caught in beta testing, so broader problems tend to be found fairly quickly.

As you have an unsupported battery, it is the suppliers responsibility to ensure their product works well within the Victron ecosystem.

The alternative is showing the problem exists with certified configurations.

It is difficult to remotely try help when there is so little to go on apart from pointing out the obvious.

Good luck.

1 ·

kenzo nickdb ♦♦ commented · Dec 07, 2023 at 02:44 PM

Thanks for your responses; this system has been up and running in this setup for more than 12 months with no major issues, and has mostly worked in a deterministic way on a daily basis. All modifications done via NodeRed have mostly worked reliably.

The MP have charged and discharged the battery and provided energy to the loads fine, and continued doing so after resets when I've run into non-deterministic issues like this one.

None of the configurations have been changed when today's event occured, and they haven't changed when I applied the workaround I mentioned multiple times.

I feel that we have now reached the stage where we're guessing what might be happening without a real path towards a deterministic and reproducible way of debugging this.

> You failed to mention what your PV configuration was, or indeed much about the system. It makes it difficult to help without all the information.

What information specifically are you missing that would give you a hint why the system stopped discharging out of the blue?

> Failure to charge properly typically comes down to two things, battery issues or config/installation.

Again, no changes in config/installation, and I still miss what battery values you consider responsible that I haven't provided yet.

> How much a battery charges depends on the voltage it is charged at.

Do you not expect a battery sitting at 52.5V to be charged when the CVL is configured at 56.5V? If so, why?

> If there is loss or it is being limited will have a direct bearing on whether it charges or not, and how much.

How would it be limited besides the parameters I've shared?

> There is nothing available to you to help debug it further, short of using bms utilities to see the battery data at the time.

What battery data are you looking for?

> It is not impossible that you have run into an interoperability issue.

It certainly is not, but it's highly unlikely and improbable, given that the system has worked fine for months.

> The alternative is showing the problem exists with certified configurations.

I am not trying to convince people that a problem exists, I'm trying to debug a condition that seems unexpected in order to figure out how the system went into that state.

> It is difficult to remotely try help when there is so little to go on apart from pointing out the obvious.

Glad to provide all data that would be useful.

> Good luck.

Thanks!

0 ·

Answer 3 · 2023-12-07T17:23:48Z

gerard-van-seventer answered · Dec 07, 2023 at 05:23 PM

Could it be a low temperature protection of the bms/battery?

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

kenzo commented · Dec 07, 2023 at 07:42 PM

How would the BMS signal that beyond CVL/CCL? Why would the BMS stop signaling it when going from "Optimized (without BatteryLife)" to "Keep batteries charged" and back?

0 ·

Answer 4 · 2023-12-07T18:28:44Z

Alexandra answered · Dec 07, 2023 at 06:28 PM

A different thought.

You are producing 4k of solar but 900 ish watts to the battery. Kind of suggests a load is using the rest?

Do the batteries charge if there are no loads with solar?

In DVCC switch off BMS control (yes the mppt may need a reprogram) and see if the solar then hits programmed target voltages. (all components of the system programmed the same and to battery manufacturers recommended values). That way you still have cell I formation to the VRM etc but rule out the BMS control as a potential source of the problems.

Seplos are an untested and unsupported BMS so to say there is a bug is not accurate. Especially since the same firmware works with other batteries and BMS setups.

If the batteries overshoot because of balancing issues, slow the charge down in DVCC to allow the balance system to keep up.

The other thing since you have modded with node red, you may need a GX reset and rest up there.

2

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Alexandra ♦ commented · Dec 07, 2023 at 06:31 PM

Check storage for node red on the GX and the d bus round trip and see if there are issues there.

0 ·

kenzo Alexandra ♦ commented · Dec 07, 2023 at 07:55 PM

> You are producing 4k of solar but 900 ish watts to the battery. Kind of suggests a load is using the rest?

Correct, this is an AC coupled system with usual loads.

> Do the batteries charge if there are no loads with solar?

The batteries have always charged, with and without loads. I have never seen this behavior before.

> In DVCC switch off BMS control (yes the mppt may need a reprogram) and see if the solar then hits programmed target voltages.

As mentioned above, the problem disappeared when I went from "Optimized (without BatteryLife)" to "Keep batteries charged" and back, hoping this would take the MP out of the state they were in. The batteries were happily charged afterwards and continued doing so, so right now it's not in a failure state. I doubt it will be for the next few weeks.

> Seplos are an untested and unsupported BMS so to say there is a bug is not accurate. Especially since the same firmware works with other batteries and BMS setups.

I am not saying there is a bug, but I suspect it - in order to be sure, more debug output from the system would be needed. And don't forget, I have been using the same BMS with the same setup for more than a year now, and have never run into a situation like this.
I have been asking the same question over and over, but don't get a response to this unfortunately: What is a possible way that anyone can think of how the BMS / the battery can be at fault here when there's no CVL/CCL/DVCC setting hinting at it, and given that I was able to move the system out of that misbehaving state using the sequence I described, and within 2 seconds?
There also was no notification about voltages or any failure in the GX device.

> If the batteries overshoot because of balancing issues, slow the charge down in DVCC to allow the balance system to keep up.

The battery was at 52.5V voltage, so far from overshooting.

> The other thing since you have modded with node red, you may need a GX reset and rest up there.

Hard to tell, since the system has been out of the malfunctioning state since the sequence I mentioned above, and has been working flawlessly the rest of the day, like the ~400 days before that.

Thanks for joining me in thinking about this!

0 ·

Answer 5 · 2023-12-07T20:23:25Z

kenzo answered · Dec 07, 2023 at 08:23 PM

We can approach this another, more systematic way - what are possible definitive reasons for MP to not charge a battery when there's surcharge AC power available?

Expected:

battery reached maximum charge voltage / CVL | SOC is 100%
BMS set CCL to 0
possibly BMS sending a value for "modules blocking charge" >= number of modules(?)
possibly BMS sending an alarm/error (should appear in notifications)
Charge limit is configured to 0 in DVCC
ESS #6 set via NodeRed

Unexpected:

error conditions (likely to appear in notifications), making up a list:
- voltage sense errors
- communication to BMS lost
- issues internal to the MP

Anything I missed?

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 6 · 2023-12-10T21:12:50Z

kenzo answered · Dec 10, 2023 at 09:12 PM

Added https://community.victronenergy.com/questions/247601/how-to-help-debug-multiplus-ii-firmware-bugs.html?childToView=248465#comment-248465; still believe it would be awesome to have some way of debugging the system so we could stop guessing, and had a better way of being well-informed about what's going on.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

question