ESS 3 phase 3 with external control, one phase sometimes ignores setpoint

Question

question

jacola asked Sep 13, '20

ESS 3 phase 3 with external control, one phase sometimes ignores setpoint

I might repeat myself but i am scratching my head and the problem is not solved.
My ESS is 3 phase, with Multiplus-II 5000 on firmware 474, Venus on 2.60. I use external control.

Sometimes one of the Multiplus units on one phase apparently decides to ignore the setpoints and starts doing odd things .. with the result being a wild ride between feed-in and consumption.

The situation resolves after some time, but that can take an hour or so. The other multiplus'es follow the instructions without issue.

this is the problematic phase only, SP is setpoint.
The overview of all 3 phases for a longer interval is here:
you can see that normally the inverter follows the instructions well ..

Any advice appreciated.

Johannes

Multiplus-II ESS

screenshot-2020-09-13-at-125742.png (229.2 KiB)

screenshot-2020-09-13-at-124726.png (740.1 KiB)

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola commented · Sep 15, 2020 at 07:43 AM

here we are again, producing 2500W instead of charging 400 .. not funny .. this is only one phase, the others show similar behavior. out of the blue for me ..

0 ·

screenshot-2020-09-15-at-084032.png (166.0 KiB)

jacola answered · Jan 02, 2021 at 03:54 PM

Update, fresh in 2021.
Victron finally took this serious, and i had a very pleasant support and cooperation by one of their core engineers. We did some remote sessions and i received new firmware which to my knowledge now has also been released.
I learned a lot over the course of the last months about Victron, and am happy with my installation now. There will be at least four more multiplus added to it.

So this case can be considered closed. It needed a firmware fix.

Johannes

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

thomasw-1 commented · Jan 02, 2021 at 05:30 PM

Hi Johannes,
Please have a short look in GX (menu MultiPlus - device) what firmware version solved that issue.

Thanks
Thomas

0 ·

Modifications Space

Node Red Space

Answer 1 · 2020-09-14T08:41:43Z

Daniël Boekel (Victron Energy Staff) answered · Sep 14, 2020 at 08:41 AM

Hi @jacola

is it ok if we look into your system? if so, what is the VRM id or system name? (a link to the VRM page is also good)

10

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola commented · Sep 14, 2020 at 10:43 AM

sorry i answered as "Answer" .. 41839

since today (portugese time) about 10:30 the inverters do "what they want" again, despite giving them setpoints. green is actual power as reported by the inverter, yellow is setpoint. note 10:42 all 3 phases show strange behavior .. at this time now things are back to ok.

we have cloudy weather today with substantial short term changes in production ..

0 ·

screenshot-2020-09-14-at-113758.png (360.3 KiB)

jacola commented · Sep 16, 2020 at 06:03 PM

Hi Daniël, did you come to any conclusions ? The problem happens intermittent, and not always on all phases. For me it almost looks as if the inverters go "on their own". Interesting is that the situation eventually fixes itself. If i should log any other values let me know please ..

0 ·

Daniël Boekel (Victron Energy Staff) ♦♦ jacola commented · Sep 16, 2020 at 07:54 PM

Hi @jacola

As you know, we're looking at it, I'll answer you as soon as I know more.

0 ·

jacola Daniël Boekel (Victron Energy Staff) ♦♦ commented · Sep 28, 2020 at 03:21 PM

Hi Daniel, here another log of a longer problem happening today.
The trigger of the situation is probably the following:
phase 1 inverts, phase 2 and 3 invert. a pump goes on (remember: nothing connected to load side, all is grid assist) drawing 5.5kw (again, on grid side) at 14:13
So the setpoints change, obviously.

From my viewpoints ESS at this timestarts to go in a havoc cycle, producing unrelated to my setpoints and fortuntaely, today did not violate grid feed rules (it produces together with production and that can drive too much triggering a protector, putting us off grid, that happened before).
It recovered around 14:38 for no obvious reason, but fortunately :)

here the graph

let me know if i can provide any other help to find this.

johannes

0 ·

screenshot-2020-09-28-at-153356.png (369.2 KiB)

jacola Daniël Boekel (Victron Energy Staff) ♦♦ commented · Oct 05, 2020 at 10:16 AM

Hi Daniel,

to avoid any doubt about how i interface to ESS mode 3, (which i do via Modbus/TCP), i added a completly independent monitoring via MQTT.
A simple software which just captures the (actively) reported setpoint and power power per phase and throws it to the reporting platform every 5 seconds. here are the subscribes:

client:subscribe("N/xxxxxxxxx/vebus/261/Hub4/+/AcPowerSetpoint",2)

client:subscribe("N/xxxxxxxxx/vebus/261/Ac/ActiveIn/+/P",2)

client:subscribe("N/xxxxxxxxx/vebus/261/Ac/ActiveIn/+/S",2)

The result shows that the ESS itself has those crazy ideas about not following the setpoint. usually for about 20 mins before getting "sane" again by itself. Green is the setpoing .. you can see hat ESS does out of it .. especially bad are the situations where the result is a fast discharge of the battery by producing a lot (and if that happens while we have production we blow the feed-in limits).

This really needs to be solved.

Johannes

0 ·

screenshot-2020-10-05-at-110749.png (393.2 KiB)

Daniël Boekel (Victron Energy Staff) ♦♦ jacola commented · Oct 06, 2020 at 08:45 AM

Hi @jacola

You mention this happens when the battery is fully charged, could it be that there is a DC overvoltage that causes the inverter to feed-in to prevent this overvoltage?

0 ·

jacola Daniël Boekel (Victron Energy Staff) ♦♦ commented · Oct 06, 2020 at 09:35 AM

@Daniël Boekel (Victron Energy Staff)

Hi Daniël, that was in one event, the above one was with an SOC of 18%. It happens quite often (usually one or multiple times per day), and i haven't found any correlation with anything until now. happens over all battery charge states, and also happens both with the batteries being charged and discharged (as you can see from the various posted graphics).
Happy to provide you with grafana access so you can soom to any area of the graphs if that would be helpful, but i would expect you can get the same data via VRM too ?
Is there anything i can do to help with the diagnosis ? I was about to write software to detect the situation so i then could try to command the ESS to do something (if it still listens in that phase).

Johannes

0 ·

Pedro Lima commented · Sep 30, 2020 at 10:54 AM

situation is really bad now, Can you please check what's the problem?

0 ·

jacola Pedro Lima commented · Oct 01, 2020 at 10:49 PM

Hi Pedro, Victron, i might have an idea. Here is a completely unbearable situation, which starts when the battery is getting full. The ESS software starts to oscilllate between massive "production" while i have full sun power and excessive energy, driving my feed-in way above the allowance .. the only thing i could do is really stop sending any setpoints and waiting. As usual, graph shows communicated setpoint and the results.

I am going to set up another independent monitoring using MQTT (currently i read back the data via modbus/tcp, and verify also that the grid connection shows those results so they are real and not fake).

Johannes

0 ·

screenshot-2020-10-01-at-234355.png (751.0 KiB)

jacola jacola commented · Oct 09, 2020 at 04:10 PM

So there are definitely 2 problems ..
the one reported where ESS in mode 3 just goes for its own decisions and does not follow the setpoints, and then the case when the battery is full - instead of leaving it alone, ESS goes to heavy discharging, trying to cut me off the grid with attempted overcurrent :(

Here is another great graphic of ESS not following the setpoints .. the green line is the setpoints

i am pretty desparate to get this solved.
The reported setpoints and other values are taken from MQTT from the venus, and directly sent to a victoriametrics/grafana instance. Setpoints are provided via modbus/tcp. so they do arrive in venus otherwise they would not be visible on this chart.

0 ·

screenshot-2020-10-09-at-170402.png (253.6 KiB)

Answer 2 · 2020-09-14T10:36:48Z

jacola answered · Sep 14, 2020 at 10:36 AM

Hi Daniel, absoluteley !

https://vrm.victronenergy.com/installation/41839/dashboard ..

i guess 41839 is the number ;)

Feel free to play, and i am more than happy to play along if you need local help, i should stop sending setpoints or the like .. linux/modbus/control specialist here ;)

If you want to see actual setpoints and the resulting data, i have that all on a victoriametrics/grafana instance, i can send you a login for that.

Johannes

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 3 · 2020-09-29T09:29:06Z

Pedro Lima answered · Sep 29, 2020 at 09:29 AM

Dear Daniel,

My customer Johannes is having some really annoying problems with this situation. Voltage from the grid is sometimes quite on the limit, and this behavior makes it even worse, triggering some equipments off due to a out-of-limits AC voltage.

Im glad Johannes runs an industrial electronics company, so he knows bugs like this happen sometimes. But if he was a regular customer, that pays a lot of money for this inverters, batteries (he has around 40kWh of Pylontech's), and this happens it would be a big mess (and the customer would be right to complain or even maybe go to court over this).

Johannes was planning to upgrade the system further, adding more inverters, batteries, panels, etc and converting to an off-grid system, to avoid the grid issues, but Im pretty sure if this software reliability problems are not sorted very soon, he will think twice about going off-grid using Victron.

Let me know if we can help in any way.

Best regards

2

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola commented · Sep 29, 2020 at 04:20 PM

Hi Pedro the situation is getting really bad now.
As of today when the battery is 100% charged, the ESS starts to "oscillate" between inverting thousands of watts (into the grid which already has to accept the solar production) and charging, constantly. Even if i stop to send setpoints, this "out of the blue starting to invert" continues for a minute before the system finally goes to idle.
See below graph. Despite ESS mode 3, the system things i t can do what it wants and invert, causing grid feed-in problems .. the green line is the actual performance (below shaded line is inverting, above is charging). remote console shows 99%/100% charged battery.
I need the system also with a charged battery to balance phases, so i should be able to set one phase to inverting if necessary and others to charge .. but i would not expect the system to deliberately invert without command.

Johannes

0 ·

screenshot-2020-09-29-at-171632.png (679.5 KiB)

Pedro Lima jacola commented · Sep 30, 2020 at 10:54 AM

Can someone from Victron please reply?!?

We have no idea what the problem is, clearly its a software fault.

Can someone let us know if we can do something on our side?!?

This are expensive equipments, this lack of replies and lack of support should not happen!

0 ·

Answer 4 · 2020-09-30T13:21:15Z

PeterM answered · Sep 30, 2020 at 01:21 PM

What is the ESS setting for Multiphase regulation ? Is it single phase or all 3 phases ? What is the external device ?

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola commented · Oct 05, 2020 at 09:35 PM

i am controlling via modbus/tcp. the ESS is set to mode 3 (external control), so i should be able to command every phase separately by setpoint. and it mostly (almost always) works well, just "sometimes" the ESS goes havoc and does whatever it wants, ignoring the setpoints given by the external control (which is a little linux appliance, a barix barionet, connected with an ET340 and controlling loads also).

To be 200% sure there is no problem with that device, i have set up yet another system, using MQTT, to just monitor the ESS. This other "Monitoring only" sytem receives both the setpoints as well as the actual data from ESS via MQTT, push, and records those to a victoriametrics cloud instance with grafana frontend. So i consider that 100% neutral, independent, just monitoring.
It shows the same data .. the ESS eventually ignores the setpoints and just does what it wants. those "havoc" phases can take 10-20 minutes, and in that time, unfortunately, ESS might deliberately go into 4k "production" on each phase, which adds to the solar production and can (and does) blow my line due to (negative!) overload. It clearly violates the feed-in regulation.

Next i am going to try to stop this, by setting the ESS to "charge only" when that happens, let's see if it accepts this.

But i have to say i am quite frustrated with this .. an ESS which decides by itself to feed-in and empty the battery, despite other commands, is useless.
As all three multiplus show the same behavior at the same time interval (usually all three start to ignore the setpoints) i have to assume it's the ESS and not the multiplus, but hard for me to diagnose this further. Sincerely hoping for Victron staff to get this diagnosed and fixed. I was about to upgrade the system substantially ...

0 ·

Answer 5 · 2020-10-17T11:03:50Z

jacola answered · Oct 17, 2020 at 11:03 AM

well, not really an answer, just more information to those interested or which run into the same problems.

The situation continues to be UNRESOLVED. Last week my installer was here to update the Multiplus Inverters as that was advised by Victron.
The problem remains.

To recapitulate: every now and then the multiplus inverters (i have to assume it is them) do not follow the setpoints (ESS Mode 3) any more but do go "havoc", and in worst case they go to full power inverting/production, causing excess grid feed-in and being cut off the network. Sometimes they just go to "0". In both cases the situation resolves itself within about 20 minutes, if we are not thrown off the grid of course. There are NO grid faults reported, nothin in NS log.
Here is a graph of the latest occurence, yesterday. At 16:43 all three phases seem to get to that point where they dont charge and dont produce, not following the setoint. At 16.53 they go "havoc". i did a multiplus subsystem reset via remote console at 16:58 which brought them back to 0 but phase 2 and three continue with havoc mode.
That data is collected by listening to the setpoint and actual power values published via MQTT by Venus, so that should be what the system should see.

Anyone else has similar problems ?
Is there any solution to listen on the VE.Bus to see what is going on there ?
For me the problem MUST URGENTLY be solved, i might be forced by the power provider to shut the inverters down otherwise as they produce excess grid feed in eventually, beyond my contracted power limit (when added with the solar production).

Any advice ? Anyone with similar problems ? If you think you might have this too, talk to me i can help you to log like this into a cloud server.

Johannes

screenshot-2020-10-16-at-170449.png (270.8 KiB)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 6 · 2021-01-02T15:37:39Z

thomasw-1 answered · Jan 02, 2021 at 03:37 PM

@jacola,

Today I performed some tests on my 3-phase system in mode 3 and the externally definable setpoints.

In principle the user definable setpoints work fine on my side with 3 exceptions:

A setting higher than 1.600W per phase is simply ignored, although battery could be charged with 4-times higher current at that point of time and MultiPlus-II inverters should support up to 3.500W (70A) charging power per phase.
In certain currently non-reproducable cases the submitted setting gets simply ignored. I have to re-submit the setting up to 4-times until it gets effective.
Interestingly the setting gets effective on 2 phases, where the remaining phase does anything else but does not align to the requested value.
Sometimes the defined power is aligned within a second only, but sometimes it takes up to 10 seconds to reach the defined limit.

3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jacola commented · Jan 02, 2021 at 03:50 PM

Hi Thomas,
since about 10 days, the problem is finally solved for me. Victron finally got serious on the problem, and they really have good engineers i have to say. Did remote sessions with the responsible engineer, he identified a reason, provided (just before christmas) a new firmware for the multiplus and the problem so far hasn't shown up again.
Which makes me quite happy. I have learned a lot on the way, and will add more victron equipment now to my setup (at least four multiplus 5000). So i definitely recommend to upgrade your Multplusses.
In my setup, i had both venus and cerbo used over the last months, setpoints sent via both modbus/tcp and mqtt, and that all works flawless (i sniffed on the VE.bus to ensure the setpoints really make it to the inverters).

Cheers

Johannes

0 ·

thomasw-1 jacola commented · Jan 02, 2021 at 03:56 PM

Great news, Johannes!
What firmware has solved your issue?
I have currently 477 running on my MultiPlus-II (2623)
The new release 478 does not refer to a fic related on mode 3.

Cheers,
Thomas

0 ·

1609604445194.png (5.5 KiB)

ondra jacola commented · Jan 06, 2021 at 08:36 AM

Hi Johanes, please, what version of FW is this, is it 478, as Thomas writes above, or some individual FW for your installation? I have a similar problem at one of the power plants. Thank you Ondra

0 ·

Answer 7 · 2021-01-24T12:04:46Z

thomasw-1 answered · Jan 24, 2021 at 12:04 PM

Hi Johannes,

@Ondra and I would highly appreciate, if you could invest 20 seconds in looking on your GX what firmware version solved your problem.

The community has the purpose to multiply solutions and not to keep it only for oneself.

Cheers,
Thomas

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 8 · 2021-03-19T07:15:28Z

gnagflow answered · Mar 19, 2021 at 07:15 AM

Hi Thomas,

i didn't read all your post, but maybe it helps, maybe in your system the max. inverterpower (see manual 2.d) is not set to "1" = no limit - that it doesnt take the setpoint > 1600W

kind regards.

1616138028682.png (41.0 KiB)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

question