External grid meter dissappears frequently while charging causing Passthru mode

[robmeerwijk]

During charging my 3-phase MPII 5000 system switches frequently to Passthru mode. There seems to be a relation between SoC and occurances. This starts to happen more often when the SoC increases. It can happen as often as 30 times per charging cycle.

When debugging this issue I used Node-RED for alarming me. I found that it happens when the connection with my external Grid Meter is lost. This is a CG-EM340. The grid meter works fine otherwise.

So I made an alarm in Node-RED for a frozen power value of the grid meter. This disconnection also only happens during charging, and synchronizes with the Passthru mode perfectly: no disconnection other than during charging. All Passthru’s matched exactly with loss of the grid meter connection.

Therefore I am possitive that Passthru mode is caused by lost connection with the grid meter.

I also found an error message in Modbus/TCP matching the exact time of the last Passthru event and related to Modbus connection to the grid meter:

"Error processing function code 3, unit id 30, start address 2600, quantity 38, src ::ffff 192.168.178.20 Error finding service device type grid at device instance 30"

The grid meter is connected via an official Victron USB-modbus device (so why error message in modbus/tcp???)

There are no other instances with VRM id 30.

The problem started on earlier versions of VenusOS, I don´t exactly know which version. It was present in 3.50/3.52. Right now I am testing 3.60~11. Still there.

Has anyone seen this problem before? Is it a bug? Why only during charging?

[M_Lange (Matthias Lange - DE)]

You can try to reproduce that by setting the grid-setpoint to a high positive or negative value.

How long is the RS485 cable?
Is it installed parallel to the power cables?

[robmeerwijk]

@M_Lange

How long is the RS485 cable?

Not that long, more or less 2 meters. It is an original Victron USB-cable. I assume that it is a twisted pair cable.

Is it installed parallel to the power cables?

It is installed next to AC-power cables going to the mains supply over a distance of only 80cm’s. It is not close to DC-power cables. The Victron wiring guide has no topic on RS485

If the RS485 suffers from vincinity to AC-power, why only during charging? During discharging the current is in the same order of magnitude without any disconnection ever. The direction of the energy flow (positive or negative) can not be a factor, can it? Or when idle at a sunny day my 10kWp AC-out connected PV-panels are supplying to the grid even more power at higher currents via the same AC-cables.

The problem resembles Em24 disappears in device list and system goes in passthru mode In this case the grid meter is connected via modbus/TCP instead of USB.

It also resembles Quattro going to Passthrough when it should be Charging In this case the internal grid meter of the Quattro is used.

The resemblence could be a red herring of course.

[robmeerwijk]

You can try to reproduce that by setting the grid-setpoint to a high positive or negative value.

I could try tomorrow. I was under the impression however, that this setting is ignored in DESS trading mode? Isn´t that correct?

[robmeerwijk]

Hi Matthias,

I moved the rs485 cable away from the power cables. Due to grey weather and equal energy prices night and day there is not much going on in the battery. Yesterday however the battery was charged during 2 hours. No problems during charging. But yesterday the grid meter readings stalled 3 times in total for about 10 seconds. With the same error message.

I find this message really strange because the grid meter is not connected via TCP.

I also tried setting the grid setpoint to 15k. Same error.

[NickJenkey (Nick)]

Hi Rob,

I posted separately about this just now after an initial search did not bring up much, and now I find all the other posts!

In my case the possible issues with electrical noise on the RS485 cable is very unlikely as the cable is carefully routed, fully shielded and the system has been up and running without fault for years, the issue started in September this year.

So I am not sure if my recent issue is just hardware age but seeing others having very similar issues might imply it isn’t

I am considering replacing the USB to RS485 adapter (Cheapest fix first!) to see if it helps but am reluctant to replace anything more expensive like the meter or CCGX if the problem is something related to firmware or aliens

I hope that you resolved the problem

Nick

[robmeerwijk]

Hi Nick,

Thanks for your answer. It is not at all resolved. It seems a bit better every now and again. But the problem keeps coming back. Then solar stops, I have a nodered flow that stops charging the EV, battery charging/discharging stops. So no risk of blowing fuses. It happens a few times every day and normally it lasts for like a minute. So I decided not to be bothered by it too much.

It only happens on high usage from the net, never during grid feed in. The RS485 cable is official Victron and follows a path far from power cables. During stops the meter is still available. I can still read from the display.

I am convinced that it is software related. The grid meter is relatively new. There are lots of issues with grid meters. So I don´t buy a new one. Every beta I hope it gets resolved. It happens predominantly on high grid loads (charging EV + charging battery) and virtually never on grid feed-in. In the logs I can not find a cause. Just meter gone, restarting services. And then it pops up immediately. The fact that is is always reconnected in the next try tells me that the meter probably was there all the time and GX lost track for a software reason.

Can you post a link to your post on this issue? And did you change anything in your config that might have triggered this behaviour?

Rob

[NickJenkey (Nick)]

Hi Rob,

Thanks for the reply and I am sorry to hear that it is not resolved yet. My other post is here Grid meter intermittantly dissapearing from CCGX

There has been a suggestion to roll back the firmware on the CCGX in case that has had an impact after an update, the CCGX was on automatic updates so it might have happened after one of these. The version it is running is 3.55 I may try a roll back this weekend to an earler version and turn off the updates.

I have also made a node red flow to monitor this so that if the MQTT topic for grid power stops the data for more than 30 seconds it send me an email with a timestamp so I can log the occurrences, it literally just happened while I was setting up the flow which is the second time in 24 hours.

I really need to get this resolved as we use low cost electricity to charge the batteries overnight in the winter and most of the solar is AC coupled so it is an expensive fault in terms of electricity bills, and the failure seems to be “sticky” until there is a reboot or I unplug the RS485 interface rather than resolving spontaneously.

Will keep you informed of progress!

Nick

[robmeerwijk]

I can not go back to previous firmware. I am on 3.70 beta now. My system needs SMA solar support, introduced in 3.6x.

[NickJenkey (Nick)]

Ah I see, the rollback was a suggestion from @ejrossouw and I can roll back as my solar inverters (both SMA but older) are on the L2 of the EM24 meter.

I have an image for 3.5 which I will try this weekend, but I am sure there must be something else as V3.55 has been out for much longer than this problem has occurred. If I had not checked all the wiring so carefully I would assume a bad connection but I am also suspecting a software issue.

I will do more investigations over the weekend and let you know how it goes.

[robmeerwijk]

I have 5 SMA-solar inverters following sunspec, installed on AC-out. In case of a blackout cerbo needs to be in control of the solar production preventing overloads. That feature was introduced in 3.6x

[NickJenkey (Nick)]

That is quite the setup you have! we have 2 AC inverters on the AC in only, one sunny boy from 2009 and a sunny mini central from 2011 both still going and pre-dating the ESS system that was installed in 2016.

All well before there was specific integration so the EM24 does the metering. When the inverters fail we will look at replacement with something that integrates with victron and look at transferring to the output, currently backup power is from the MPPT system only.

[NickJenkey (Nick)]

A bit of an update with some data gathered:

@robmeerwijk there seems to be some similarity with the phenomenon you have observed that the grid meter data flow is lost for a few minutes at a time but recovers automatically.

The node red monitoring I have set up emails me if there has been no data received for 5 minutes, and since 2pm there have been several periods where the alert has been triggered: 14:13, 15:05, 15:27 and 17:20 so there is no discernable pattern it has been very dark and rainy here (Northern UK) so there has been little PV and the battery is in sustain mode having not been charged overnight last night as the system was in Passthrough.

I also looked at the VRM data from last night and it looks like the system entered passthrough mode when the scheduled charge began, there was a brief jump in DC Bus current and thus battery voltage at 01:30 (scheduled time for charge) but then stopped after 1 data point was collected:

Comparing the Grid input power over the last 7 days the charging has worked fine until last night:

The cursor is on the brief rise around 0130 last night but for the previous 6 nights the charge has happened normally.

Still no real idea what is happening but at least in this case there is an obvious trigger event, even if it is not consistent !

[robmeerwijk]

Nice! I check if the grid meter is present. If it switches to null I can capture the exact moment. And I send the info to my home assistant via mqtt.

[robmeerwijk]

Have you ever tried to look at the logs?

cat /data/log/dbus-cgwacs.ttyUSB0/ | tai64nlocal
cat /data/log/dbus-systemcalc-py/current | tai64nlocal
cat /data/log/serial-starter/current | tai64nlocal

[NickJenkey (Nick)]

Hi Rob,

Its interesting to see the flow you have used, it’s a better approach than my one (I did rather rush it this afternoon on a tea break!) I can try that approach for logging.

I will definitely look at the log files you suggest, Thank you I was not sure where they were kept on the CCGX and the loss of ther grid meter does not seem to trigger a “top level” error the device just vanishes from the list.

I will report back if I find anything interesting

[robmeerwijk]

I think you can configure a warning in vrm under ‘settings’ | ‘set alarm’. For device choose cerbo-gx and next step is parameter: “no grid meter alarm”

[NickJenkey (Nick)]

Brilliant, Thanks Rob, I should have had more of a look on VRM before baking my own version in Node red on the Raspberry Pi!

[NickJenkey (Nick)]

An update and some more information on this after a scheduled charge overnight:

• I moved the scheduled charge start to 0015 so as to be awake when it started to charge, which it did normally on VRM so I went to bed.

• This morning the system was in passthrough again I re-boother the CCGX remotely around 0850 and normal operation resumed

• Now watching it in near real time the system went back into passthrough at about 0930

• I looked at the device list in VRM and got the following:

  

So it looks like the problem is not just the RS485 communication from the meter as the inverter is “missing” too and of course is connected using a different cable and interface.

I tried a reboot and after that it was working again:

So I might try a firmware roll-back today and see if that helps, but the fault seems to be increasingly frequent and with no obvious trigger event, the scheduled charge at full power went off without incident last night and the second “outage” this morning was at a time with little solar, background baseload only and whilst I was looking at the VRM screen.

If the firmware roll-back has no effect I am now beginning to suspect that the CCGX is at fault, possibly it is just getting to the end of its life, obviously not an issue for you @robmeerwijk as you have much more recent controller.

updates to follow!

[robmeerwijk]

My cerbo is only 18 months in operation.