I have an ESS system which has developed an intermittent fault, which seems to be getting worse and I am hoping someone might have some insight before I start replacing components!
The system has been operating perfectly for 9 years:
Quattro 8000
CCGX firmware 3.55
Carlo Gavazzi EM24 energy meter on RS485 connected through USB adapter
Lithium battery with internal BMS
MPPT Charger (2.2kW)
Solar inverters on AC input, measured on L2 of EM24
A few weeks ago I checked the VRM and saw the system in “Passthrough” and that the grid meter was missing from the device list.
I checked the Meter, wiring and USB adapter and there was no sign of physical issues so i unplugged the USB adapter and reconnected, the meter was back on the display and the system returned to normal operation.
I put the problem down to an unknown glitch and moved on, but since then the problem has happened a few more times, and it seems to be happening more often. I “fix” it by doing a remote reboot on the CCGX (Easier than physically unplugging as I dont have to go to the plant room) which also “brings back” the meter.
Obviously this could be a faulty meter, faulty 485-USB adapter or a faulty CCGX (assuming the wiring is not at fault) and I can start replacing those from cheapest to most expensive but has anyone else had this problem and if so what was the solution?
@NickJenkey Welcome Nick. 8 years of operation would prompt me to first revisit/redo ALL data and power connections including those into the EM24, swap between the usb ports on the CCGX. Also, did you notice this after a firmware update or change in the system?
I have thoroughly checked the physical layer, all terminals are tight, there is no sign of corrosion on the terminals and the twisted pair cable is undamaged and checks for continuity. The first time it happened I swapped the USB adapter to the other USB port on the CCGX, where it has remained.
The system is set to auto update the firmware, so it might have been prompted by an update, I notice that this post Color Control GX automatic updates back to v3.55 (no longer v3.60) flags issues with the older hardware and 3.6 (but mine is on 3.55) just not sure when the update happened, do you know if there is a version update log file on the device?
It is possible that we will replace the CCGX with a new Cerbo soon anyway but for now loosing charging is annoying, I will set up an alert and log when it happens now so I can see the frequency.
You can roll back to the last version which may be worth trying. Below is from my CCGX having installed the latest via beta tests. It could just be that it struggles a bit with the new venus os versions. Disable auto updates to at least have a better control over the system changes.
Not done any changes or physical investigation yet but have noticed intermittant loss of data from the meter, I have posted a longer reply in the other thread here:
I wonder whether this may have something to do with your problem? “The Color Control GX (CCGX) and the CanVU GX are not able to run Venus OS Large.” You may just be pushing such an old device beyond its limits resuting in the occasional comms loss with e.g. the now heavier venus OS.
@ejrossouw Unfortunately not, I am not running the Venus OS large image on the CCGX it is just the standard version, I am aware of the device’s limitations. the instance of Node Red I am using is running on a completely separate system (Raspberry Pi 5) and I am just looking at the data values via MQTT for the grid metering as a way to indicate if there are glitches.
The initial problems predate any connection to the ESS system via node red, which I only did this afternoon as part of the troubleshooting process.
@ejrossouw I have rolled back the firmware to V3.5 this morning from an SD card and turned off automatic updates to the CCGX.. We shall see if this has any bearing on the situation, in the other thread I posted this image that shows that when the system drops into passthrough there is a loss of communication shown in VRM to the inverter as well which kinda implies that it is not the meter or RS485 converter etc as the inverter is connected via VEbus.
Fingers crossed it is as simple as that. Even on my Cerbo I found occaionally the time does not correctly update. In the absence of a clear pattern / triggers, best to monitor various parameters in vrm to try and establish something.
I installed V3.5 yesterday morning and if anything the issue seemed worse, with more frequent “events” yesterday evening I tried a factory reset with an SD card as described in the CCGX manual. After that and various faffing to get the system back visible on VRM etc there was still no improvement.
I set up logging of the “RTT Value” as suggested by Victron to monitor the load on the compute side and apart from the odd spike it sits below 30ms so it does not look like the device is overloaded with tasks.
I am truly stumped by this and can only assume it a hardware fault or a software/firmaware issue that has become embedded in the system somehow and is not being resolved by a factory reset.
So unless there is some breakthrough I am going to just replace the CCGX and RS485/USB converter this week, I will install a Cerbo as I rarely need the display and see what happens, I guess it could be something else but as there seems to be a loss of comms to the inverter in some way at the same time (both the Quattro and the Grid meter show long times since “last seen” on VRM when this happens but unplugging and re connecting the grid meter solves the issue, unplugging/re-plugging the VE.bus cable to the inverter does not.
To be fair the system has worked mostly perfectly for nearly 10 years, the battery monitor had to be replaced a few years back but that has been the only problem the CCGX does not owe anything really.
Thx for that. Last question, how up to date is the firmware on the Quattro and is it of the same age? One can spend more time (time=money) on something than it is worth and after 10 years of good service, a new Cerbo is a worthy upgrade.
As the ccgx ages particularly after the five year mark, working in warmer environments, the power supply struggles, usually the issue is first seen on the usbs. Intermittent ‘powercuts’ on the ports happen. I have a few i have swapped out to cerbos.
@ejrossouw The quattro firmware was updated a few years ago (its not the original from the first installation) currently V497 so not bang up to date but has been working fine. So I guess I could grab my MK3 and update it but the response from @lxonline seems to offer an explanation that fits with the observed and I am always suitably nervous flashing the inverter EEPROM in case something powers off and I get a blue brick!
@lxonline The environment the CCGX is living in is not particularly hot (North side of the building and unheated but dry) however at nearly 10 years old a gradually deteriorating power supply does seem a strong possibility, the Cerbo is not expensive so I will give it a go.
If you are willing to pay postage back and forth (UK address) I am happy to lend you a Cerbo or CCGX unit from my test bench. I personally have a unit going back to 2019 that survived ant infestations, heat (southern hemisphere coastal climate and it has not blinked once since installation.
That’s a really kind offer @ejrossouw thank you! I will probably go with a new unit as they are under £150 and given the time to configure and run tests its probably worth it, especially given the age of the existing unit. The current CCGX can then be used in another less critical application or given to a new home.
Thanks also for the link on the remote VE configure, it will definitely be easier than the traditional MK3 approach as I wont have to go to site, which while not far feels a lot further in the dark and rain like last night!
In other developments that suggest something bad in the CCGX, the MPPT charger connected on VE.direct has “vanished” and no amount of power cycling or unplugging of the direct cable makes it re-appear, the BMV700 continued to be visible.
I tried swapping the ports over (VE direct 1 and 2) and this did not help, however if either 1 of the devices were connected then they appeared in the device list, so it seems that the VE direct ports are failing too.
I noticed whilst in the back of the unit connecting and disconnecting the cables that with both of the VE direct cables plugged in the lights on the 485/USB converter were not flashing as normal, generally looking weak and dim, with only one VE direct port in use they picked up.
I am now fairly convinced that in this particular case the issues are due to some kind of creeping hardware failure, quite possibly the power supply issue flagged by @lxonline but whatever the underlying cause its time for the CCGX to be retired as soon as I can get a Cerbo.
Thanks to everyone who has responded to my question in this thread and others, It has been a bit of a real time commentary on a failing device, hopefully will be useful to others.
