question

Simon Churchill avatar image
Simon Churchill asked

Color Control GX hanging, freezing

We are experiencing an intermittent case of a Color Control GX hanging, or freezing. The normal symptoms are the display looks live with an indication of AC input and battery charging etc. But in reality it has hung and showing a state that it was in when it hung.

So the first question is this common and can it be cured?

The firmware is latest and its connected to four BV700s, a Quattro and Ethernet. It is using MODBUS to communicate with PLC electrical variables and MODBUS via PLC HMI controls max AC input, i.e. 16A, 32A and 63A shore power.

There is a degree of worry that there might be an issue where the Quattro in conjunction with CCGX may be creating the hanging and thus reluctant to treat this as a pure CCGX issue. The reason for this is there have been some issues where Quattro has not always being doing exactly as expected and have experienced issues where after a hanging we can only get everything looking right after a number of resets of both Quattro and CCGX.

One of the most odd indications of something is not right is the MODBUS command to switch between 16A and 63A works fine and the CCGX shows the correct value on change. Thus that looks good. But when in VRM the Input Current Limit is showing 32A. Also I cannot connect to CCGX in Remote Console on Desk PC, Laptop or iPhone App, irrespective of what network I'm on. So something is not right.

Any thoughts welcome. Thank You

Simon

MultiPlus Quattro Inverter ChargerCCGX Color Control
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

2 Answers
JohnC avatar image
JohnC answered ·

Hi @Simon Churchill

Just a suspicion, you may be overloading it. The CCGX is one of the lesser powered GX units, and I believe hanging is one of the symptoms. Some explanation in this comparison (see Note 1).. https://www.victronenergy.com/live/venus-os:start

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Simon Churchill avatar image Simon Churchill commented ·
We only have 4 VE Direct devices where max is claimed to be 5. So would like to think not.
0 Likes 0 ·
JohnC avatar image JohnC ♦ Simon Churchill commented ·
@Simon Churchill

Yeh, but that's roughish and probably applicable to 'average' systems. Modbus gets a mention as adding to cpu load, and your system likely rates overaverage duty.

The literature on VRM Dashboard in Realtime also warns it increases cpu load, and with features getting added regularly these things creep up on us. I've seen on my own CCGX (with 2x VE.Direct devices) Dashboard throw a 'Too busy/wait' warning. Rare, but still can happen. Maybe you could use that feature of Dashboard to give a clue as to what's happening.

1 Like 1 ·
von-baron avatar image von-baron JohnC ♦ commented ·

Firstly, I don't know too much about the GX, but assume it runs GNU/Linux, most likely a Debian variety. If so, this operating system is very stable and it should not freeze, even though to you are a user it might appear so, it is likely to be still operating, just the program being run for Victron is having issues.

I would SSH into it and run top or preferably htop if that is installed and then you can see what is happening in real time with all the processes (applications), root and user.

I can not imaging at all why extra ports using Modbus would present a serious enough load for any of these systems to cause it any problem.

If I had to make a guess, the load would be under 10% of CPU time for multiple Modbus handling on serial connections and less for TCP on Ethernet.

Unless the underlying code is poorly written. An example of this is code that blocks, for instance if a communications request is made and the other end doe not reply, then the main system sits there waiting, usually until a time out, which could be around one second. And if this happens continually, then it might present a problem.

I am suspecting that one of the connected devices is not replying as a Modbus slave to the GX which I am assuming is the Modbus master, and thereby possibly halting communications to other devices. Quattro maybe ?

You mention a PLC and HMI, what type of PLC and what type of HMI is connected and on what media, serial or Ethernet ?

How many Modbus master are part of this system, could that be a problem ?

Could you have what is called a 'race condition' where data in one device is being read by another to be overwritten by yet another and they effect each other and so around it goes ?

Just some thoughts that came to me as a programmer :)

0 Likes 0 ·
Simon Churchill avatar image
Simon Churchill answered ·

This may be the case. But I have been asked to switch the CCGX out for other and see what happens, and send current unit back to supplier for inspection. I'm hesitant to order a new unit and charge client for new if the problem just returns. We have a sister vessel with same set up and they had similar issues, but I was not involved in the issue at that time due to vessel location. It got swapped out for new and as far as I'm aware problem has gone.

I have been made aware of a possible issue that led to a series of CCGX being replaced on warranty and both vessels units would have been ordered by us at the same time, so it may be the case that this one is suffering the same. Both CCGX units were ordered back in March 2018 so will not be under warranty and the problem has generally be ignored and a simple power cycle has been applied each time, by the crew.

The reason for posting this issue here is to get some feedback from others who might be familiar with this issue and have an opinion that will make it easier to justify the replacement and risk that problem does not go away. Hence the question about possible cases where VE Bus devices might have an influence on each other and cause a crash or freezing of each other.

Thus what I'm hoping for is someone who is familier with this issue to give a little support.

Thank you again

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

JohnC avatar image JohnC ♦ commented ·

@Simon Churchill

If you were to switch it out, consider replacing it with a Cerbo GX. This has the option of a Touch50 (or 70) screen, No buttons (a touchscreen), and even an optional bezel to make it fit the CCGX panel cutout. Much more power and connection options. In it's heyday the CCGX was great. Cerbo now the flagship, and such an upgrade might help the client see where the cost has been applied.

My money's still on the overload thing. Should fix that admirably.

0 Likes 0 ·
Simon Churchill avatar image Simon Churchill JohnC ♦ commented ·
I had thought about Cerbo. We have used before, but was not available when we specified these boats. The trouble is justification. I could suggest to the client that it maybe a good idea, but as I said earlier the other vessel is the same and not having this issue.
0 Likes 0 ·
JohnC avatar image JohnC ♦ Simon Churchill commented ·
I appreciate the justification thing. And I wish someone else would raise other possibilities, as 'overloading' is hard to quantify as the culprit.

Firmware up to date? Not just the CCGX (v2.92 the latest), but the other kit too. That vintage CCGX dates to about the time the earlier ones couldn't handle the later added remote (VRM) Quattro fw update feature (#1342 message). Something to add to the justification list if applicable.

0 Likes 0 ·