Quattro 48/10000 failure during firmware update / VRM rescan — only DC connected, no AC input/output

[andrewd (Andrew Doroshenko)]

Hello everyone,

I would appreciate your advice regarding an incident with one of my Quattro 48/10000 units.

My main working system is currently based on a Multi RS 48/6000. Later I upgraded the installation and added three Quattro 48/10000 units, intended to be configured as a three-phase VE.Bus system.

The three Quattros are connected together through VE.Bus in a daisy-chain arrangement and connected to the Cerbo GX through a separate MK3-USB interface, not through the built-in VE.Bus port of the Cerbo GX.

All inverters are connected to the DC bus through a Lynx / power distribution system with fuses. After the DC input section I have a custom pre-charge module with a switch and a pre-charge push button. Before turning on the main rotary DC switch, I hold the pre-charge button for about 30 seconds.

This is my second high-power Victron installation. Previously I installed several single-phase Victron systems and also one three-phase system based on MultiPlus-II 5000 units, which I still maintain and regularly update. On that three-phase MultiPlus-II system I usually update the firmware of all inverters together, not one inverter at a time as I did in this case.

About one year ago I made the main DC, AC wiring and VE.Bus connections for these Quattros. I do not clearly remember whether I had already configured them as a three-phase system at that time. However, since the inverters were able to start individually one by one, most likely I had not yet configured them as a three-phase system.

Each inverter had previously been tested individually in single-phase mode and worked correctly.

The AC wiring was prepared, but no AC input source and no AC output load were connected during this incident. The AC input neutrals are connected together separately, the AC output neutrals are connected together separately, and the PE / grounding conductors are also connected separately.

Then the project was paused for about one year. During this time all three Quattro units, including the second Quattro that failed today, were switched off using their front rocker switches. However, they remained connected to the DC bus and therefore were still under DC voltage. The site itself was powered through the Multi RS 48/6000.

Today I decided to update the firmware and configure the three Quattros as a three-phase system.

The sequence was as follows:

1. I first switched on each Quattro individually using its front switch, checked that it started, and then switched it off again after about 30 seconds.

2. Then I started again from the first Quattro, switched it on, updated the firmware successfully, waited until the process was complete, and switched it off.

3. I then switched on the second Quattro. It initially powered up.

4. After that I triggered a device rescan from the VRM / device list. At that moment there was a loud pop and smoke came from the second Quattro. The Cerbo GX also started beeping.

5. I immediately switched everything off using the inverter switches and turned the main DC switch to OFF.

6. After some time I restarted the DC system: first by holding the pre-charge button for about 30 seconds, then by turning the main DC switch ON.

In the logs there was something that looked like a short-circuit event on the Multi RS, but I do not believe there was an actual short circuit on the Multi RS. The Multi RS continues to work normally, and the load was transferred back to it through the ATS.

I tried to start the second Quattro once more. It attempted to start, but again made an abnormal sound and produced smoke, so I immediately switched it off again. I have not tried to start it since.

After that I tested the third Quattro. This time I did not immediately trigger a VRM rescan. I waited a little, then started the firmware update. The update process seemed to start and finish, but the firmware version did not actually change. I tried again with the same result.

Then I powered everything down and started again, this time switching on only the third Quattro. After waiting for some time and then doing the rescan, the firmware update completed successfully.

Important details:

• No AC input source was connected.

• No AC output load was connected.

• The AC wiring was prepared: input neutrals are connected separately, output neutrals are connected separately, and PE / grounding conductors are connected separately.

• Only the DC side and VE.Bus communication were active.

• All three Quattro units, including the second unit that failed, had been switched off by their front switches for about one year, but remained connected to the DC bus.

• The installation has DC fuses and a pre-charge circuit.

• The Quattros are connected to the Cerbo GX through an MK3-USB dongle, not through the built-in VE.Bus port.

• This connection method was chosen because the Multi RS is the main working system on this site, while the VE.Bus Quattro system is intended to be a secondary system visible on the VRM portal.

• Photos of the installation and the damaged unit are attached.

• I have already contacted my dealer and sent photos of the installation.

My questions are:

1. Could the firmware update or VRM device rescan realistically trigger such a hardware failure, or is it more likely just a coincidence?

2. Could the fact that all three Quattro units, including the failed second unit, were connected to the DC bus for about one year while switched off by their front switches have contributed to this failure?

3. Before commissioning or firmware updating a VE.Bus system that has been sitting connected to DC for a long time, is it recommended to fully disconnect the DC supply, wait several minutes for capacitors and the GX system to discharge, and then restart everything from zero?

4. Is it acceptable to start a firmware update when the VE.Bus system is connected to the Cerbo GX through an MK3-USB dongle, rather than directly to the built-in VE.Bus port of the Cerbo GX?
   This connection method was used because the Multi RS is the main system on the site, while the VE.Bus Quattro system is intended to be a secondary system on the VRM portal.

I understand that the damaged inverter should be inspected by the dealer/service center, and I do not plan to open it myself. I am mainly trying to understand whether I may have made a commissioning mistake, or whether this looks more like an internal hardware fault that happened to appear during the update/rescan process.

Thank you for any advice.

Підсумок

Цей текст буде приховано

[lxonline (LX)]

@andrewd

1. Could the firmware update or VRM device rescan realistically trigger such a hardware failure, or is it more likely just a coincidence?

Coincidence.

@andrewd

3. Before commissioning or firmware updating a VE.Bus system that has been sitting connected to DC for a long time, is it recommended to fully disconnect the DC supply, wait several minutes for capacitors and the GX system to discharge, and then restart everything from zero?

No. I basically have pulled one out of storage. Inspected it physically for insects etc. powered it up updated it programmed and used it

@andrewd

4. Is it acceptable to start a firmware update when the VE.Bus system is connected to the Cerbo GX through an MK3-USB dongle, rather than directly to the built-in VE.Bus port of the Cerbo GX?

Its preferred (at least by me) to use the mk3 to the laptop compared to the vrm update particularly with more complex systems. Less chance of problems.

[andrewd (Andrew Doroshenko)]

The thing is that I had already performed the same process before. This time I updated firmware version 558, which can be seen in the screenshot, to version 559. When the units were new out of the box, they had a much older firmware version with a significantly lower number.

So I do not think that incorrect wiring could be the cause, because the system had already started before with the same connections.

At this point I cannot understand what the actual cause could be.

Today is a weekend, so the dealer’s representative only asked me to send photos for now. He also hinted that equipment of this type does not normally produce smoke “for no reason”.

[lxonline (LX)]

Deleted the double post. Not sure what happened there.

Other than a wiring issue, i can’t think of a reason for the pop related to any activity. Did you recheck wiring and oahs3s etc things before power on.
Other possibilities (besides a capacitor issue or current on ground) are insects or rodents shorting something.

@andrewd

The thing is that I had already performed the same process before

Which is why i think there it’s a coincidence. But a firmware update does perform a reset which is a power up and a change in state.

Its also of course entirely possible that there was an issue with the unit, warranties being there for a reason. Doesn’t stop the feeling in the pit of your stomach though and then the endless self examination

@lxonline

Its preferred (at least by me) to the vrm update particularly with more complex systems. Less change of problems.

Realised this was galf finished answer- i use the mk3 directly to the inverter from the device i am using to update and program with. (Edited my previous answer)

You can connect to the ve bus port when running it otherwise.

You have an interesting wiring colour scheme there.

[andrewd (Andrew Doroshenko)]

Visually I cannot see any signs of an external short circuit, damage, insects or anything similar. Of course, I have not opened the Quattro itself, so I can only speak about what is visible from outside and around the installation.

Before the failure happened, I had already switched on and off each Quattro individually today. Each unit started normally when tested separately.

Then I remembered one more detail.

After checking each unit individually, I switched on the first Quattro and then the second one. At that moment some LEDs started blinking — I do not remember for sure whether it was on both units or only on the second one, but the second unit definitely had blinking LEDs. I assumed this was because the units had not yet been configured as a three-phase VE.Bus system.

After that I switched both of them off and decided to start from the first unit again: update firmware first, and then later configure the three-phase system in VEConfigure.

The first Quattro firmware update completed successfully. Then, when I proceeded to the second Quattro, the failure happened as I described earlier.

So the part I cannot understand is this: the second Quattro had already powered up normally earlier today during the individual test, and only later failed

[lxonline (LX)]

@andrewd

So the part I cannot understand is this: the second Quattro had already powered up normally earlier today during the individual test, and only later failed

Yeah. I and i can only speculate.

@andrewd

The update process seemed to start and finish, but the firmware version did not actually change.

@andrewd

but the second unit definitely had blinking LEDs. I assumed this was because the units had not yet been configured as a three-phase VE.Bus system.

The events logs shows 3 updates from 558 to 559, is that from the same unit?
Could have been something related to the failure maybe a loose cable or something. As there was something not quite right. I am sure the warranty centre will know more.once it has been set up on the bench there and investigated.