Need help troubleshooting a pair of MPII's arranged in a split phase configuration showing errors

[MotoringMariah (Scott)]

Hi,

I have had just over a year of use of our new Victron installation on our 55’ motor yacht and about a week after returning from a trip I started getting weird errors on my Cerbo GX display and now after several reboots of my MPII’s I think I have an internal error that says I should contract Victron for support. Hoping sharing the details here someone can confirm my best course of action to fix this.

For reference high level diagram of my components. Boat has 24VDC system and we take 240Vac from shore power so each MPII gets a 120Vac leg

Boat is now docked in my backyard on shore power so in theory little to no load on the inverters. Late this past Thursday night I started getting VRM alerts about an overload on L2 and then a loss of Time Synchronization and device cannot transmit data:

I didn’t discover these until the next morning so I came down to the boat and started investigating. The LED’s on the front of both MPII’s were flashing different codes and I looked them up and came to the conclusion the best first step is to reboot both MPII’s. I had one MPII flashing bulk and float LED’s and the other flashing all three bulk, absorption, float LED’s. I got my MKIII device out and my laptop and tried to connect to second MPII as it has an open VE.Bus port for the MKIII so I leave it connected. At first I couldn’t connect even though I have used this port before to program both MPII’s.

When I launched the VE.Bus System Configurator it said nothing else should be connected to I removed the RJ45 connection from my first MPII and plugged the MKIII device in that port and am able to connect and see both MPII’s and their configuration appears correct:

System Check shows OK for both MPII’s and the asterisk tells me L1 is the Master and L2 is the one I’m having a problem with:

I rebooted both MPII’s a couple of more times and now have the first one flashing just Bulk which I believe means it can not be a Slave. And the second MPII is still flashing three bulk/absorb/float LED’s and has a steady low voltage light but I have 26.5Vdc. I think the low voltage is a false flag as one of my Lynx Smart BMS’s is beeping continuously. The battery SOC is 36% and I have an auxiliary battery charger now charging the main house bank.

Am I correct to interpret 3 flashing as Internal error?

I don’t have a picture of my first MPII but it has just the Bulk LED flashing and I think this table reflects “cannot function as a slave” as the problem IF the pictured indicators in the table below are supposed to be flashing. The last one Code 26 seems like root of my problem and IDK who/how to contact Victron for support on this and could use some input. Anybody know what type of failures these MPII’s typically exhibit? To me it seems like it’s a CPU or communication error on the main board. Hoping a simple board swap would fix it vs having to uninstall re-install a new unit.

When I first came to the boat I saw this flashing on the first MPII and the time sync error in the Cerbo. It’s confusing that error code 10 in the table below appears to be flashing but the error codes above seem to indicate steady state not flashing and my LED’s are flashing.

Additional data points I keep seeing a time sync alert appear in the Cerbo making me think it’s a communication issue between the MPII’s but after several reboots I noticed I have 3 active alerts. The first not all devices found makes sense as my second MPII doesn’t seem to be communicating. The Low Battery Alarm is a red-hearing IMO as the battery SOC is upto 45%. The High DC Ripple warning has my puzzled as to what that is/how it’s detected etc. Boat currently setting idle at the dock, no engines running and the aux 24VDC charger is on bringing the batteries back to 100% SOC since it’s overcast and my Solar is not producing enough to charge the batteries…

Scott in Sunny FL aboard M/V MARIAH

[lxonline (LX)]

Error 14 - Device cannot transmit data - VE.Bus Error Codes [Victron Energy]

Sometimes replacing the rj45 patch comms cables sorts it out.
If not then the units communication boards are faulty.

When using system configurator both (all inverters configured as asystem) inverters should be connected together, when the instructions say ‘nothing else’ they mean nothing but the inverters.(No gx etc)

Direct from victron support is done through a network. Not direct direct…

First line of support is supposed to be the installer.
Failing that the place that supplied the installler (ideally distributor)
If the unit is faulty it can go to the nearest repair centre for a warranty claim. You can look up victron where to buy.

If you are going to handle it yourself start here. It will help if your testing uses a full battery.

4. Pre-RMA test form - Inverter/charger

You can also submits a request here.

Need support? - Victron Energy

[MotoringMariah (Scott)]

LX,

OK Thanks for the feedback and I can see the Pre-RMA test form offers some things to check/try/test but to me the picture is becoming clearer it’s probably the communication board in my second MPII that failed. My LiPoe4/house batteries SOC are now 100% after I used the engine start battery charger to bring the house bank back up to full. I live in Sunny Florida the lightening capital of the USA. We have had a lot of thunderstorms recently and it’s not uncommon for me to find our VHF Radio or random LED lights turned on after a thunderstorm passes through. The MPII units were under no load and this communication problem randomly started last Thursday night…I suspect after a storm rolled through but IDK for sure. I could use some input on the meaning of the blinking LED’s on each unit before I start the RMA process. I believe I mis-read/understood the LED lights table in the MPII User manual the first time so after spending much time re-reading it AND explaining my situation to my Victron Sales Rep I discovered I have different error codes. My Sales Rep sold me my equipment directly and I am the installer so as the installer I am my own Support. My Victron Sales Rep said the same as you try replacing the RJ45/VE.Bus cables first and I have done that between the units and to the Cerbo. I did find a plastic locking tab broken off the RJ45 to the Cerbo and the cable between the units was an older cable so I now have two brand new ethernet cables connected and as I explained the situation to the Sales Rep I had the epiphany that I was mis-reading the LED Tables.

I’m 90% certain a blinking BULK LED on the first MPII means I can’t be Slave and IMO it should be the Master it’s L1 of my pair. The second MPII is L2 and it has blinking bulk/absorb/float and I think that means internal error contact Victron for help. The table in the User Manual is kind of cryptic in how they describe solid vs flash. I thought that meant Error code 26 but this table suggests the LED’s are on solid not flashing:

Looking at the Flashing Codes it says “Is the ‘inverter on’ LED flashing? If not, then there is no VE.Bus error code.” I have no Inverter LED’s on.

Next step it say “If one or more of the LEDs ‘bulk’, ‘absorption’ or ‘float’ flashes, then this flash must be in phase opposition to the ‘inverter on’ LED, i.e. the flashing LEDs are off if the ‘inverter on’ LED is on, and vice versa.” So only Bulk flashing on first other two off I think is Error code 9 It’s hard to see this in the manual because the table spans two pages AND error code 9 is not in the table below that!

For the second MPII I have all three flashing which I think is Error Code 13 from table above and that too is not in the Error Code Tables so IDK what I’ve got:

So what are Error Codes 9 and 13?

When I use my MK3 device with a laptop and VE.Bus Config I can communicate via either VE.Bus port on my first MPII. When I do that on the second MPII the one I think has a bad communication board only one of the two VE.Bus ports works with my MK3 to allow me to talk to the MPII. Keeping the RJ45/ethernet communication cable between the units in the good VE.Bus port I mainly see over and over a time sync problem and missing VE.Bus device being reported which seems a like a communication issue:

Before I start the RMA process I’d like to know the meaning of the blinking LED’s i.e. Error codes 9 & 13 which are not listed in the error codes table of the User Manual. My Victron Sales rep says he never looks up the LED codes anymore because the Cerbo usually tells you the problem as seen above. I really hope the RMA process is me replacing the communication board because unmounting and shipping this thing off for repair is gonna be a BIG project and I know cause I installed both of them!

Scott in RAINY Florida…

[lxonline (LX)]

Very simply if you cannot return them to stand alone and updating them and test each one separately, then it has to come out.
This would be the steps i would take, as reprogramming to split pahse is easy if it resolves with doing that.

[MotoringMariah (Scott)]

LX

As I finished my Reply above and was communicating with my Sales Rep we had the same idea that I need to re-config them back to default/standalone to remove the time sync dependency. IF this had happened while we were in the Bahamas we would have been screwed! One of the reasons I liked having two MPII’s was we would have redundancy but now because of the dependency between the units neither one is working. I’m thinking to try to flash them back to default config or turn off the inverted phase sharing to see if I can get one unit working. This will help to prove the second one is the problem. Fortunately my sales rep said there is a Victron Repair Center in Ft Lauderdale Florida about 2.5 hour drive from me. As a boater far from home it’s important to have spare parts to be able to sustain yourself when things go wrong. I asked my sales rep for a quote for a new unit with the idea to install the new unit in place of the failed unit and then if that fixes the problem ship the bad unit off for repair. When the bad unit comes back repaired we will keep it on board as a SPARE!

Last question though…it still bugs me what is the meaning of Error Codes 9 & 13? The tables show the blinking lights mean that but don’t list what they mean. When I search the Community for “Error Code 9” or “Error Code 13” my post is the only match…

[MotoringMariah (Scott)]

OK So new error codes now that I have reverted my MPII’s to default config. Using my MK3 and a laptop with VEConfig I was able revert my config to two standalone inverter/chargers, no Virtual Switch, no assistants, no special config which I think is a good thing they are talking to my laptop via the MK3. After doing this one would start charging the batteries and then stop after 5 seconds while the other just continues to flash builk/absorb/float LED’s. Before changing the config from out of phase pair to stand alone I was getting errors related to time-sync between the units and a missing VE.Bus device which I suspect is my second MPII. I changed the RJ45/Ethernet cables between the MPII’s and used a new cable to my Cerbo to eliminate the possibility it could be a cabling problem. So now on the first MPII I have Inverter/Bulk alternately flashing with the Float LED on. The reference table says this is Error Code 12 which is not defined in the table BUT in the Cerbo I keep seeing messages about the BMS is missing…so the charger can’t charge if doesn’t know the battery SOC

The Cerbo display for the Inverters alternates between Fault condition and Off making me think it’s showing one is off and the other Faulted

The Pre-RMA instructions seem very generic as they give instructions to do things I can’t do. The MPII’s don’t support Bluetooth and the tell you to use Victron Connect w/Bluetooth to connect and check/set configs. Then they say use Victron Connect to reset config to Default by My MPII’s will not allow this and I don’t know what config to change to enable this:

There is references to checking voltages and fuses inside the unit so I guess next is removing the cover to see what’s inside and look for evidence of damage, blown fuses and checking voltages before RMA/Warranty claim. Both have low Voltage indicated but the battery SOC is 95%

Inverter/bulk alternately flashing on first:

3 always on LED’s on second, plus now both have low battery making me think it’s a communication problem with the BMS/Cerbo yet the Cerbo seems to know the state of the inverter/chargers:

[lxonline (LX)]

Error 15 - VE.Bus combination error - VE.Bus Error Codes [Victron Energy]

I think maybe you will need some training to be able to diagnose and recover your own system.

They both cannot connect at the same time to the cerbo via ve bus or each other in stand alone.

Testing as stand alone mean a comms disconnect and a settings reset. (Isolating them) then individually powering them on and testing them. Which the basic gist of the pre rma (obviously skipping irrelevant parts in this case you don’t have a smart product)

[MotoringMariah (Scott)]

@LX I’m always open to education and love posting here for the information that gets shared helps to educate me! I do understand the idea of isolation and diagnosis and thought/did the same initially over the weekend. I disconnected the VE.Bus connections between the units and to the Cerbo. I have the latest firmware installed and I then changed the Firmware settings on both MPII back to what I thought was default but didn’t see a “return to Default” setting anywhere so I did my best to revert it back to factory default IE no longer an inverted phase pair/now standalone, no assistants, no VS switch settings, no Conditional AC input settings. Lithium NG batteries which I think/thought required the VE.Bus connection so the MPII’s would know the SOC of my batteries from the BMS and when to charge them. After doing that the first MPII booted and flashed Inverter/Float alternately with solid Low Battery light. The second one flashed absorb/float/bulk LED’s with a solid Low Battery. I recall when I initially installed this I had similar low bat LED and thought that’s why you need the VE.Bus connection to get the Battery SOC. I returned the VE.Bus connection to the first MPII to the Cerbo and then got Inverter/Bulk flashing with Absorb on solid initially later Float also on continuously. I tried adding in the Second MPII with the VE.Bus connection but that didn’t help and I’m stuck.

I reached out to my Victron Distributor begging for help and using VRM he started to look for me today and offer similar suggestions; we need to get these working in isolation, then see if one is bad or both are bad, once one is bad add it back to the system config etc. IDK all the steps as I don’t do this for a living and the dependencies, naming and nuances of configuring this stuff is best learned from experience and at the moment I only have a year of OJT on my own system only. Thanks for your support, fingers crossed my distributor can offer some more helpful insight…

[lxonline (LX)]

Glad there is closer help. And a good set of helpful eyes.
Crystal balls and the internet being what it is. I feel like sometimes i am shooting in the dark. And sometimes its a small mis step in confir or uodate that cuases an issue.

Seeing the install and having hands on makes diagnostics different.
I am sure there will be a solution.

[MotoringMariah (Scott)]

Yeah frankly I can figure anything out but need insights/secrets/things I don’t know I need to know and I find forums can be valuable for these tidbits. Recall when you were setting me up for the Generator/shore power start/stop node-red logic and Bart mentioned something about on/off switches on the MPII. Mine are face down on the bottom of the unit and I forgot they were there. Yesterday I had the epiphany to use them instead of the master DC power disconnect. What I discovered was my first MPII when switched to charger only mode with no VE.Bus comm cables connected will power up and start charging the battery bank. This is a good sign IMO! When I plug the Cerbo VE.Bus cable in I get an Error 15 on the Cerbo display but LED’s display Error 17 “One of the devices has assumed ‘master’ status because the original master failed.” The second MPII was powered off with no VE.Bus connection so somewhere, I think in the Cerbo, the system is still looking for the pair of MPII’s to be linked as an out of phase pair. I have a call scheduled with my Victron distributor later this morning so this should be a good starting place.

[MotoringMariah (Scott)]

Progress made today after speaking with my Victron Rep. Standard troubleshooting requires you to be on the latest released firmware and I was off on my Cerbo because I was happy with the stable release I was running V3.70-96 but ok I understand I upgraded to V3.73 I was also off a .1 release in my Lynx Smart BMS’s so upgraded from V1.16 to V1.17 I was initially just focusing on my MPII’s but I get it best way to ask for help is to have everything on the latest stable release. After doing that we went through my MPII configurations and he pointed out while I had reverted them to default config I changed the Battery Type to Victron Smart Lithium batteries with VE.Bus BMS which is what I have! BUT he says no that needs to be set to Lithium Iron Phosphate batteries. Making that change got rid of the Low Battery Voltage LED on both MPII’s and I was able to get the first one up and running normally in stand alone mode:

I mentioned to my rep the Low Battery LED was Deja vu from my initial install as I’d swear I made the same mistake last year during my setup. This makes no sense but it’s small stuff like this from others that saves the day.

Feeling Bold I went back in and reconnected the second MPII communication cable and removed the Cerbo cable from the first MPII so I could use my MK3 device to reconfigure the pair of MPII’s as an out of phase L1/L2 pair. As soon as I pushed the config I got Error 10 on the first MPII can’t sync time and the error 13 3 LED’s still flashing on second. I restored the Cerbo connection and now see these errors. No Flashing Inverter LED so no VE.Bus errors supposedly…

Error 13 is the crux IMO as I have had a bad feeling about the Second MPII from the beginning of this fiasco based on the 3 LED’s always flashing. I found the Help Screens in the VE.Configure utility explains a lot of the error codes that are not defined in the MPII User Manual. For Error 13 it says this:

I’m waiting for next steps from my dealer…

[MotoringMariah (Scott)]

FYI Just to update this thread, worked with my Victron rep/distributor today to determine my second MPII is in a weird state. Tried loading different versions of firmware and strange things were happening. Like the Serial number in the Firmware disappeared and the unit got in a worse state and would not turn on. Eventually got back to the latest firmware v556 but still have 3 flashing LED’s and no communication on one of the VE.Bus ports so it was determined it needed to be RMA’d for repair. Wish it was a simple board swap vs having to uninstall/ship/reinstall a 64lb inverter/charger 90% of which is just coils/transformers that are not likely to fail…but it is what it is. Will try to remember to close this thread/loop but at the moment IDK how long it’s going to take to repair.

[chrigu]

@MotoringMariah

changed the Battery Type to Victron Smart Lithium batteries with VE.Bus BMS which is what I have!

You have victron SmartLithium, yes, but with a lynx BMS, not a VE.Bus BMS.

The difference is that the lynx works like other canbus BMS, it reports to the GX, which then controls the MP over the VE.Bus.

Whereas the VE.Bus BMS is put into the connection between GX and MP, and interceptd that.

[MotoringMariah (Scott)]

Chrigu,

THX for the explanation can’t say it clears up my understanding 100% but I get the gist of what your saying. To me the Lynx BMS is putting battery status on the VE.Bus so it’s a VE.Bus BMS…AH but to make my point I just went to my engineering drawings and see the Lynx BMS actually talks to the Cerbo over VE.Can NOT VE.Bus which directly correlates with your explanation:

So I get what your saying about the GX controls the MPII’s via the VE.Bus messages as I typically see “External Control” displayed in the Cerbo for the MPII’s. What concerns me about this complexity is what happens when one dependency fails the whole system fails. I wanted redundancy and durability that if for some reason a component fails the rest keeps working. This is exactly what happened and started this thread when one of my MPII’s went bad it prevented the other one from working since they were a phase coupled pair. Through work I was able to decouple them and get one working standalone but it drives me nuts that the MPII’s are not aware of the battery SOC. IF the MPII’s were aware of the Battery SOC from the Lynx BMS they could continue to operate on their own to safely charge the batteries. Why doesn’t the MPII’s know the Battery SOC, see Pic below from Cerbo? What happens if my Cerbo dies do I loose all charging/inverting/battery management capability? IF so I need to stock a spare Cerbo on board! Notice in my Eng Drawing we have 2 MPII’s, 2 Lynx BMS’s we also have two engines and two alternators…when your out to sea and something dies you need to be self reliant to repair/survive till you get to port.

[chrigu]

@MotoringMariah

IF the MPII’s were aware of the Battery SOC from the Lynx BMS they could continue to operate on their own to safely charge the batteries

No, that has nothing to do with the battery SOC. The VE.Bus system is not designed to work with a faulty/missing unit. Once a few MPs are programmed into running as a system, they can then only run as this system, together. If one units fails, the rest switches off. Thats always been the case for VE.Bus, even before LiFePo and SOC were a thing.

As for the missing SOC, did you enable the battery monitor in the MPs config? Since that needs to be enabled in order for the MPs to receive the SOC information from the GX, which otherwise works fine.

@MotoringMariah

What happens if my Cerbo dies do I loose all charging/inverting/battery management capability?

I havent had a dead Cerbo yet, but i would expect the system to shutdown, yes. Simply based on the fact that the MPs get turned on/off over pins 6+7 of the VE.Bus cable, and if the device that gives that command switches off, then also the MPs will switch off. However you should be able to restart them by disconnecting the VE.Bus cable going to the Cerbo and flipping the on/off/charger only rocker switches first to off and then to on again.

The MultiRS can run with a unit (or even multiple units) missing in a system, since they use Canbus based communication. But they have their own flaws/shortcomings.

[MotoringMariah (Scott)]

IF your right that unplugging the VE.Can cable to the Cerbo and restarting the MPII’s via the switch would enable standalone I’d be ok with that. I will have to test that once I get my second MPII back online. How would the MPII know the SOC of the battery to know when to stop/float charge without a connection to the Lynx BMS or the Battery BMS…purely on Voltage level given the MPII’s are configured for Lithium Batteries not Victron LiPoe4 NG? Again I’d be ok with that just want to know how it works/what to expect in that situation…

As I’m typing this I’m having an Ah moment realizing more about how my config works. To your point the Cerbo controls the MPII when to charge. The Lynx BMS’s control the Engine Alternators via the Allow To Charge Signal. Having redundant Lynx BMS’s was tricky to wire the ATC signal to stop both engine alternators when one or the other or BOTH Lynx BMS says STOP CHARGING. I got it figured it during install but it was a nuance that was important to consider. So ultimately the Engine Alternators are unaffected by the Cerbo so they can run purely with just the Lynx BMS connection which is GOOD IMO!

[chrigu]

@MotoringMariah

the VE.Can cable to the Cerbo

The VE.Bus would it need to be.

@MotoringMariah

purely on Voltage level

Thats how battery charging works. SOC does nothing here, even when its available. Its either constant current (bulk phase) or constant voltage (every other phase but with different voltage levels) mode.

Thats also the reason you still need to set the charge parameters correctly on the MP itself (or any charger), for the event that theres no communication to the BMS anymore, as a fallback.

@MotoringMariah

To your point the Cerbo controls the MPII when to charge

Well yes and no. The Cerbo is mainly just relaying information in this example here. The main thing here is DVCC and its parameters CVL Charge Voltage Limit, CCL Charge Current Limit and DCL Discharge Current Limit. These parameters are sent by the BMS, in accordance to what it thinks is best for the battery at this point. These parameters are then sent, mostly unaltered, to the MPs as well as other connected chargers like an MPPT.

So if the BMS requests a CVL of 28.4V then the chargers will start charging up to that level. Once they reach that voltage, they just stay there. Its up to the BMS to then lower CVL again for Float or Storage or whatever. Same for the CCL, if the batteries are getting full, CCL is gradually lowered by the BMS in order to ease in on the full charge target (and to give cell balancing some time if necessary).

CCL is being adjusted down by the BMS as the SOC closes in on 100%

Whenever a charger is controlled by DVCC then its state will be “External control” meaning it is no longer using its own charge voltages and algorithm, but solely listens to what the BMS and DVCC say.

As for the alternator control, im not sure, but i would expect it to work without a Cerbo, since its an ATC-like signal coming from the BMS itself

[MotoringMariah (Scott)]

@chrigu thanks for the detailed explanation AND I realize in my previous post I confused VE.Can and VE.Bus. Your explanation about SOC and voltage levels totally makes sense and I get it, I’ve been surprised how high the voltage levels are with LiPoe4 batteries vs my old AGM’s and how accurate the SOC is based on those voltage levels even under load. I like data and it would make me feel better if the MPII’s knew the SOC of my bank and since that doesn’t come via VE.Bus or VE.Can I played with some of the MPII settings using the VE.Bus System Configure utility and was able to turn on SOC in the MPII’s so I see it and feel better. IDK if this value is calculated or found from other data but it tracks with what my battery bank shows now, where before SOC would be blank when I drilled into the MPII’s via the Cerbo or WebUI

To go full circle and enable the admin to close this post my system is back up and working 100% using a pair of MPII’s in Split Phase/Inverted configuration. The MPII that died was sent off to a Victron Repair Depot for warranty repair. The part was not in the USA and the estimated time for the repair was 2-4 weeks so rather than wait I bought and installed another MPII in place of the failed unit. The failed unit was repaired and being sent back to me this week so we’ll stock it as a spare should this ever happen again when we are out to sea or in some remote location. I asked the Repair Depot tech what was wrong/replaced thinking it was the communication board to the VE.Bus and received this reply “there is only one board inside the MPII’s that goes bad, why did it go bad? No idea. We see about 2-4 of those particular units a month.” WOW so they are seeing that many of these units in for a repair a month! So logical next question can I buy a replacement board and stock it as spare NOPE: “I cannot sell parts directly to customers, victron policy. Sorry. But the newly updated boards are supposed to be more resilient and can accept and dissipate voltage spikes better than the old boards.” So fingers crossed this doesn’t happen again and instead of stocking a spare board I have a whole MPII as a spare! We like spares…spare water pumps, spare pump impellers, spare fuel/oil filters, and on and on…THANKS for all the input! AND for any other boaters out there @lxonline NodeRed code to get automated generator/shore power start/stop is AWESOME working well!