[CAN dropped messages] Multiplus II GX + Pylontech Force L2 = no data

Question

question

xzv asked Feb 9, '24

[CAN dropped messages] Multiplus II GX + Pylontech Force L2 = no data

I tried to build up two systems consisting of:

Multiplus II GX
Pylontech Force L2 (stack consists of 3x Battery module + 1x BMS)

I expected it to be "Plug & Play" but actually I'm now stuck for two days because I don't receive any detail data from the battery. The Multiplus recognizes a Pylontech battery but that's it. No SoC, no module level data.

However, when I unplug the CAN cable it's immediately reflected in the Remote Console. Plugging it back in brings the battery back.

It can be observed that it's very likely that it's a CAN or protocol error, as the status page suggests:

What I've done and tried so far:

Updated the GX and Multiplus to the latest firmware (v3.13).
Changed VE CAN to CAN-bus BMS (500 kbit/s).
Changing the termination resistor (even leaving it unplugged).
Following Victron's and Pylontech's manuals rigorously (here and here), paying special attention to pin mappings on both sides.
Made 3 different cables ("Type A"), each with or without a second CAN termination resistor.
Started the Multiplus first, or the Pylontech Force L2 first.
Waited for long times between boots.

The manufacturing date of the Pylontech Force L2 is Feb 2023, having installed firmware MW3_15_B505_1.27. Maybe it's a firmware problem on Pylontech side? I don't have access to a newer firmware.

My interpretation of the CAN port screenshot above with the partially lost TX data packets is that the Pylontech BMS is responding to most, but not all, of Victron's data requests. Therefore it could be a software problem on the Victron side or on the Pylontech side.

I saw similar posts in this forum dating a year back, but without a suggestion on how to solve this.

Any ideas, or someone who could fix this in the past?

Multiplus-II cerbo gx Pylontech VE.Can

as.png (23.2 KiB)

a.png (208.4 KiB)

s.png (23.8 KiB)

4

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ejrossouw commented · Feb 09 at 10:04 PM

@XZv For what it is worth try swapping the type A cable around. I had similar problem on a US3000C stack and that strangely solved it. Also, not sure if it applies, but Pylontech suggests not having the ground wire as it can interfere.

0 ·

xzv ejrossouw commented · Feb 10 at 04:10 AM

I already tried it without the GND wire connected. No luck so far :/

0 ·

ejrossouw xzv commented · Feb 10 at 08:35 AM

Have you tried swapping the cable around using bms end at the Cerbo end?

0 ·

xzv ejrossouw commented · Feb 10 at 10:02 AM

Yes (even if it doesn't make sense ;) )

0 ·

156 People are following this question.

Modifications Space

Node Red Space

Answer 1 · 2024-02-11T12:37:00Z

xzv answered · Feb 11 at 12:37 PM

I tried the same cables and settings on my Cerbo GX and it works instantly. Any idea why it doesn't work with the Multiplus II GX on VE CAN?

1707655002600.png (163.4 KiB)

9

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ejrossouw commented · Feb 11 at 02:02 PM

@XZv Rather baffling and all that is left is to update the Cerbo firmware to the latest official 3.14 and then visit the Pylontech facebook support group if it still fails. What firmware version is the Cerbo GX on and are you using the VE.can or BMS.Can port?

0 ·

xzv ejrossouw commented · Feb 11 at 02:30 PM

Both have the latest 3.14 installed, so no difference there. The only difference I see is that the Cerbo GX has is dedicated BMS CAN port, where as I use the VE CAN port set to BMS CAN on the Multiplus II GX.

0 ·

ejrossouw xzv commented · Feb 11 at 03:32 PM

I have a customer with a Force2 and Multiplus-II GX so this is most peculiar. There are actual differences between the dedicated "scaled down" BMS.can implementation of canbus and VE.can. On the latest MK II Cerbo however, there is only a VE.can port, which begs the question whether we will see more issues like this, not to mention the speed issues of BMS'es often operating at 500 kbits vs other devices at 250 kbits, which may limit the usefulness of a port intended for daisy chaining multiple devices. Would it be worth trying your luck with the 3.20.50 Beta firmware given it is easy to roll back?

0 ·

xzv ejrossouw commented · Feb 11 at 04:53 PM

Thanks for letting me know that you already saw it working in the field.

Good idea to try the beta. I'll let you know if it makes any difference. I can also try a downgrade, as the compatibility page suggests that v2.20 or later works with Force L2.

0 ·

xzv ejrossouw commented · Feb 12 at 08:18 AM

The upgrade to v3.20 beta didn't change the situation. Then I tried to downgrade the device to the version specified as compatible, v2.20. This downgrade bricked my GX. :(

Right now I have no idea how to bring it back to life.

0 ·

xzv ejrossouw commented · Feb 12 at 07:58 PM

@ejrossouw What firmware version does your customer use? Maybe that helps narrowing down what change in code (?) in Venus OS disabled the support of the Pylontech Force L2?

0 ·

ejrossouw xzv commented · Feb 13 at 08:46 AM

@xzv I also sent an email to Pylontech support and await feedback. I noticed your other efforts and just wondered whether you tried both GX ports and checked the actual pins are ok?

0 ·

56d17f93-663d-47b2-801d-063e08f1a9e7.jpeg (322.5 KiB)

xzv ejrossouw commented · Feb 13 at 09:58 AM

Thanks, I'll try a downgrade to v3.10 later.

I checked all pins, I even exchanged the cables and Multiplus II GX to another one.

0 ·

xzv ejrossouw commented · Feb 13 at 12:37 PM

@ejrossouw I downgraded to v3.10 (the one your customer has installed). It doesn't change to the better, still no detail data. Very strange!

0 ·

Answer 2 · 2024-02-12T08:39:49Z

Alex Pescaru answered · Feb 12 at 08:39 AM

Hi @XZv

Re bricked Cerbo:

See https://community.victronenergy.com/questions/204255/cerbo-gx-bricked-how-to-recover.html

Re CAN communication:

Cerbo doesn't "ask" for anything from battery. It just emits a specific presence ping on specific CAN IDs. In contrast, the battery is broadcasting its status over CAN and Cerbo just take the info from there.

Do you have superuser enabled on your Cerbo? Can you ssh log in into your Cerbo with the root user and password?

If yes, try to give the following command (without quotes): "candump can1" and report back here what you see there.

Of course, after you connected the battery cable on dedicated BMS CAN port, with proper terminator in the second BMS CAN port, because the BMS CAN port is "can1" interface in Cerbo hardware.

And also see if on the battery you have some specific dip switch configuration in order to enable CAN terminator there too.

Alex

2

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv commented · Feb 12 at 03:31 PM

@Alex Pescaru

On the Multiplus II GX (port labelled "VE CAN"):

root@nanopi:/dev# candump can0
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  can0  359   [8]  00 00 00 00 03 50 4E 00
  ...
root@nanopi:/dev# candump can1
SIOCGIFINDEX: No such device

Interesting enough is that there is no time delay when it prints that hundreds and hundreds of identical messages!

After attaching the CAN cable to the Cerbo GX (port labelled "BMS CAN"):

root@einstein:~# candump can1
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  359   [8]  00 00 00 00 03 50 4E 00
  can1  351   [8]  14 02 E8 03 E8 03 C2 01
  can1  355   [8]  38 00 64 00 00 00 00 00
  can1  356   [8]  35 13 FC FF C2 00 00 00
  can1  35C   [8]  C0 00 00 00 00 00 00 00
  can1  35A   [8]  AA AA AA 02 00 00 00 00
  can1  35E   [8]  50 59 4C 4F 4E 20 20 20
  can1  372   [8]  03 00 00 00 00 00 00 00
  can1  373   [8]  CE 0C CF 0C 24 01 25 01
  can1  374   [8]  30 31 30 31 00 00 00 00
  can1  375   [8]  30 31 30 32 00 00 00 00
  can1  376   [8]  30 31 30 32 00 00 00 00
  can1  377   [8]  30 31 30 31 00 00 00 00
  can1  379   [8]  DE 00 00 00 00 00 00 00
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  305   [8]  00 00 00 00 00 00 00 00
  can1  307   [8]  12 34 56 78 56 49 43 00
  can1  359   [8]  00 00 00 00 03 50 4E 00
  can1  351   [8]  14 02 E8 03 E8 03 C2 01
  can1  355   [8]  38 00 64 00 00 00 00 00
  can1  356   [8]  35 13 FD FF C2 00 00 00
  can1  35C   [8]  C0 00 00 00 00 00 00 00
  can1  35A   [8]  AA AA AA 02 00 00 00 00
  can1  35E   [8]  50 59 4C 4F 4E 20 20 20
  can1  372   [8]  03 00 00 00 00 00 00 00
  can1  373   [8]  CD 0C CF 0C 24 01 25 01
...

0 ·

Alex Pescaru xzv commented · Feb 12 at 04:56 PM

OK. @XZv

On the first case, on the VE CAN port, because the battery didn't see any inverter presence there, only broadcasts its own presence and tells "the world" that it's a Pylontech battery with 3 modules.

On the second case, on the BMS CAN port, because the inverter makes its presence felt there on the IDs 305 and 307, the battery starts to publish all the information it has available.

So, from the battery CAN communication point of view all is OK and all the info is there.

The problem, I'm afraid, it's on the Multiplus GX's side...

There the CAN profile should be set as CAN-BMS 500 kb/s.

And the battery won't start to send its data until it sees an inverter "ping" there and it seems that the nanopi doesn't send any presence pair/ping. Neither on 305 nor on 307.

Alex

0 ·

Answer 3 · 2024-02-12T18:01:13Z

xzv answered · Feb 12 at 06:01 PM

Thanks for the analysis. Exactly because those two devices (Cerbo and Multiplus) show different behavior with the same firmware version (v3.14) I wanted to downgrade it to the version that is explicitly mentioned as being compatible to the Force L2.

Here's a comparison video of the Cerbo GX vs. the Multiplus II GX. First, I started candump on the Cerbo, then plugged in the CAN cable, waited for a few seconds before removing the cable. Then I did the same on the Multiplus II GX (candump, cable in, cable out). You can obviously see that the Multplus shows a strange behavior. No "presence" messages while disconnected, instead a load of messages from the battery once the cable is plugged in.

candump.mov

I read here that the serial bus could be a limitation in terms of throughput. But it's only one BMS, so that CAN bus is not congested...

I'm at a point where I doubt that I can generate more insights or debug it further.

Maybe @Matthias Lange - DE can comment on this? I'm pretty convinced that I found a bug or at least some unintended side effects. I'm happy to provide all information necessary to reproduce this, and as I mentioned before I'm not the only one affected.

candump.mov (51.2 MiB)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 4 · 2024-02-14T06:30:49Z

@Alex Pescaru

This is what dmesg has to say:

root@nanopi:~# dmesg | grep spi
[    2.373335] spi_master spi0: will run message pump with realtime priority
[    2.504228] mcp251xfd spi0.0 can0: MCP2518FD rev0.0 (+RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized.
[ 4130.521713] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4141.854706] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4152.854269] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4164.853296] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4174.855157] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4177.853663] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4199.853685] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4210.852941] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4220.850968] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4231.852828] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4241.853853] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4251.854103] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4262.853848] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4273.852063] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4284.853368] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4296.851815] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4306.854960] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4389.855231] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4394.858522] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4405.851135] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4415.854147] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4425.852498] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
[ 4437.851836] mcp251xfd spi0.0 can0: bus-off, scheduling restart in 100 ms
...

Any idea what goes wrong? Any options I can try on the CAN bus?

@Matthias Lange - DE This is not only my problem, as you can find similar cases in the forum like here. Can you please help?

Answer 5 · 2024-02-15T05:34:15Z

xzv answered · Feb 15 at 05:34 AM

Now I can share an update on my insights:

When I login via ssh and enter this:

root@nanopi:~# ip link set can0 down
root@nanopi:~# ip link set can0 up type can bitrate 501000

I get this:

In other words: If I set the baud rate manually from 500 to 501 kbit/s, the Multiplus II GX recognizes the battery.

This must be redone after each boot.

There must be an incorrect setting in Venus OS or a hardware timing problem.

How can I reach someone from Victron who can fix this?

present.png (80.6 KiB)

5

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

nickdb ♦♦ commented · Feb 15 at 06:09 AM

Have you considered you have a BMS fault, maybe it was set incorrectly at the factory?

Have you contacted pylontech support?

There are thousands of pylon batteries in the field, and almost all battery BMS's run at 500kb, if the software was off by 1, many people would have problems.

0 ·

Guy Stewart (Victron Community Manager) ♦♦ commented · Feb 15 at 06:13 AM

Hi @XZv,

Thanks for persisting thru this and posting the results.

I am wondering what made you decide to try manually changing the baud rate to 501 kbps?

0 ·

xzv Guy Stewart (Victron Community Manager) ♦♦ commented · Feb 15 at 06:36 AM

Pure despair :)

I tried many other things but lead by intuition and interpretation of dropped TX messages I just gave this a try.

0 ·

Alex Pescaru commented · Feb 15 at 07:43 AM

Hi @XZv

I believe you may have a dodgy CAN bus controller. Happens.

I am saying this because if it was a problem with the software it would appear in many more cases from the thousands equipment on the field.

Of course, in many cases engineers could implement in software workarounds for the hardware problems.

Anyway, glad that you are in the right direction and begin to see the light at the end of the tunnel. :-))

Alex

0 ·

xzv Alex Pescaru commented · Feb 15 at 07:47 AM

Hi Alex,

I saw other cases in this forum already. I posted a link above. And I also exchanged the Multiplus and BMS with a replacement device, so it’s not a sibgle occurence.

I guess it could be that particular combination of hardware. But if it were in the majority of cases it would already be fixed :)

But I‘m glad I could identify at least a temporary workaround.

0 ·

Answer 6 · 2024-02-15T06:52:07Z

mvader (Victron Energy) answered · Feb 15 at 06:52 AM

Hey @XZv , thank you for all reports and information. We’re looking into this and will come back to you.

6

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv commented · Feb 15 at 07:07 AM

Thanks in advance!

0 ·

ejrossouw commented · Feb 15 at 10:37 AM

There may be a link with this? VictronConnect on Android, MP 800VA using the MK3 dongle. Does not accepts the actual value.

0 ·

strange-behaviour.mp4 (6.9 MiB)

xzv commented · Feb 22 at 06:36 AM

Good morning @mvader (Victron Energy) ,

any updates on this? Could you reproduce it in your lab?

Thanks!

0 ·

ejrossouw xzv commented · Feb 22 at 07:56 AM

Pylontech confirmed V1.6 is the latest firmware for the Force L2. Maybe get in touch with the reseller for Pylontech's local support information relevant to your location and also try the Pylontech Support on facebook.

0 ·

xzv ejrossouw commented · Feb 22 at 08:19 AM

I have doubt that it's on Pylontech's side. As stated above, two different Victron models either can deal with it, or not. As the software is the same, I conclude it must be a difference in the Victron hardware and its implication on the CAN communication.

0 ·

ejrossouw xzv commented · Feb 22 at 08:25 AM

I agree ;)

0 ·

question