Lost connection to MultiPlus from RPi 3b+

Question

question

questvent asked Mar 10, '21

Lost connection to MultiPlus from RPi 3b+

I lost connection to my MultiPlus 12/3000/120-50 120V, which I had connected to the RPi 3b+ via an MK3-USB. It has disappeared before a couple of weeks ago, but a combination of rebooting the MP, unplugging the MK3-USB for a while, and rebooting the RPi made it appear again. This time it seems that no matter what I do, it will not show up again.

I connected it to a laptop running VE.Configure 3 and was able to communicate with it that way, so the MK3-USB does still work...

So, things I have tried this time are:

1. Disconnect all devices (BMV-712, MPPT 150/85, MK3-USB), connect them back to different ports, which worked before when either the BMV-712 or MPPT disappeared. Also left the MK3-USB totally disconnected from CAT5 and USB for a while. Also tried JUST the MK3-USB and no other devices in each of the 4 USB ports on the RPi. No change.

2. Updated RPi from version 2.60 to version 2.63. No change.

3. Used the physical switch on the MP to power it off, waited several minutes and powered back on again.

4. Used VE.Configure on a laptop to pull settings, modify them, and push them back. Used the virtual control panel to turn it off and on again using the laptop.

I'm a software engineer, I'm okay-comfortable with signing on using SSH and looking at config files, etc. Does anyone have any suggestions that I can try? Thanks for any help that you can provide.

MultiPlus Quattro Inverter Charger Raspberry Pi Bluetooth MK3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) answered · Mar 20, 2021 at 05:31 PM

Hi,

Try a different mk3-usb. Perhaps yours lost the special value in the eeprom of the ftdi-usb-serial IC that makes for it be recognised to be an MK3.

could indeed be that VEConfigure doesn’t use that to detect it.

i’ve heard of that eeprom loosing its value once or twice before.

here is how my records show that an MK3USB should look like:

lsusb -v on a mk3-usb

Bus 001 Device 009: ID 0403:6015 Future Technology Devices International, Ltd Bridge(I2C/SPI/UART/FIFO) 
Device Descriptor: 
 bLength                18 
 bDescriptorType         1 
 bcdUSB               2.00 
 bDeviceClass            0 (Defined at Interface level) 
 bDeviceSubClass         0  
 bDeviceProtocol         0  
 bMaxPacketSize0         8 
 idVendor           0x0403 Future Technology Devices International, Ltd 
 idProduct          0x6015 Bridge(I2C/SPI/UART/FIFO) 
 bcdDevice           10.00 
 iManufacturer           1 FTDI 
 iProduct                2 MK3-USB Interface 
 iSerial                 3 HQ1639PKSJU 
 bNumConfigurations      1 
 Configuration Descriptor: 
   bLength                 9 
   bDescriptorType         2 
   wTotalLength           32 
   bNumInterfaces          1 
   bConfigurationValue     1 
   iConfiguration          0  
   bmAttributes         0x80 
     (Bus Powered) 
   MaxPower               90mA 
   Interface Descriptor: 
     bLength                 9 
     bDescriptorType         4 
     bInterfaceNumber        0 
     bAlternateSetting       0 
     bNumEndpoints           2 
     bInterfaceClass       255 Vendor Specific Class 
     bInterfaceSubClass    255 Vendor Specific Subclass 
     bInterfaceProtocol    255 Vendor Specific Protocol 
     iInterface              2 MK3-USB Interface 
     Endpoint Descriptor: 
       bLength                 7 
       bDescriptorType         5 
       bEndpointAddress     0x81  EP 1 IN 
       bmAttributes            2 
         Transfer Type            Bulk 
         Synch Type               None 
         Usage Type               Data 
       wMaxPacketSize     0x0040  1x 64 bytes 
       bInterval               0 
     Endpoint Descriptor: 
       bLength                 7 
       bDescriptorType         5 
       bEndpointAddress     0x02  EP 2 OUT 
       bmAttributes            2 
         Transfer Type            Bulk 
         Synch Type               None 
         Usage Type               Data 
       wMaxPacketSize     0x0040  1x 64 bytes 
       bInterval               0 
Device Status:     0x0000 
 (Bus Powered)

11

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

questvent commented · Mar 20, 2021 at 06:07 PM

Yeah, mine shows up as an FT232EX under iProduct. I showed the output of that command in another comment below in case it helps.

Unfortunately I only have the one MK3-USB. I was trying not to have to replace it, but knowing that it is possible for VEConfig to just use it without the correct IC identifier helps.

I just connected it to the laptop, ran VEConfig, and it pulled up the device. I then ran VictronConnect on the same laptop, and it is not seen by VictronConnect.

My system isn't really large enough to justify 2 of the MK3-USB adapters. If anything, I'd replace the RPi with an actual Victron hardware device with native VE.Direct and VE-Bus ports and not use the USB adapters. However, I'll order another MK3-USB for the sake of troubleshooting.

I do have a usb-serial interface that I've used to flash micro-controllers, but I assume the process of getting the correct eeprom code and flashing it to my MK3-USB would be an arduous task.

And thanks for the reply, I'll be sure to mark that as the answer once I get the replacement MK3-USB to verify.

0 ·

mvader (Victron Energy) ♦♦ questvent commented · Mar 20, 2021 at 06:16 PM

If you have the time, what you can try is change the ft232 data.

0 ·

mvader (Victron Energy) ♦♦ mvader (Victron Energy) ♦♦ commented · Mar 20, 2021 at 06:19 PM

Ps you could also try and get a new one under warranty. But with the Rpi being unsupported and all, I’m not 100% sure how easy you’d get through the process.

0 ·

Kevin Windrem questvent commented · Mar 20, 2021 at 06:17 PM

It seems the MK3 is DEFECTIVE so you should be able to return it for replacement or credit.

0 ·

mvader (Victron Energy) ♦♦ Kevin Windrem commented · Mar 20, 2021 at 06:45 PM

i’m only 90% sure it is - and there is the unlikely chance that venus os or or the user breaks it.

but yes you’re right Kevin.

@questvent if you want, ask for a new one and if necessary point to this thread.

0 ·

mvader (Victron Energy) ♦♦ questvent commented · Mar 20, 2021 at 06:22 PM

Ps2 I found this for you:

ftdi-control --usbid=1-1.2 --ee-init --ee-description="USB-RS485 Cable" --ee-manufacturer="VictronEnergy BV" --ee-serial="1"

we run that during manufacturing of the Octos.

Notes:

A) you’ll need to change that example. Running that as is will make venus os think its a rs485 cable. :-).

B) at your own risk. Make sure to not have other devices, such as vedirect usb cable, connected on your rpi at the same time.

C) I dont know if you’ll have that ftdi-control binary available on your rpi. What I do know is that at the ftdi website you can also download a windows equivalent for it.

0 ·

questvent mvader (Victron Energy) ♦♦ commented · Mar 20, 2021 at 06:32 PM

Awesome, thank you! Thanks for the warnings, I will ensure to unplug the other devices.

Looks like the binary you referenced is on the VenusOS 2.60:

root@raspberrypi2:/dev# find / -name ftdi-control
/usr/bin/ftdi-control

0 ·

mvader (Victron Energy) ♦♦ questvent commented · Mar 20, 2021 at 07:12 PM

Ps112: I’m super curious now ofcourse if you can fix it with that binary.

0 ·

questvent mvader (Victron Energy) ♦♦ commented · Mar 20, 2021 at 07:18 PM

mvader, you're totally awesome. I had to get the serial number for the MP from the VEConfig app to complete the parameters, but it's fixed now. Hopefully 4-ever.

root@raspberrypi2:~# ftdi-control -L
1-1.1.2  : FT232EX / FTDI

ftdi-control --usbid=1-1.1.2 --ee-init --ee-description="MK3-USB Interface" --ee-manufacturer="FTDI" --ee-serial="HQ1927JXDSB"
root@raspberrypi2:~# reboot -n

SHOWS UP NOW!!

root@raspberrypi2:~# tail -f /var/log/serial-starter/current
@40000000605648ee27d8c864 *** CCGX booted (0) ***
@40000000605648ef17d7d02c serstart starting
@40000000605648ef18023f74 INFO: loading config file /etc/venus/serial-starter.conf
@40000000605648ef23f2cc04 INFO: loading config file /data/conf/serial-starter.d
@40000000605648f01e5ce93c INFO: Create daemontools service vedirect-interface.ttyUSB0
@40000000605648f125c68a34 INFO: Create daemontools service mk2-dbus.ttyUSB1
@40000000605648f2304cff74 INFO: Create daemontools service vedirect-interface.ttyUSB2
@40000000605648f6273f727c INFO: Start service vedirect-interface.ttyUSB0
@40000000605648f72963a654 INFO: Start service mk2-dbus.ttyUSB1
@40000000605648f833d7b0bc INFO: Start service vedirect-interface.ttyUSB2

Thanks again, you the best.

0 ·

1616267279408.png (14.8 KiB)

1616267677616.png (785 B)

mvader (Victron Energy) ♦♦ questvent commented · Mar 20, 2021 at 07:39 PM

Hah cool! Well done.

Very welcome.

And my apologies ofcourse that this was necessary. Its for now a mystery to me how those ICs can lose their eeprom / what makes it loose their eeprom.

0 ·

questvent mvader (Victron Energy) ♦♦ commented · Mar 20, 2021 at 08:27 PM

I can tell you that it's all installed in an RV with a 30Amp AC system, and our microware seems to be taking more and more current to start up over time. It's often the case that we're either plugged into shore, or using the genset, and the microwave will cause an inverter overload. I then reset the system through VenusOS to get power restored.

I was initially puzzled by this because we're not taking 30A at those times, so how can it overload? I only recently found that I can disable the 'Dynamic Current Limit', and also now I just switch it to 'Charger Only' when on the genset or shore, and it no longer happens. But it was apparently trying to help ease the power draw on shore and overloaded frequently.

So, possibly that is a contributing factor, overloaded inverter and reset via VenusOS. Something to keep in mind.

0 ·

Modifications Space

Node Red Space

Answer 1 · 2021-03-11T09:31:35Z

Peter Buijs - NL answered · Mar 11, 2021 at 09:31 AM

Maybe take a look at dbus command line

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

questvent commented · Mar 11, 2021 at 03:41 PM

I tried to view the settings and also to just connect directly, but didn't seem to find the device:

root@raspberrypi2:/opt/color-control/mk2vsc# /opt/color-control/mk2vsc/mk2vsc -r  -s dbus://com.victronenergy.vebus.ttyO1/Interfaces/Mk2/Tunnel
000.008 ERR.dbus-serial: veDbusAddRemoteService failed
000.008 ERR.init: cannot open 'dbus://com.victronenergy.vebus.ttyO1/Interfaces/Mk2/Tunnel'

root@raspberrypi2:/opt/color-control/mk2vsc# /opt/color-control/mk2vsc/mk2vsc -r -s /dev/ttyO1
000.000 ERR.serial: could not open com-port
000.000 ERR.init: cannot open '/dev/ttyO1'

I used the following resources to get these commands:

https://www.victronenergy.com/live/open_source:ccgx:commandline

https://github.com/victronenergy/venus/wiki/dbus-api

0 ·

Answer 2 · 2021-03-13T17:09:34Z

questvent answered · Mar 13, 2021 at 05:09 PM

I think it might have something to do with a hidden setting where it thinks that ttyUSB0 should be a GPS. There is a file in /tmp called ttyUSB0.prog that just contains the string 'GPS', and if I delete it, it will refresh to vedirect for just a moment, then switches back to GPS again. I don't have a GPS connected. Is there a way to change the setting so that it defaults back to vedirect?

root@raspberrypi2:/tmp# cat ttyUSB0.prog
vedirect
root@raspberrypi2:/tmp# cat ttyUSB0.prog
gps

3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

questvent commented · Mar 13, 2021 at 05:33 PM

That file, /tmp/ttyUSB0.prog, it's just switching constantly between GPS and look like vedirect, so it's probably just a passive reading of what the system thinks is on ttyUSB0.

root@raspberrypi2:/tmp# tail -f ttyUSB0.prog
vedirect
gps
rect
gps
rect
gps
rect

So I guess I'll keep looking for the cause. I know that actual Victron devices just connect directly to VE.direct and VE.dbus without the USB adapters, so my issue is probably just limited to RPi-based devices. But it would be really nice to continue using the RPi instead of investing $300 in yet another device to fix the issue...

0 ·

questvent commented · Mar 13, 2021 at 06:40 PM

Looking further into it as I find where to look...the system seems to think that the mk3-usb interface is something called a FT232EX.

root@raspberrypi2:/dev/serial/by-id# lsusb -v

Bus 001 Device 004: ID 0403:6015 Future Technology Devices International, Ltd Bridge(I2C/SPI/UART/FIFO)
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x0403 Future Technology Devices International, Ltd
  idProduct          0x6015 Bridge(I2C/SPI/UART/FIFO)
  bcdDevice           10.00
  iManufacturer           1 FTDI
  iProduct                2 FT232EX
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower               98mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              2 FT232EX
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

Note that I have 3 devices, 2 are ve.direct USB interfaces, and one is a MK3-USB. There are 3 devices showing up in /dev/serial/by-id, two seem to be ve.direct and the other is this FT232EX.

root@raspberrypi2:/dev/serial/by-id# ls -ltr
lrwxrwxrwx    1 root     root            13 Jan  1  1970 usb-VictronEnergy_BV_VE_Direct_cable_VE2NI3UV-if00-port0 -> ../../ttyUSB1
lrwxrwxrwx    1 root     root            13 Jan  1  1970 usb-VictronEnergy_BV_VE_Direct_cable_VE28DT5F-if00-port0 -> ../../ttyUSB2
lrwxrwxrwx    1 root     root            13 Jan  1  1970 usb-FTDI_FT232EX-if00-port0 -> ../../ttyUSB0

0 ·

questvent commented · Mar 20, 2021 at 05:06 PM

Haven't really gotten anywhere on this so far, RPi still can't see the mk3-usb interface. I re-flashed the storage with 2.60, wiping it completely with balena-etcher. I also re-flashed to 2.63 with no difference, but since it worked most reliably for me on 2.60, it's on that for now.

The mk3-usb interface still wouldn't show up, so I imported the settings.xml from the old install.

I looked at the logs for the port, serial-starter seems to be hunting between gps and vedirect:

root@raspberrypi2:/var/log/serial-starter# tail -f current
@40000000605615a115c03acc INFO: Start service vedirect-interface.ttyUSB2 once
@40000000605615a620a2f13c INFO: Start service gps-dbus.ttyUSB2 once
@40000000605615af2d9a7234 INFO: Start service vedirect-interface.ttyUSB2 once
@40000000605615b43878a84c INFO: Start service gps-dbus.ttyUSB2 once
@40000000605615be09cc3784 INFO: Start service vedirect-interface.ttyUSB2 once
@40000000605615c3143dd0ec INFO: Start service gps-dbus.ttyUSB2 once
@40000000605615cc21aa109c INFO: Start service vedirect-interface.ttyUSB2 once
@40000000605615d12c7566ac INFO: Start service gps-dbus.ttyUSB2 once
@40000000605615da39ff213c INFO: Start service vedirect-interface.ttyUSB2 once
@40000000605615e0092506bc INFO: Start service gps-dbus.ttyUSB2 once
@40000000605615e916345d44 INFO: Start service vedirect-interface.ttyUSB2 once

Those are the default ports in the settings file. The log for vedirect on this port:

root@raspberrypi2:/var/log/vedirect.ttyUSB2# tail -f current
@4000000060561a6823d43fb4 003.001 INF.task: done in 003.001
@4000000060561a6d37e85ecc vedirect-dbus 3.50
@4000000060561a703807410c
@4000000060561a70380750ac -------- dumping backlog -------
@4000000060561a7038075c64 000.025 INF.task: using ttyUSB2 as unique service part
@4000000060561a703807681c 000.025 INF.VE.Direct: Setting status to VdUnknown
@4000000060561a70380777bc 000.036 INF.settings: connected to settings on dbus (com.victronenergy.settings)
@4000000060561a7038078b44 003.002 ERR.values: timeout connecting device
@4000000060561a70380a13b4 003.002 INF.task: done in 003.002

So, it can't see the device through the port.

Again, the mk3-usb works every time on a laptop using the older VEConfig tool. So it seems to still work, just something is different with how Venus OS on the RPi is interacting with it.

0 ·

Answer 3 · 2021-03-13T19:05:59Z

Kevin Windrem answered · Mar 13, 2021 at 07:05 PM

Could you have a bad/flakey USB port on the PI or cable connecting to the MK3? Try a different port and different cable just to rule them out. I guess the port on the MK3 could be flakey also.

Do you have any other USB devices that could contribute to the maximum USB power provided by the PI? You get a total of 1.2 amps between all 4 USB ports. The MK3 and USB Ve.Direct adapters use minimal power: 90 mA each if I recall correctly.

USB connections are assigned to /dev/ttyxxx on what should be considered random.

You might look at bus-spy to see if the Multi is even showing up on dBus (which it probably is not).

/sys is a file system that provides access to all peripherals. You might look through that.

Most connections made at boot show up in dmesg.

dmesg | grep "MK" will tell you where the system found the MK3 (if any). You can then grep for the usb port number for more info, including which /dev/ttyXXX it attached to this physical port.

I've had no issues with my PI 4 connecting to the MK3, two Ve.Direct adapters, a USB CANbus adapter and a GPS.

Come back here for more help if you need it.

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

questvent commented · Mar 13, 2021 at 07:56 PM

Thanks for the suggestions, I'll look in the areas that you've mentioned. I did check dmesg as you suggested, and it just shows the wrong device (FT232EX) being detected at boot.

Your suggestions about the hardware are good ones. When this happened a few days ago, I powered everything off and just tried the system with the MK3-USB\Multi connected and nothing else, but it still didn't recognize it. I tried connecting the MK3 to each port. I also tried using a laptop with it and it worked fine with the laptop, but maybe that method is more tolerable to bad connections?

My 3 devices were working great for the better part of a year before this happened, although I've had either the BMV or the MPPT drop off in a similar manner over the years before getting the Multi a year ago. Usually just switching ports fixes it, albeit with a different identifier that sort of messes up the VRM statistics, but not a big deal.

From threads that I've read, there's some added complexity with the MK* devices, it's tougher for the system to identify them reliably along with GPS devices and others. So unfortunately there seem to be some process that's getting stuck now that something changed, not sure what that something was since I was on 2.60 firmware since September and it was fine until a week or so ago.

I'll try to swap the USB and CAT5 cables out to see if that changes anything. Thanks again.

0 ·

question