VM3 meter via ethernet - problems after reboot

Q&A and troubleshooting
, , ,
1

Hi all,

We have a Cerbo GX on Venus 3.72 and a VM3 energy meter on fw v1.11.

The VM3 meter is configured with an EV Charger role and connected via ethernet.

When the Cerbo reboots, the VM3 does not re-connect with the Cerbo automatically. I have to go in to the Victron Connect app, select the meter from the detected device list in the “Local” tab, and then it is detected by the Cerbo.

I feel I must have something configured incorrectly… could anyone share some wisdom?

I believe the same behavior occurs when the VM3 reboots as well, while the Cerbo remains on. Sometimes the VM3 won’t show in Victron Connect until after I reboot it a couple of times.

Once it’s detected, there are no problems - the connection remains solid.

I have “Automatic scanning” enabled in the Modbus menu.

Thanks,

AP

2

Are you by any chance using both the wifi and ethernet connections on the Cerbo?
If you use DHCP, have you tried using a static address instead?

3

Hi @A-P ,

The VM-3P75CT is announces itself using mDNS, no need to enable the automatic scanning.
Can you try if it helps to disable automatic scanning?

Once the Cerbo has seen the meter, you still need to enable it on the Modbus Devices->Discovered Devices menu.
But since you’ve mentioned that it sometimes works, I suppose you already enabled it.

4

Hi @mbosma @nickdb thanks for the replies.

  1. I just disabled ‘automatic scanning’
  2. I’ve assigned static DHCP leases to both the VM3 and the Cerbo (previously it was just the VM3)
  3. I did have both WiFi and Ethernet active — so I just disabled WiFi.

After rebooting the Cerbo, the VM3 is no longer being shown on the home screen. When going into the Modbus menu, nothing shows in Discovered devices. In Saved devices, I see the saved IP address for the VM3. I just removed it, just to start fresh. Still nothing shows in the Discovered devices menu.

I am not on site at the moment, though I bet the VM3 shows up in the Victron Connect app, and when I interact with it via the app it’ll show up in the Cerbo. I don’t know why this would be the case…

I ran a packet dump on the VM3 (IP 172.19.25.22) below. The Cerbo IP is 172.19.25.23. This seems to indicate the VM3 is indeed broadcasting and it seems the Cerbo is picking this up? I don’t know why there are multiple ARP requests from the Cerbo and no responses from the VM3…

This dump was from the network router, so I don’t think we’d expect to see much traffic beyond the broadcasts (given all the traffic is switched within the same VLAN).

core1:~$ tcpdump -i any host 172.19.25.22
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
12:28:25.572331 eth0  M   IP 172.19.25.22.mdns > mdns.mcast.net.mdns: 0- [0q] 7/0/0 PTR Victron-VM-3P75CT-HQ24149EXAN._victron-energy-meter._
udp.local., (Cache flush) SRV Victron-VM-3P75CT-HQ24149EXAN.local.:502 0 0, PTR Victron-VM-3P75CT-HQ24149EXAN._vedirect._tcp.local., (Cache f
lush) SRV Victron-VM-3P75CT-HQ24149EXAN.local.:19200 0 0, (Cache flush) TXT "victron=1" "available=1", (Cache flush) A 172.19.25.22, (Cache f
lush) TXT "victron=1" "prodid=A1B1" "serial=HQ24149EXAN" "appver=1.11.ff" "bootver=1.6.ff" "mode=app" "updatable=1" (700)
12:28:25.572331 eth0.20 M   IP 172.19.25.22.mdns > mdns.mcast.net.mdns: 0- [0q] 7/0/0 PTR Victron-VM-3P75CT-HQ24149EXAN._victron-energy-meter
._udp.local., (Cache flush) SRV Victron-VM-3P75CT-HQ24149EXAN.local.:502 0 0, PTR Victron-VM-3P75CT-HQ24149EXAN._vedirect._tcp.local., (Cache
 flush) SRV Victron-VM-3P75CT-HQ24149EXAN.local.:19200 0 0, (Cache flush) TXT "victron=1" "available=1", (Cache flush) A 172.19.25.22, (Cache
 flush) TXT "victron=1" "prodid=A1B1" "serial=HQ24149EXAN" "appver=1.11.ff" "bootver=1.6.ff" "mode=app" "updatable=1" (700)
12:28:28.800967 eth0  B   ARP, Request who-has 172.19.25.22 tell 172.19.25.23, length 46
12:28:28.800967 eth0.20 B   ARP, Request who-has 172.19.25.22 tell 172.19.25.23, length 46
12:28:29.838444 eth0  B   ARP, Request who-has 172.19.25.22 tell 172.19.25.23, length 46
12:28:29.838444 eth0.20 B   ARP, Request who-has 172.19.25.22 tell 172.19.25.23, length 46
12:28:30.878414 eth0  B   ARP, Request who-has 172.19.25.22 tell 172.19.25.23, length 46
12:28:30.878414 eth0.20 B   ARP, Request who-has 172.19.25.22 tell 172.19.25.23, length 46
12:28:34.470866 eth0  M   IP 172.19.25.22 > mdns.mcast.net: igmp v2 report mdns.mcast.net
12:28:34.470866 eth0.20 M   IP 172.19.25.22 > mdns.mcast.net: igmp v2 report mdns.mcast.net
^C
10 packets captured
61 packets received by filter
0 packets dropped by kernel

EDIT: The ARP requests without a reply got me thinking…

  1. It seems I can ping the VM3 from the router’s shell, but NOT from the Cerbo’s shell
  2. The network switch to which the Cerbo is connected DOES have the VM3 in its ARP table (the VM3 is on an upstream switch). I rebooted to switch to confirm it wasn’t a cached entry (though it may be propagated by the router’s cache, I don’t know how this works in detail).

So I guess the question is… why isn’t the VM3 responding to the ARP request by the Cerbo specifically…

5

In the new OS you can choose which interface is internet facing and which isn’t.
In the past we needed to set a 0.0.0.0 gateway on the ethernet port so the GX didn’t get confused.

6

All that VictronConnect does, is run a mDNS query. That might trigger the process on the Cerbo. But the meter also announces itself periodically which t hen should be picked up by the Cerbo.

Can I have a quick peek in the logs of your system to see if there is anything obvious?
I see you already have enable remote support, assuming the system’s name is “Gr*ch”.

7

Thanks @mbosma - yes feel free to poke around!

8

I did not find anything obvious in the logs. I do see some mDNS records from the meter but it doesn’t seem to respond when queried.

Currently, I’m thinking it might be network related, because you cannot ping and my attempt to open a tcp port also fails:

root@victron-cerbo-gx:\~# nc -vz 172.19.25.22 19200
nc: connect to 172.19.25.22 port 19200 (tcp) failed: No route to host

Can you try to power cycle the meter and perhaps (temporarily) move it to the same switch as the Cerbo?

9

Hi @mbosma

I just got back on-site.

  1. Prior to doing anything, I confirmed I still could not ping the VM3 from the Cerbo. I checked /proc/net/arp and it showed the entry for 172.19.25.22 but with the mac address of all zeros. So for some reason, the Cerbo wasn’t able to ARP the MAC address for the VM3.
root@einstein:~# cat /proc/net/arp
IP address       HW type     Flags       HW address            Mask     Device
172.19.25.22     0x1         0x0         00:00:00:00:00:00     *        eth0
  1. I simply opened the VictronConnect app on the local network, which showed the Cerbo and the VM3 meter. That alone fixed the issue.

Very interesting observation:

  1. From a working state, if I briefly power cycle the VM3, everything remains working (e.g., off for ~5-10 seconds — just long enough after Venus realizes the meter is missing)
  2. From a working state, if I slowly power cycle the VM3, the issue occurs again (e.g., off for a few minutes)

I’m quite confused!

So what have we learned:

  1. Confirmed mDNS broadcast packets make their way from VM3 → Cerbo
  2. Confirmed ARP request broadcast packets make their way from Cerbo → VM3
  3. ARP responses seem to be getting from the VM3 → Cerbo AFTER the VictronConnect app opens once (I would need to install tcpdpump on the Cerbo to better understand this — is this possible?)

What I don’t know yet:

  1. Can other devices on the same switch as the Cerbo resolve the MAC address of the VM3 (which is on an upstream switch), while this problem is occuring?
  2. How opening the VictronConnect could possibly affect ARP responses from the VM3…?
  3. Does putting them on the same switch affect the behavior here? (this test is harder to run, given the physical set up)
10

OK so I think it was indeed a switch issue. To resolve, I simply disconnected and reconnected the network cable between a downstream “leaf” switch to its upstream “spine” switch. The VM3 was connected to one “leaf” switch, and the Cerbo was connected to another “leaf” switch. Both “leaf” switches had the same STP priority, which may have caused some weirdness….thanks again for the help.

11

Good to hear it has been solved.