Heavy Bandwidth usage after updating to 3.60 (MQTT/flashmq)

Modifications
, , , ,
1

Since getting updated to Venus 3.60 on my Cerbo, I’m using 500MB/day in bandwidth from my Cerbo to mqtt40.victronenergy.com

Prior to 3.60, I was using 50-85MB/day (still very high), but now it’s crazy high.

The IP address of my Cerbo is 172.30.76.5. I have enabled local SSH access, and temporarily installed a couple of binaries to verify what is causing the traffic (tcpdump and lsof).

image
image802×182 18.7 KB

Yesterday.

image
image775×141 10.6 KB

On the Cerbo, I verified that the IP address and process being used was flashmq.

PING mqtt40.victronenergy.com (18.185.161.164)
...
root@victron:/var/run/flashmq# netstat -tun | grep 18.185.161.164
tcp        0    558 172.30.76.5:46032       18.185.161.164:443      ESTABLISHED
root@victron:/var/run/flashmq# netstat -tun | grep 18.185.161.164
tcp        0    562 172.30.76.5:46032       18.185.161.164:443      ESTABLISHED
root@victron:/var/run/flashmq# netstat -tun | grep 18.185.161.164
tcp        0    374 172.30.76.5:46032       18.185.161.164:443      ESTABLISHED
root@victron:/var/run/flashmq# netstat -tun | grep 18.185.161.164
tcp        0    470 172.30.76.5:46032       18.185.161.164:443      ESTABLISHED
root@victron:/var/run/flashmq# netstat -tun | grep 18.185.161.164
tcp        0   1360 172.30.76.5:46032       18.185.161.164:443      ESTABLISHED

root@victron:/var/run/flashmq# cat vrm_bridge.conf
...
bridge {
  protocol_version mqtt5
  address mqtt40.victronenergy.com

...
}

image
image822×141 15.1 KB

In the above, the two connections on port 443 are for MQTT (flashmq), and the port 22 (SSH) connection is the reverse proxy my installer is using to monitor my system remotely from VRM.

On VRM, I have disabled real-time updates.

image
image423×199 7.18 KB

On my Cerbo, I have updates set to 30 minutes.

image
image1100×439 18.6 KB

As stated, this behavior is new since 3.60, as I was running 3.55 prior to a few days ago. The site in question has limited connectivity, so using 500MB/day means that in a few days I will no longer have access, as my connectivity is limited.

Thanks for any help you can provide. I am VERY comfortable debugging OS and configuration issues regarding services, but am new to solar configuration.

2

Followup:

I think I found the culprit. At around the same time that 3.60 was released, I installed venus-grafana (GitHub - victronenergy/venus-grafana: Default Grafana dashboards, for docker solution and Venus OS Large) and was connecting it from my non-remote site to VRM to keep track of things.

Unfortunately, it appears that if you connect to VRM to monitor the system (even if real-time is disabled), it ‘pulls’ data at a much higher rate from the monitored system, causing much higher data usage.

When I realized this this morning, I disabled the monitoring setup, and traffic has SIGNIFICANTLY decreased.

What would be nice would be the ability to pull the stored data in VRM without it negatively effecting the bandwith on the system, but at this point, at least I have what I believe to be a solution to reduce bandwidth on my remote system.

I’ll continue to monitor the bandwidth and update this topic if anything more comes out.

1 Like
3

Hi, nice detailed analysis!

The main purpose / reason of existence of Grafana is to store low interval data.

Retrieved via MQTT.

Extracting it from the VRM database instead, or getting the (for you now set to 30 minutes) normal VRM data transmits forwarded, is not implemented and also not trivial to implement

4

Thanks for the details. I’m now running venus-grafana and storing it on a local computer located at my remote instance for detailed analysis.

However, it does make me a bit more leery of using VRM, as doing so causes a huge data hit from my remote site. With that in place, what is the point of reducing the log interval if VRM is just going to be fetching the data in realtime whenever it’s used?

Thanks in advance!

5

The configured interval is how often the Gx should send data to the vrm database.

Totally separate from that is the mqtt stream of data. Which is on change, used for the real time feature in VRM, used by Grafana, and its active as soon as anyone of those tools “asks for it” (= sends keep alives).

6

In VRM, per installation, you can disable real time updates.

By doing that, it won’t set up the mqtt link, and then VRM uses no further data.

Note that that data link is only active while having your browser open.

I hope our VRM manual explains that setting in more detail? Can’t check now myself.

IMG_0603
IMG_06031179×1974 201 KB

7

So, if I just ‘use’ VRM (with real-time data disabled), it shouldn’t pull data in real-time from my remote site?

(ie; as long as I don’t hook up venus-grafana, I should be safe to monitor the remote site using VRM).

Thanks!

8

I had already disabled real-time updates (as mentioned in my original post), simply because of the limited bandwidth I am forced to deal with.

9

Ok.. but then I don’t understand this question?

VRM won’t fetch real time when you have real time updates disabled in VRM.

10

Sorry for the confusion. When using venus-grafana to connect to VRM with real-time updates disabled, the data usage was huge. I was not connected to my local box, I was connected to VRM, and this was causing a huge amount of data to get pulled from my remote to VRM and then over to my grafana instance.

My venus-grafana instance was only connected to VRM.

Because connecting to VRM was the cause of the data usage, that was why I was concerned about using VRM.

1 Like
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.