question

mgoldbach avatar image
mgoldbach asked

MQTT - only see system/0/Serial & how do I find which MQTT Broker Server my GX connects too?

Hi,

I have activated MQTT and have been able to successfully connect to mqtt.victronenergy.com. However the only topic where I see activity in is /system/0/Serial. I have run a topics scan and I have tried other documented topics like /system/0/Ac/Consumption/Total/Power. Am I missing something?

MQTT
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

6 Answers
shalombi avatar image
shalombi answered ·

Other solution all the way down

OK since I was having the same issue but the answers were a bit cryptic here is the solution. Most of this information and other things can be found on the main GitHub page https://github.com/victronenergy/dbus-mqtt

Make sure MQTT is turned on in the CCGX service menu.

Commands included are for windows, they are almost identical for other systems.


1- Find the right server

Victron changed the mqtt servers from a single mqtt.victronenergy.com to multiple instance mqttXX.victronenergy.com where XX is a number from 0-127.

To be able to access your data, first step is to figure out which broker to talk to. For this you need you VRM portal ID, you can get that from the VRM portal in settings, on CCGX or by polling the local MQTT.

With the VRM portal ID you shove it in the code described here : https://github.com/victronenergy/dbus-mqtt#determining-the-broker-url-for-a-given-installation

Here is an online version of the script, just replaced the portal id string at the top with yours and it will print the right broker:

https://onlinegdb.com/ByqR27fEL

OK, so now you have a number and you know which MQTT broker to connect to.


Fun Fact : if you connect to mqtt.victronenergy.com and not mqttXX.victronenergy.com it will succeed but there won't be any data beyond the serial no matter what you do.


2 - Download the certificate file from here : https://github.com/victronenergy/dbus-mqtt called

venus-ca.crt


3 - Connect to the Victron VRM MQTT with the following command

mosquitto_sub.exe -v -I myclient_ -t '#' -h mqttXX.victronenergy.com -u <your_username> -P <your_password> -p 8883 --cafile .\venus-ca.crt

Replace the XX with your broker and your username and password (those you use to connect to the VRM portal) with yours. make sure to point to the certificate file downloaded in step 2.

Alternatively use the GUI MQTT.fx, settings are pretty much the same, enable SSL/TLS and select TLSv1.2 and the CA certificate file from step 2.


OK so now you are connected and can read a single value, the VRM portal ID.

Where is your data you ask? Well that the whole keep-alive thing. The CCGX isn't reporting any MQTT values back to VRM until you wake it up even though you turned on the setting on the CCGX.


4 - Ok so let's poll the CCGX locally every 5 seconds to keep him awake.

On the github page they give a nice script, basically publish a message with nothing in it regularly.

https://github.com/victronenergy/dbus-mqtt#keep-alive

Here is my powershell equivalent

while (1){  .\mosquitto_pub.exe -m ' ' -t 'R/YOUR_PORTAL_ID/system/0/Serial' -h LOCAL.IP -p 1883 ; sleep 5;}

To do this you need to be on the same local network as the CCGX and enter its IP address after -h

If you are not on the same LAN you can spam the online MQTT for similar results, values are identical as those from step 3:

while (1){ .\mosquitto_pub.exe -m ' ' -t 'R/YOUR_PORTAL_ID/system/0/Serial' -h mqttXX.victronenergy.com -u <your_user> -P <your_password> -p 8883 --cafile .\venus-ca.crt ; sleep 5;}


Now all of a sudden you're going to get flooded with messages on the MQTT subscription from step 3. \o/


You can't stop the polling or after 60 seconds the data is going to stop, find a way to keep that script running on some machine.

My needs are to poll once every 5 minutes from my HassIO installation, will now see how to tie it all nicely together.


Can someone explain the whole keep-alive design decision?

That MQTT isn't on by default I get, but if I want MQTT on why should I have to spam the server to keep it going? Wouldn't an only local always on MQTT or VRM with a set interval reporting like the VRM make more sense?


[UPDATE AND SOLUTION]


OK, so want you want to control the MQTT to do things nicely?


So, really I don't want to be running a polling script that also spams victron MQTT when all I want is to know the percentage of charge of my battery...

This is definitely hacky and won't survive a firmware update, I'll look into making something more solid later but it gives you full control of MQTT behavior.


1 - Get super user and SSH into your Venus


2 - Change MQTT bridges

vi /usr/sbin/start-mosquito

2.a - Add your own MQTT bridge file

Add this line so that it will copy your own MQTT bridge configuration file, change cloud_mqtt_bridge.conf to something that makes you happy. We will create this file later.

ln -sf /data/conf/mosquitto.d/cloud_mqtt_bridge.conf /run/mosquitto/cloud_mqtt_bridge.conf 

2.b [OPTIONAL] stop updating victron VRM:

If you want to get rid of the Victron VRM MQTT and setup your own or just have it ignore the VRM on local polling comment this line:

#    ln -sf $bridge_conf $bridge_link 

So this looks something like Before:

if [ "$(get_setting /Settings/Services/MqttVrm)" = 1 ]; then                                     ln -sf $bridge_conf $bridge_link                                                     else                                                                                          rm -f $bridge_link                                                                       fi   

And After having also disabled victron:

if [ "$(get_setting /Settings/Services/MqttVrm)" = 1 ]; then                                 #ln -sf $bridge_conf $bridge_link                                                           ln -sf /data/conf/mosquitto.d/cloud_mqtt_bridge.conf /run/mosquitto/cloud_mqtt_bridge.conf else                                                                                         rm -f $bridge_link                                                                       fi   


3 - Make the MQTT keep alive always

Edit this file:

vi /service/dbus-mqtt/run  

And add the --keep-alive 0 option at the end

Before:

exec softlimit -d 100000000 -s 1000000 -a 100000000 /opt/victronenergy/dbus-mqtt/dbus_mqtt.py --init-broker

After:

exec softlimit -d 100000000 -s 1000000 -a 100000000 /opt/victronenergy/dbus-mqtt/dbus_mqtt.py --init-broker --keep-alive 0  


4 - Create your own bridge configuration:

Create the file you named in step 2.a, in my case:

vi /data/conf/mosquitto.d/cloud_mqtt_bridge.conf

Add to it your mqtt bridge config, this one is for cloudmqtt

connection cloudmqtt address mXX.cloudmqtt.com:17342 remote_username <Username> remote_password <Password> clientid <ID> try_private false start_type automatic topic N/VRMID/system/0/Dc/Battery/Soc out


Ok so, change username password, address... to fit your config and then we need to chose what is going to get sent.

[option 1] sync everything to your new broker, ok if you are doing this locally or have no bandwidth limit on the connection + broker

topic # out

[option 2] Or choose the items you're interested in, add as many lines as you want

topic N/VRMID/system/0/Dc/Battery/Soc out


Conclusion

So this will solve most issues as you can configure however you like.

This has no impact on the VRM portal functionality and also if you disable report MQTT to Victron is not spammy.

For me this solves all my needs, getting the 2-3 values I wanted for home assistant to take some decisions. If you want to control the CCGX from MQTT just add some in parameters.


I do recommend removing the victron MQTT bridge unless you need it as it will send a significant amount of data every second. Or if you want to use their MQTT only export the parameters you really need instead of everything.

6 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ben avatar image ben ♦ commented ·

I don't know the rationale for the keepalive, but I have a guess.

The original implementation is pretty basic/simple. It leverages the mqtt broker to propagate all the messages locally and to the VRM server in the cloud. This makes configuration really simple and serves a variety of different use cases.

Those messages, however, can be quite verbose, they are transmitted uncompressed, and they are pretty big and fine-grained compared to the data VRM gathers. It's a substantial data egress cost if everyone started leaving MQTT on full blast everywhere. (Moreover, on metered connections, it could be even more expensive.)

I do think it makes sense just to have a local-only MQTT that doesn't have the extra hoop-jumping. But setting that up has probably just not come into scope. They are pretty busy and have limited dev resources, and we MQTT consumers are still a very small group.

You could easily do it yourself and share your changes. (You can even document your results on that same wiki.)

0 Likes 0 ·
frederic-olenergies avatar image frederic-olenergies commented ·

So i follow you steps after opening a remote MQTT broker, but nothing happens while I was expecting a flood of messages..

Is it normal that you don't point out the new cloud_mqtt_bridge.conf file anywhere is the venus GX ? The symlink is not used in the start-mosquitto script so it should not impact much the Venus GX...

0 Likes 0 ·
shalombi avatar image shalombi frederic-olenergies commented ·

The symlink is run when the service is started. The service should be started if you have MQTT enabled in the CCGX.

Test this by hand and check if the file is getting symlinked or not.

0 Likes 0 ·
katia-khennoune avatar image katia-khennoune commented ·

Hello,

I am a beginner in this field and I have a question.

Do I have to download all the source code from github to raspberry pi for this to work? thank you.

0 Likes 0 ·
peternielsen avatar image peternielsen commented ·

@shalombi so did you find a way so the changes survives an update by using /data to run a script that does the changes necessary ? Some details are explained here :https://www.victronenergy.com/live/ccgx:root_access#hooks_to_install_run_own_code_at_boot

0 Likes 0 ·
mkh avatar image mkh commented ·

Thank you shalombi, this is first description of MQTT topic here in the forum which helps me to get data from VRM via MQTT.

Only confusing is keep-alive as this is not keep-alive implemented in MQTT clients but own Victron implementation. I understend, why to send over thousand of topic to VRM when data are not displayed.

For testing, if you want to see data in your MQTT client, is not necessary to publish periodically "keep-alive" topic, just open your dashboard in VRM and messages are comming automatically.

0 Likes 0 ·
wkirby avatar image
wkirby answered ·

I think you are supposed to connect to the local broker, which is the CCGX / VenusGX it's self if you have enabled MQTT on the device.

8 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mgoldbach avatar image mgoldbach commented ·

As I understand documentation, once MQTT is activated it should also communicate with Victron's cloud server. So I should be able to pub/subscribe to either one. @mvader (Victron Energy Staff) can you advise?

0 Likes 0 ·
ben avatar image ben ♦ mgoldbach commented ·

You can use either one. The issue is you haven't "woken up" the server.

0 Likes 0 ·
mgoldbach avatar image mgoldbach ben ♦ commented ·

Thank you @ben. I connect to mqtt.victronenergy.com. I publish to R/<ID>/system/0/Ac/Consumption/Total/Power and subscribe to N/<ID>/system/0/Ac/Consumption/Total/Power. That's what you meant about waking up the server, right? I don't see any payload coming back from the publish. I do have full permissions to this site on vrm.victronenergy.com. Is there something else?

0 Likes 0 ·
mgoldbach avatar image mgoldbach mgoldbach commented ·
Update: I was successfully able to publish then subscribe on another system. So I understand the process. Thank you! The system that works well for me is in the US. The ones that are not working as expected are located in Kenya. I have verified that I have full control and the MQTT is activated. Do you have any guidance?
0 Likes 0 ·
ben avatar image ben ♦ mgoldbach commented ·

Maybe a network reachability issue. Did you try sshing into them directly?

0 Likes 0 ·
mgoldbach avatar image mgoldbach ben ♦ commented ·

I'm not sure why, but I have now been able to successfully wake up all my systems mqtt servers by publishing a read to a battery monitor topic. So now everything is working as expected. Thank you guys.

0 Likes 0 ·
ben avatar image ben ♦ mgoldbach commented ·

Glad to hear it. If you experience issues again going forward, one thing that you might keep in mind is that only the Serial number topic is guaranteed to be available for that initial read. A standard implementation of keep-alive, thus, should probably be polling that specific (guaranteed present) topic.

0 Likes 0 ·
mgoldbach avatar image mgoldbach ben ♦ commented ·

Perfect. Thank you.

0 Likes 0 ·
ben avatar image
ben answered ·

Yes, you're missing something. Read the documentation carefully regarding "keep alive." You need to do more to get it to work.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

manowell avatar image
manowell answered ·

The documentation says: (emphasis mine)

dbus-mqtt

A python script that publishes values from the D-Bus to an MQTT broker. The script also supports requests from the MQTT broker to change values on the local D-Bus. This script only works with the D-Bus interface defined for use with the Color Control GX (CCGX).

By default, dbus-mqtt will connect to a Mosquitto MQTT broker running on the CCGX itself. The broker is accessible on the local network at TCP port 1883. Furthermore the broker is configured to forward all communication to the central Victron MQTT broker (mqtt.victronenergy.com), which allows you to monitor and control your CCGX over the internet. You'll need your VRM credentials to access this broker. See 'Connecting to the Victron MQTT server' below.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

georgelza avatar image
georgelza answered ·

... ok also battling here, have opened 2 MQTT clients, can subscribe to # and getting serial number back, and nothing more, have subscribed from a second client, have published from a 2nd client, no change, I'm expecting a flood of messages....

How do I wake mqtt on my venusGX up to start pushing messages at me ?

G

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ben avatar image ben ♦ commented ·

Show us your code?

0 Likes 0 ·
katia-khennoune avatar image katia-khennoune commented ·

Hello,

I followed all the steps to recover some data but the mosquitto_sub command returns me error: connection timed out

Do you know why?

K

0 Likes 0 ·
pnd001 avatar image
pnd001 answered ·

Hi everyone, I know this is an old thread but if anyone out there is still struggling with this; have a look at this blog post: https://www.imval.tech/index.php/blog/victron-mqtt-server-bridging..

It's a much cleaner way of getting this to work.

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

David Borg avatar image David Borg commented ·

I come to the "Wake up the MQTT script on Venus" then i dont know where to put the text?

0 Likes 0 ·