Very strange. Only difference is that you’re running on raspberrypi4 while I’m running 3.60~39 on a CerboGX (Einstein). I hope this will be solved someday.
Its very weird, because I have 2x Raspberry Pi’s 4 (VenusOS/ESS) systems, It worked on one of them, and doesn’t work on other Pi. Same errors you are having after the upgrade to the latest beta. I saw these packages installed on one system not the other. I cant see how this is possible, unless I installed these required packages previously long time ago during Pi CAN-BUS Hat extension setup. I have wasted enough time by pulling packages manually and just reverted one system back to v3.53~2.
I think you are right Russ, look at your image of your screen output, it found a package in the /tmp folder. I guess you had downloaded it some time ago. Therefor not really strange.
Anyway I have also reverted back as I am confident the feeds will be corrected by adding the missing libraries. The mr-manuel mqtt drivers and freakent drivers rely on this library and I think enough people are using it.
1 Like
Hi all,
I just did a quick test using v3.60~46 and the missing opkg packages are now available. Of course you still need to select the candidate feed.
However paho.mqtt is now also a new version (upgrade from Dunfell to Scartgap) and needs a modification in your driver. In principle it is an easy fix of adding mqtt.CallbackAPIVersion.VERSION1 as the first parameter when initializing the mqtt client in your driver. If modifying code is not your thing, please wait for the developer of your driver to do it for you and/or raise an issue on their github site. Possibly they will even upgrade to callback api V2, which involves some more code changes.
For now, Victron has done what they can do by adding the missing packages. Thank you for that.
It is up to the driver developers now to update their drivers.
1 Like
The packages are now available indeed. Setting up dbus-mqtt-devices seems to work until it tries to install PyYAML due to a missing tomllib.
root@einstein:/data/drivers/dbus-mqtt-devices-0.8.0# bin/setup.sh
dbus-mqtt-devices: Setup in /data/drivers/dbus-mqtt-devices-0.8.0 started
dbus-mqtt-devices: Checking to see if Python’s Pip is installed
/usr/bin/python: No module named pip
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/all/Packages.gz.
Updated source ‘all’.
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/Packages.gz.
Updated source ‘cortexa7hf-neon-vfpv4’.
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/einstein/Packages.gz.
Updated source ‘einstein’.
Installing python3-xmlrpc (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-xmlrpc_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-image (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-image_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-compile (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-compile_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-plistlib (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-plistlib_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-difflib (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-difflib_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-pkg-resources (69.1.1) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-pkg-resources_69.1.1-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-unittest (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-unittest_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-setuptools (69.1.1) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-setuptools_69.1.1-r0_cortexa7hf-neon-vfpv4.ipk.
Installing python3-pip (24.0) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-pip_24.0-r0_cortexa7hf-neon-vfpv4.ipk.
Configuring python3-difflib.
Configuring python3-compile.
Configuring python3-plistlib.
Configuring python3-pkg-resources.
Configuring python3-unittest.
Configuring python3-setuptools.
Configuring python3-xmlrpc.
Configuring python3-image.
Configuring python3-pip.
dbus-mqtt-devices: Pip install module dependencies
Collecting PyYAML~=6.0 (from -r /data/drivers/dbus-mqtt-devices-0.8.0/requirements.txt (line 1))
Downloading pyyaml-6.0.2.tar.gz (130 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.6/130.6 kB 452.0 kB/s eta 0:00:00
Installing build dependencies … done
Getting requirements to build wheel … error
error: subprocess-exited-with-error× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [38 lines of output]
Traceback (most recent call last):
File “/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 353, in
main()
File “/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 335, in main
json_out[‘return_val’] = hook(**hook_input[‘kwargs’])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py”, line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File “/var/volatile/tmp/pip-install-itl2plgw/pyyaml_70c1c9168d55483c8a5f82b830459319/packaging/_pyyaml_pep517.py”, line 47, in _expose_config_settings
return real_method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/build_meta.py”, line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/build_meta.py”, line 304, in _get_build_requires
self.run_setup()
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/build_meta.py”, line 320, in run_setup
exec(code, locals())
File “”, line 333, in
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/init.py”, line 117, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py”, line 160, in setup
dist.parse_config_files()
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/dist.py”, line 652, in parse_config_files
pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py”, line 72, in apply_configuration
config = read_configuration(filepath, True, ignore_option_errors, dist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py”, line 108, in read_configuration
asdict = load_file(filepath) or {}
^^^^^^^^^^^^^^^^^^^
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/config/pyprojecttoml.py”, line 37, in load_file
from ..compat.py310 import tomllib
File “/var/volatile/tmp/pip-build-env-qapwm9ob/overlay/lib/python3.12/site-packages/setuptools/compat/py310.py”, line 7, in
import tomllib
ModuleNotFoundError: No module named ‘tomllib’
[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.note: This error originates from a subprocess, and is likely not a problem with pip.
We’re not in the green yet unfortunately.
1 Like
opkg install python3-tomllib should be able to workaround that I guess
1 Like
Can confirm, both of my systems work with v3.60~46. No problem. opkg install python3-tomllib indeed needed for my second system with the missing packages. Everything works now…
1 Like
Great, we’re one step further. So for now, I did the following:
Change package repository to “Candidate”:
/opt/victronenergy/swupdate-scripts/set-feed.sh candidate
The Python tomllib needs to be added manually:
opkg install python3-tomllib
Installation of dbus-mqtt-devices:
root@einstein:/data/drivers/dbus-mqtt-devices-0.8.0# ./bin/setup.sh
dbus-mqtt-devices: Setup in /data/drivers/dbus-mqtt-devices-0.8.0 started
dbus-mqtt-devices: Checking to see if Python's Pip is installed
pip 24.0 from /usr/lib/python3.12/site-packages/pip (python 3.12)
dbus-mqtt-devices: Pip install module dependencies
Collecting PyYAML~=6.0 (from -r /data/drivers/dbus-mqtt-devices-0.8.0/requirements.txt (line 1))
Using cached pyyaml-6.0.2.tar.gz (130 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting dataclasses~=0.6 (from -r /data/drivers/dbus-mqtt-devices-0.8.0/requirements.txt (line 2))
Downloading dataclasses-0.6-py3-none-any.whl.metadata (3.0 kB)
Downloading dataclasses-0.6-py3-none-any.whl (14 kB)
Building wheels for collected packages: PyYAML
Building wheel for PyYAML (pyproject.toml) ... done
Created wheel for PyYAML: filename=pyyaml-6.0.2-cp312-cp312-linux_armv7l.whl size=45425 sha256=f5715fdd2d9ba463cb21839dd68c4ebbbbe374e61ae922797c50a990fbba9199
Stored in directory: /data/home/root/.cache/pip/wheels/db/db/e2/4a1264f6c5192c518338cd8c226caae9f43c610fdb76c1ebcb
Successfully built PyYAML
Installing collected packages: dataclasses, PyYAML
Successfully installed PyYAML-6.0.2 dataclasses-0.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
dbus-mqtt-devices: Set up Victron module libraries
dbus-mqtt-devices: Set up device service to autorun on restart
dbus-mqtt-devices: Adding device service to /data/rc.local
Now, the installation is working fine but I’m running into the Paho MQTT version incompatibility:
ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details
As dbus-mqtt-devices is referring to the /opt/victronenergy/dbus-mqtt library, I’m not sure how and where to change the CallbackAPIVersion attribute so we’re not there yet. I think this is where the driver needs to be updated @freakent
1 Like
Can you provide more of the error message surrounding thge “ValueError:…” I’d like to see where this error is occuring.
Hi Jeroen,
Thanks but I already read those docs. My suspicion is that the error is actually in Victron’s code. My driver extends a class called MqttGObjectBridge which is part of the dbus-mqtt library that is superceeded by dbus-flashmq. The only place that I can see the mqqt client being initialised is in that class. That’s why I asked @remco to provide more detail from around the error, just to confirm.
Martin
Hi Martin,
This is the full error report in the logging:
@4000000067daba743197a7e4 *** starting dbus-mqtt-devices ***
@4000000067daba77037ea54c dbus_mqtt_devices v{} 0.8.0
@4000000067daba77037ec0a4 INFO:logger:Loglevel set to INFO
@4000000067daba77039c108c INFO:dbus_mqtt_devices:-------- dbus_mqtt_devices, v0.8.0 is starting up --------
@4000000067daba77051cff84 INFO:device_manager:Using portalId <<redacted>>
@4000000067daba77269bc03c Traceback (most recent call last):
@4000000067daba77269e40dc File "/data/drivers/dbus-mqtt-devices-0.8.0/dbus_mqtt_devices.py", line 83, in <module>
@4000000067daba7726b20f2c main()
@4000000067daba7726b84504 File "/data/drivers/dbus-mqtt-devices-0.8.0/dbus_mqtt_devices.py", line 65, in main
@4000000067daba7726c8cb7c handler = MQTTDeviceManager(
@4000000067daba7726cb86b4 ^^^^^^^^^^^^^^^^^^
@4000000067daba7726cce25c File "/data/drivers/dbus-mqtt-devices-0.8.0/device_manager.py", line 43, in __init__
@4000000067daba7726dc6aec MqttGObjectBridge.__init__(self, mqtt_server, CLIENTID, ca_cert, user, passwd, debug)
@4000000067daba7726e57f24 File "/data/drivers/dbus-mqtt-devices-0.8.0/ext/dbus-mqtt/mqtt_gobject_bridge.py", line 25, in __init__
@4000000067daba7726f4b5ac self._client = paho.mqtt.client.Client(client_id)
@4000000067daba7727001f8c ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@4000000067daba7727018ad4 File "/usr/lib/python3.12/site-packages/paho/mqtt/client.py", line 766, in __init__
@4000000067daba772731979c raise ValueError(
@4000000067daba7727334d1c ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details
Thanks, as suspected, it’s the MqttGObjectBridge.
Just to see if the workaround actually works, you could edit line 25 in /data/drivers/dbus-mqtt-devices-0.8.0/ext/dbus-mqtt/mqtt_gobject_bridge.py and change
self._client = paho.mqtt.client.Client(client_id)
to
self._client = paho.mqtt.client.Client(mqtt.CallbackAPIVersion.VERSION1, client_id)
If that works, I guess I could look to include an amended version of that class in my code. Or maybe Victron will accept a pull request with a suitable update?
It does raise another question, @jeroen, how much longer will Victron include the dbus-mqtt python library in VenusOS?
Martin
The change in the mqtt_gobject_bridge.py results in the following error:
@4000000067dac2373b530cc4 INFO:dbus_mqtt_devices:-------- dbus_mqtt_devices, v0.8.0 is starting up --------
@4000000067dac23801ad4c64 INFO:device_manager:Using portalId <<redacted>>
@4000000067dac2381c4c25f4 Traceback (most recent call last):
@4000000067dac2381c4ee12c File "/data/drivers/dbus-mqtt-devices-0.8.0/dbus_mqtt_devices.py", line 83, in <module>
@4000000067dac2381c622aac main()
@4000000067dac2381c68358c File "/data/drivers/dbus-mqtt-devices-0.8.0/dbus_mqtt_devices.py", line 65, in main
@4000000067dac2381c797f54 handler = MQTTDeviceManager(
@4000000067dac2381c7be0b4 ^^^^^^^^^^^^^^^^^^
@4000000067dac2381c7cc344 File "/data/drivers/dbus-mqtt-devices-0.8.0/device_manager.py", line 43, in __init__
@4000000067dac2381c8cf7b4 MqttGObjectBridge.__init__(self, mqtt_server, CLIENTID, ca_cert, user, passwd, debug)
@4000000067dac2381c96041c File "/data/drivers/dbus-mqtt-devices-0.8.0/ext/dbus-mqtt/mqtt_gobject_bridge.py", line 25, in __init__
@4000000067dac2381ca463cc self._client = paho.mqtt.client.Client(mqtt.CallbackAPIVersion.VERSION1, client_id)
@4000000067dac2381cab0ed4 ^^^^
@4000000067dac2381cb0621c NameError: name 'mqtt' is not defined
@4000000067dac2382e1952ac *** starting dbus-mqtt-devices ***
@4000000067dac23a37a5eefc dbus_mqtt_devices v{} 0.8.0
@4000000067dac23a37a60a54 INFO:logger:Loglevel set to INFOAh, sorry try this instead.
client = paho.mqtt.client.Client(paho.mqtt.client.CallbackAPIVersion.VERSION1)
1 Like
Looking at the logs, you seem to include your own copy. The project is archived and what is installed is dead code for us. I think it should simply be removed now.
Hi Jeroen,
During driver install, I create a soft link from /opt/victronenergy/dbus-mqtt to my driver directory. I did this to make sure I was always running the latest version of your library. And I didn’t want deal with any legal issues by distributing your code in my project.
@4000000067dad99724e620fc *** starting dbus-mqtt-devices ***
@4000000067dad99a1a6e74e4 dbus_mqtt_devices v{} 0.8.0
@4000000067dad99a1a6e903c INFO:logger:Loglevel set to INFO
@4000000067dad99a1aae21ac INFO:dbus_mqtt_devices:-------- dbus_mqtt_devices, v0.8.0 is starting up --------
@4000000067dad99a1d456e5c INFO:device_manager:Using portalId <<redacted>>
@4000000067dad99a38910a7c INFO:mqtt_gobject_bridge:[Init] Connecting to local broker
@4000000067dad99a3b1052bc INFO:device_manager:[Connected] Result code 0
@4000000067dad99c2351a61c Traceback (most recent call last):
@4000000067dad99c2370b354 File "/data/drivers/dbus-mqtt-devices-0.8.0/ext/velib_python/ve_utils.py", line 24, in exit_on_error
@4000000067dad99c2370ceac return func(*args, **kwargs)
@4000000067dad99c2370da64 ^^^^^^^^^^^^^^^^^^^^^
@4000000067dad99c23728fe4 File "/data/drivers/dbus-mqtt-devices-0.8.0/ext/dbus-mqtt/mqtt_gobject_bridge.py", line 71, in _on_socket_timer
@4000000067dad99c2372a36c if self._client.loop_write(10) != paho.mqtt.client.MQTT_ERR_SUCCESS:
@4000000067dad99c2372b30c ^^^^^^^^^^^^^^^^^^^^^^^^^^^
@4000000067dad99c2375df8c TypeError: Client.loop_write() takes 1 positional argument but 2 were given
This also breaks unfortunately 
This doesn’t look like an easy thing to completley debug. This isn’t my code so I’m not familiar enough to say if we fix this error there won’t be more.
If you want to try one more thing, edit line 71 and change
if self._client.loop_write(10) != paho.mqtt.client.MQTT_ERR_SUCCESS:
to
if self._client.loop_write() != paho.mqtt.client.MQTT_ERR_SUCCESS:
i.e.remove the 10 parameter.
1 Like
It is MIT licensed, so you won’t get into legal issues. Just fork it if you depend on it, fix it, and install that. There is no “latest version” any more since 3.20, it is no longer supported see GitHub - victronenergy/dbus-mqtt: Venus OS service mapping the D-Bus on Venus OS to MQTT.