V3.60~39 dbus mqtt devices not working - Scarthgap packages unavailable

After upgrading to ~39 I noticed that the dbus-mqtt-devices doesn’t work anymore. A reinstall fails because the package repository at updates.victronenergy.com for Scarthgap is not yet available so I’m missing some necessary Python3 modules now.

dbus-mqtt-devices: Setup in /data/drivers/dbus-mqtt-devices-0.8.0 started
dbus-mqtt-devices: Temporarily enable writing to root partion
dbus-mqtt-devices: Checking to see if Python's Pip is installed
/usr/bin/python: No module named pip
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all/Packages.gz, wget returned 8.

Browsing to https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all results in a 404 Not found.

Same for me:

root@raspberrypi4:/data# ./TODO_after_firmware_upgrade.sh
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all/Packages.gz, wget returned 8.
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/cortexa7hf-neon-vfpv4/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/cortexa7hf-neon-vfpv4/Packages.gz, wget returned 8.
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/raspberrypi4/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/raspberrypi4/Packages.gz, wget returned 8.
 * opkg_prepare_url_for_install: Couldn't find anything to satisfy 'python3-pip'.
./TODO_after_firmware_upgrade.sh: line 3: pip3: command not found

Best would be if you open an issue on GitHub that the needed packages are shiped directly with the driver.

I understand your reply but I don’t see how a missing repository (Scarthgap) from Victron could be a third-party issue. These packages are also needed for a regular Python module installation with opkg which also fails. Opkg update failing seems to be related.

I agree with Remco.
For Release the Scarthgap is not available at all, but for Candidate packages are available, but if I understand it correctly, the Scarthgap is missing the following packages:

packagegroup-venus-optional-packages-dev-….ipk
packagegroup-venus-optional-packages-…ipk

Hope somebody can solve/add the packages.

1 Like

It’s about stability. If you want a stable plugin/driver, you ship the needed modules with it. If not then you have those kind of problems with external dependencies. Therefore this would be a stability improvement for the driver and it would not depend on Venus OS version.

Let’s not enter a discussion on library dependancies and third party software. On an OS level (Linux in particular) it’s very common to depend on libraries and modules delivered by the OS itself. Especially if it’s as generic as Python, PIP, pyYaml and other modules in use by tons of other plugins and/or software. It’s not common to package them in third party software as it will become a mess. Maybe @freakent has an opinion on this as maintainer of the driver.

Anyway, of course this is Beta software so I’m convinced this will be solved somewhere in the future as it is clearly a missing package mirror for Scarthgap. It’s not a big issue as long as it’s beta software.

Thanks for tagging me on this. First thing I would say is anyone having problems with dbus-Mqtt-devices should start a discussion on the projects GitHub page.

This is the first time I’ve heard of Scarthgap and Yocto. From what I can see from the output of the setup script it looks like it can’t find pip. That would definitely be an issue. I have tried many different ways of including yaml parser with the project and pip was the what worked best.

If I get time this week, I’ll try updating my test raspberry pi to the beta of Venus OS and take a look.

1 Like

Same problem here, cant install custom packages:

root@raspberrypi4:~# bash /home/root/velib_python-master/test/run_setup.sh
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all/Packages.gz, wget returned 8.
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/cortexa7hf-neon-vfpv4/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/cortexa7hf-neon-vfpv4/Packages.gz, wg                                                                  et returned 8.
Downloading https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/raspberrypi4/Packages.gz.
 * opkg_download_backend: Failed to download https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/raspberrypi4/Packages.gz, wget return                                                                  ed 8.
kernel-module-ipip-5.10.110-rpi-venus-6 - 5.10.110-rpi-venus-6-r0
util-linux-pipesz - 2.39.3-r0
 * opkg_prepare_url_for_install: Couldn't find anything to satisfy 'python3-pip'.
/home/root/velib_python-master/test/run_setup.sh: line 9: pip3: command not found
/home/root/velib_python-master/test/run_setup.sh: line 11: pip3: command not found

The problem lies in the repository URL. Scarthgap packages are uploaded in the candidate directory at Index of /feeds/venus/candidate/packages/scarthgap which of course makes sense. The Opkg repository on 3.60~39 points to the release directory which is obviously incorrect.

root@einstein:/etc# cd opkg/
root@einstein:/etc/opkg# ls -l
-rw-r--r--    1 root     root           287 Mar  9  2018 arch.conf
-rw-r--r--    1 root     root           925 Mar  9  2018 opkg.conf
lrwxrwxrwx    1 root     root            47 Mar  9  2018 venus.conf -> /usr/share/venus-feed-configs/opkg-release.conf

root@einstein:/usr/share/venus-feed-configs# cat opkg-release.conf
src/gz all https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/all
src/gz cortexa7hf-neon-vfpv4 https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/cortexa7hf-neon-vfpv4
src/gz einstein https://updates.victronenergy.com/feeds/venus/release/packages/scarthgap/einstein

As this is on a read-only filesystem, I cannot change it unfortunately.

Hi all,

To install any packages you first need to change your rootfs to be read/write.

And secondly, when having a release candidate installed, you always need to run /opt/victronenergy/swupdate/set-feed.sh candidate.

And thereafter “opkg update”.

Usually omitting that set-feed.sh is not really an issue. But now, with Dunfell in release, and Scartgaph in candidate it is mandatory again.

Hope this helps.

And to learn more:

And for making it read/write, see:

Matthijs

Unfortunately no python3 packages yet?
image

Hi Matthijs @mpvader,

I agree with what you are saying, the problem is we are missing some libraries in the Index of /feeds/venus/candidate/packages/scarthgap feed. Most of us are looking for paho.mqtt, which used to be in packagegroup-venus-optional-packages-dev-….ipk in the Dunhill feed.
I have searched the Packages file, and could not find any reference to pip3 for Scarthgap, but did find it in Dunhill. So I think this file (library) is missing in the Scarthgap feed.
I am not an experienced Linux programmer, but I hope this helps.

do ‘python -m ensurepip’ if you need pip

Solution for pip3 in v3.60~39:

/opt/victronenergy/swupdate-scripts/set-feed.sh candidate
python -m ensurepip
Looking in links: /tmp/tmpps2wovk9
Processing /tmp/tmpps2wovk9/pip-24.3.1-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-24.3.1

Unfortunately, that won’t work in my case:

root@einstein:~# /opt/victronenergy/swupdate-scripts/set-feed.sh candidate
Switching swu feed to candidate
resizing /dev/mmcblk1p3
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/mmcblk1p3 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mmcblk1p3 is now 327680 (4k) blocks long.

Switching opkg feed to candidate
root@einstein:~# python -m ensurepip
/usr/bin/python: No module named ensurepip
root@einstein:~#

What do you see if you run this command: opkg list | grep pip

You should see something like this:

root@raspberrypi4:~# opkg list | grep pip
kernel-module-ipip-5.10.110-rpi-venus-6 - 5.10.110-rpi-venus-6-r0 - ipip kernel module
perl-module-io-pipe - 5.38.2-r0 - perl module io-pipe
ppp-password - 2.5.0-r0 - Plugin for PPP to get passwords via a pipe
python3-ensurepip - 3.12.8-r0 - Support for bootstrapping the pip installer
util-linux-pipesz - 2.39.3-r0 - util-linux pipesz

Did you do: opkg update ?

Yes, I did an opkg update. “python -m ensurepip” won’t work but there is indeed an ensurepip package in the opkg repository.

root@einstein:/etc/opkg# opkg install python3-ensurepip
Installing python3-ensurepip (3.12.8) on root
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/python3-ensurepip_3.12.8-r0_cortexa7hf-neon-vfpv4.ipk.
Configuring python3-ensurepip.

Unfortunately, this crashes:

root@einstein:/data/drivers# python3 -m ensurepip
Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/__main__.py", line 22, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_internal/cli/main.py", line 11, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 10, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_internal/cli/main_parser.py", line 9, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_internal/build_env.py", line 18, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_internal/cli/spinners.py", line 9, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_internal/utils/logging.py", line 13, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_vendor/rich/console.py", line 47, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_vendor/rich/themes.py", line 1, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_vendor/rich/default_styles.py", line 3, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_vendor/rich/style.py", line 8, in <module>
  File "/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl/pip/_vendor/rich/color.py", line 3, in <module>
ModuleNotFoundError: No module named 'colorsys'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.12/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 284, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmp7i9tfw64/pip-24.3.1-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmp7i9tfw64\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

And the dbus-mqtt-devices setup still cannot find pip.

root@einstein:/data/drivers# ./dbus-mqtt-devices-0.8.0/bin/setup.sh
dbus-mqtt-devices: Setup in /data/drivers/dbus-mqtt-devices-0.8.0 started
dbus-mqtt-devices: Checking to see if Python's Pip is installed
/usr/bin/python: No module named pip
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/all/Packages.gz.
Updated source 'all'.
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/cortexa7hf-neon-vfpv4/Packages.gz.
Updated source 'cortexa7hf-neon-vfpv4'.
Downloading https://updates.victronenergy.com/feeds/venus/candidate/packages/scarthgap/einstein/Packages.gz.
Updated source 'einstein'.
 * opkg_prepare_url_for_install: Couldn't find anything to satisfy 'python3-pip'.
dbus-mqtt-devices: Pip install module dependencies
/usr/bin/python: No module named pip
dbus-mqtt-devices: Set up Victron module libraries
dbus-mqtt-devices: Set up device service to autorun on restart
dbus-mqtt-devices: Adding device service to /data/rc.local
dbus-mqtt-devices: Setup complete

I give up for now and will revert back again until this is solved. Don’t want to manually fiddle too much in the repository and installed Python modules.

This is strange because I was able to install pip3 on v3.60~39, everything works now, Installed all other required packages with pip3, no problem. pip3 does work though in v3.60~39.

root@raspberrypi4:~# pip3 --version
pip 24.3.1 from /usr/lib/python3.12/site-packages/pip (python 3.12)

Use ensurepip instead:

root@raspberrypi4:~# /opt/victronenergy/swupdate-scripts/set-feed.sh candidate
Switching swu feed to candidate
resizing /dev/mmcblk0p3
resize2fs 1.47.0 (5-Feb-2023)
The filesystem is already 2578688 (4k) blocks long.  Nothing to do!
Switching opkg feed to candidate

root@raspberrypi4:~# python -m ensurepip
Looking in links: /tmp/tmpps2wovk9
Processing /tmp/tmpps2wovk9/pip-24.3.1-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-24.3.1