Why did the SSL certificate change after update to Venus OS 3.64?

DIY
, , ,
1

I am integrating my Cerbo into Home Assistant using MQTT. MQTT using TLS required me to extract the self-signed CA certificate from the Cerbo. That same certificate has worked since December last year. It did not change during previous updates to Venus OS.

Last night I updated to Venus OS 3.64, then this morning noticed that MQTT is no longer working. Booting back into 3.62 (the version I had before), did not help.

Turns out during the update the SSL certificate was changed. After downloading the new certificate and installing it in my MQTT broker, everything works again now.

The question to developers: why did the certificate change during this particular update? Can this be avoided in future? If it is unavoidable during certain updates: can that fact please be prominently listed in the release notes, so we know what to expect?

2

Here’s an interesting new observation: I turned off my system completely today in order to install a smart shunt. The cerbo was powered off, disconnected from battery and MPPTs. When I turned everything back on, the SSL certificate had changed again!

My gut feel tells me now: some functions that are only supposed to run the very first time a Cerbo boots are running at every reboot in Venus OS version 3.64.

Yes, a brand new Cerbo needs to generate a certificate. An existing one does not need to do that. Developers: please fix this bug.

3

The challenge with customisation of a system is the engineers are unlikely to consider the niche use case. This has to be the first time this has been raised.
There have been quite a few security enhancements and changes lately. We can ask the question but can’t promise a favourable response.

4

Do you have any mods running on the Cerbo?
Maybe a mod causes a full data partition which could cause issues?

5

I agree, if it was an intentional change that improves security, I’ll happily accept that and will figure out a way to live with it.

What surprised me is that I did not find anything in the release notes. That’s why I am still leaning towards the more likely explanation that it was an unintentional change, i.e. that some setting that was used for testing made it into the production build by accident.

It would be good if whoever made that change sees this thread and replies either way.

6

No mods on the Cerbo.

I am using the “Large” image. I had intended to check out Node-Red, but I didn’t get around to it yet. Or rather: I didn’t need to, because Home Assistant does everything I need via MQTT. Mostly: adjusting the DVCC current to slow down charging when the battery gets close to full, and increasing it again when SOC drops.

7

Can you confirm the /data partition (or any partitions for that matter) aren’t full.
On the current version you can check via the UI in settings → general → support.
It isn’t clear from your reply. thanks.

Also, FYI as per new standards:

The maximum certificate lifetime is going down:

  • From today until March 15, 2026, the maximum lifetime for a TLS certificate is 398 days.
  • As of March 15, 2026, the maximum lifetime for a TLS certificate will be 200 days.
  • As of March 15, 2027, the maximum lifetime for a TLS certificate will be 100 days.
  • As of March 15, 2029, the maximum lifetime for a TLS certificate will be 47 days.
8

In Settings → General I don’t find “Support”.
Perhaps this info found under Settings →VRM refers to the same partition?

image
image1421×193 15.8 KB

Interesting info about certificate expiries. Ok, I’ll write a script to regularly update the certificate downloaded to my MQTT broker. That will cover both this current issue and the future reduction in certificate lifetimes.

9

What version are you on? It might be called modifications in an earlier version but this pretty much since 3.63 iirc. VRM often refers to the logging SD card assuming you have one.

IMG_4918
IMG_49181302×853 107 KB

10

The certificate is not supposed to change every time, naturally. It’s generated when absent (first time) or regenerated when deemed corrupt/unusable. Something may be going wrong in this detection.

Are you able to ssh into the cerbo and give me the output of openssl x509 -text -in /data/keys/mosquitto.crt.

About the certificate lifetime BTW: that’s irrelevant. The MQTT certificate is a long-lived self-signed certificate to be used by non-browsers, meaning the two parties who enforce that lifetime rule (certificate authorities and browser makers) don’t come into play.

11

Thanks for persisting. I found it. :slight_smile:

Called “Modification Checks” for me. Still plenty of storage, 1GB available.

image
image1481×888 69.7 KB

12

image
image1166×548 82.6 KB

I did a bit more digging. This is the directory they keys are stored. Only the mosquitto key was renewed, the host key was not changed since it was first created back in November.

image
image1308×309 64.6 KB

I also checked /var/log/mosquitto/current. It only shows ‘Generating a RSA private key’ one single time, and I believe that log file dates back to when I first started the Cerbo and enabled MQTT. The number of times it shows ‘*** CCGX booted (0) ***’ right at the end is consistent with what I’d expect since back then.

I am at a loss why the key was changed. I even did a reboot again (not power off), this time the key remained unchanged.

13

The correct logs would actually be in /var/log/flashmq. Can you look there as well, including the rotated files (so not just current)? Use ta64nlocal to see the actual times.

For example:

grep -A 5 -i -F '*** starting flashmq' /var/log/flashmq/*
14

Unfortunately there are only 4 files in /var/log/flashmq, all last modified today. The only thing grep finds is from today’s reboot:

image
image1520×258 62 KB

Is there a way to increase the number of rotated files that are kept in future?

tai64nlocal has been a great tip though. It helped confirm 2 things: the only time mosquitto generated a key was the day I got the Cerbo:

2024-11-06 11:29:45.999476500 unable to load Private Key
2024-11-06 11:29:46.041894500 Generating a RSA private key
2024-11-06 11:29:46.049319500 .........+++++
2024-11-06 11:29:46.351976500 .....................................................+++++
2024-11-06 11:29:47.897775500 writing new private key to '/data/keys/mosquitto.key'

But: the modification time of the key file is exactly the time of Friday’s reboot:

2025-08-15 05:22:05.124365500 *** CCGX booted (0) ***
2025-08-17 06:49:25.102505500 *** CCGX booted (0) ***
root@einstein:/var/log/mosquitto# ls -la /data/keys/
drwx------ 2 root root 4096 Jul 14 02:55 .
drwxr-xr-x 13 root root 4096 Nov 6 2024 ..
-rw-r--r-- 1 root root 1123 Aug 15 05:22 mosquitto.crt
-rw------- 1 root root 1704 Aug 15 05:22 mosquitto.key

I guess for now, with the old flashmq logs gone, we have reached the limits of the investigation? At least if it happens again, I can go and check that log straight away.

15

Ok, after a couple more reboots, the issue did happen again. Here’s the relevant part of the output:

/var/log/flashmq/current:@4000000068a1b92a0daf807c *** starting flashmq ***
/var/log/flashmq/current-@4000000068a1b92b058f550c /usr/sbin/start-flashmq:
line 28: 1487 Hangup /usr/bin/flashmq --test-config --co
nfig-file "$tempconf"
/var/log/flashmq/current-@4000000068a1b92b134382cc .+......+.......+..+.+..

No idea what 1487 Hangup might mean, and Google was of no help unfortunately. But the testing the test configuration file fails, and as a result new key is generated again.

16

Venus platform sends a HUP signal to reload FlashMQ settings. It does this with ‘killall -HUP flashmq’. But, apparently it does this even before FlashMQ is started, and targets the temporary flashmq used for testing the cert. Because in that mode it doesn’t have a signal handler, it aborts.

It’s an unexpected and very coincidental race condition.

You can temporarily just disable the check in /usr/sbin/start-flashmq(note that it requires changing the root fs to read-write):

# diff -u start-flashmq.bak start-flashmq
--- start-flashmq.bak   2025-07-31 08:02:30.000000000 +0000
+++ start-flashmq       2025-08-17 11:56:55.641155749 +0000
@@ -20,7 +20,7 @@
 
 # FlashMQ actually creates an SSL context with key and cert, so it will detect
 # it if they are invalid.
-if ! /usr/bin/flashmq --test-config --config-file "$tempconf" ||
+if false ||
    ! openssl verify -CAfile $cert $cert ||
    ! openssl x509 -noout -checkend 31536000 -in $cert; then
     openssl req -subj /CN=venus.local -newkey rsa:2048 -nodes \

We’ll get it fixed.

2 Likes
17

Thank you so much for figuring out this puzzle so quickly. Amazing!

If I may make a suggestion: it would have been easier if more log files were kept. If possible, please increase either the file size or number of files in /service/flashmq/log/run.

18

Sometimes the logs can be too short, but for you it only contained stuff of today? Can I see what your logs contain?

19

Yes sure.

Right now they full with the same 3 lines, repeated every few minutes. I closed the browser window that was connected straight to the Cerbo, and have had no new messages since.

I am pretty sure when I looked yesterday there were other messages as well, but those old files are gone by now. I’ll have another look tomorrow to see if anything else gets added now that I don’t have webui open.

@4000000068a30430372c4a5c [2025-08-18 10:44:54.925] [NOTICE] [main] Accepting connection from: address='127.0.0.1',
transport='TCP/Websocket/Non-SSL', fd=19
@4000000068a304303827a21c [2025-08-18 10:44:54.941] [NOTICE] [T 0] Client '[ClientID='guiv2a7d284e2c8738fcd', userna
me='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]' logged in s
uccessfully
@4000000068a304a607a0df8c [2025-08-18 10:46:52.127] [NOTICE] [T 0] Removing client '[ClientID='guiv2a7d284e2c8738fcd
', username='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]'. R
eason(s): Browser navigating away from page, socket disconnect detected
@4000000068a304a937d9d03c [2025-08-18 10:46:55.936] [NOTICE] [main] Accepting connection from: address='127.0.0.1',
transport='TCP/Websocket/Non-SSL', fd=19
@4000000068a304a9386bdedc [2025-08-18 10:46:55.946] [NOTICE] [T 0] Client '[ClientID='guiv2e03aa8e066c7cd21', userna
me='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]' logged in s
uccessfully
@4000000068a3051f0590c054 [2025-08-18 10:48:53.092] [NOTICE] [T 0] Removing client '[ClientID='guiv2e03aa8e066c7cd21
', username='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]'. R
eason(s): Browser navigating away from page, socket disconnect detected
@4000000068a30522386eccdc [2025-08-18 10:48:56.946] [NOTICE] [main] Accepting connection from: address='127.0.0.1',
transport='TCP/Websocket/Non-SSL', fd=19
@4000000068a30522396c012c [2025-08-18 10:48:56.963] [NOTICE] [T 0] Client '[ClientID='guiv2877bd2186e743528', userna
me='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]' logged in s
uccessfully
@4000000068a3059808a105c4 [2025-08-18 10:50:54.143] [NOTICE] [T 0] Removing client '[ClientID='guiv2877bd2186e743528
', username='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]'. R
eason(s): Browser navigating away from page, socket disconnect detected
@4000000068a3059b37955ccc [2025-08-18 10:50:57.932] [NOTICE] [main] Accepting connection from: address='127.0.0.1',
transport='TCP/Websocket/Non-SSL', fd=19
@4000000068a3059b388e91ac [2025-08-18 10:50:57.946] [NOTICE] [T 0] Client '[ClientID='guiv29a2fae4b58b66b43', userna
me='', fd=19, keepalive=60s, transport='TCP/Websocket/Non-SSL', address='127.0.0.1', prot=3.1, clean=0]' logged in s
uccessfully

20

Yeah, those kind of log messages easily flood out the rest unfortunately.

BTW, tip: aside from one back-tick, there is also the the three back-tick option. This is better for multi-line code-like blocks, and makes a log dump a bit more readible.