question

mvader (Victron Energy) avatar image
mvader (Victron Energy) asked

Victron Bluetooth advertising protocol

Good morning!


In 2021 I shared details of our Bluetooth GATT protocol. Which works for a few products.

Since then we came up with what I think is a better idea for 3rd party integration for read-out, which is using BLE advertising data, sometimes also referred to as BLE Broadcasting.

Thats the same mechanism behind what we commercially call “Instant read out over BLE”. Full Victron product compatibility list is here in the manual for the VictronConnect App.

This message is to share the protocol PDF of that more widely.

To use this data, you need to get your hands on the encryption key. It can be copied from VictronConnect, see my answer below with a few screenshots on where to find that.

60186-1693142210358.png


Alternatively, there is a trick to look into VictronConnect files on your computer. As documented by @netsrac27: in Library/Application Support/Victron Energy/Victron Connect you'll find a file called "Victron Connect.sqlite". Inside of this database you'll find a table called "advertisementKeys". It lists MAC address of the sensor and the advertising key. So there is a way to extract the encryption key from that database.


I’m quite sure that there is already code published by people that takes advantage of this.

Have a good weekend!

Matthijs

extra_manufacturer_data - 2022-12-14.pdf


Bluetooth
5 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Jason avatar image Jason commented ·
Is there any update on making the advertisement key available in VictronConnect? Especially hoping to get it in the IOS app...
2 Likes 2 ·
Hi @Jason,

The post has now been updated with the latest info on getting the advertisement key in VictronConnect.

0 Likes 0 ·
loadshedbuddycoza avatar image loadshedbuddycoza commented ·
I hope this paves the way for Venus OS 3 to be able to add the smartshunt and smart mppt to connect to the Rpi4 over Bluetooth.


I don't want to run Ve Direct to usb cables it's expensive and messy. And at times not practical.

1 Like 1 ·
kurtinge avatar image kurtinge loadshedbuddycoza commented ·
...and at least, the cable connector is really bad. Vibration and so on makes them to slip out
1 Like 1 ·
knauer avatar image knauer commented ·

Hey, very nice this advertisement. Would it be possible (is it planned or can it be taken as a feature request) to include also the state of the load output of the Solar Chargers into the advertisement message? Use case (just one out of many): Use this signal in automation software (i.e. home assistant) to set any states or switch any devices.

Further maybe also include panel Voltage and panel current?

That would make it perfect and no GATT would be neccesary any more (for my use cases).


Thank You very much!

0 Likes 0 ·
18 Answers
netsrac27 avatar image
netsrac27 answered ·

So happy you decided to publish this. Thanks!!!

I know it was a good choice to move all my solar chargers and battery monitors to Victron products :-)


2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

netsrac27 avatar image
netsrac27 answered ·

…and just in case someone is interested…yes, there is already some code available on GitHub to read the data:

https://github.com/keshavdv/victron-ble

With the new documentation, I‘m sure keshavdv (the project initiator) and other contributors (like myself) can work on a complete integration.

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

timh767 avatar image timh767 commented ·
I've spent all day trying to get this working (mostly trying to get the advertising key, which involved mutiple Linux installs. Victron Connect doesn't appear to install on Ubuntu on a RPi4b - worked ok on Ubuntu on a x86 laptop usb stick, got there in the end).


The integration installed ok, picked up my BMV712 and detected the name and mac address, accepted the advertising key, but I can't see any device or entity for the Victron and getting an error in the log.

I raised an issue on github but wondering if others have got it working?


1 Like 1 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·

Nice! Good to have that link.


Same library is used for a signalk plugin as well: https://www.npmjs.com/package/signalk-victron-ble


0 Likes 0 ·
fabians avatar image
fabians answered ·

Thank you I have updated my ESPHome (ESP32) integration to use this:

https://github.com/Fabian-Schmidt/esphome-victron_ble


Hoping the updated Victron App is soon available and will display encryption key and mac address. Was a bit painful to fetch them as the Windows app does not support Bluetooth connections.

Cheers,

Fabian

3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

steveuk71 avatar image steveuk71 commented ·

Hi

ESPHome is new to me, so forgive the question.

What is the 'ESPHome (ESP32) integration', can you use it with HomeAssistant and ESPHome to view the Victron BLE information directly, or do you need a ESP32 Hardware device also??

Steve


0 Likes 0 ·
fabians avatar image fabians steveuk71 commented ·
Correct you need an ESP32 to fetch the data.

This integration will make the data available to ESP Home. ESP Home has a nice and deep integration with HomeAssistant, but it works also very well without HA. In fact I don't have a HA and just use ESP Home with a display to quickly see all details.

I don't know if there as a native HomeAssistant to Victron integration.

0 Likes 0 ·
oldeddy avatar image oldeddy commented ·

Hi fabians,

currently I've got big trouble with getting the encryption key out of the VictronConnect application on Windows. As you've written windows has no Bluetooth capability in the app. How did you got it from the windows app?

Best regards

oldEdDy


0 Likes 0 ·
oldeddy avatar image
oldeddy answered ·

Hi mvader,

thanks for publishing the BLE format information! I'm waiting for this still a long time to read out my SmartSolar charger 100/50 via BLE. Is there any update on the VictronConnect app to get the advertisement keys and MAC directly from the app?

Best regards

oldEdDy

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

oldeddy avatar image
oldeddy answered ·

Hi all,


I got now my advertisment key from my solar charger with a Linux system. For all who want to now what's to do here a small instruction:

I used an old laptop (it has to be at least 64bit capable!) plus Asus USB-BT400 Bluetooth 4.0 stick, other sticks will do the job as well if they are BLE capable. Installed the current Linux Mint 21.1 with Xfce (Source: https://linuxmint.com/edition.php?id=304). Downloaded the VictronConnect Linux AppImage v5.83 (Released officially) from the Victron page https://www.victronenergy.com/live/victronconnect:beta. Made it executable with

 chmod +x *.AppImage


in the terminal and started the app with

./*.AppImage


Thanks Victron team for your description at https://community.victronenergy.com/questions/43667/victronconnect-for-linux-download-instructions.html.

After the first connection with the solar charger I followed the instruction from https://github.com/keshavdv/victron-ble. Installed sqlite3 and used the following command in terminal:

sqlite3 ~/.local/share/Victron\ Energy/Victron\ Connect/d25b6546b47ebb21a04ff86a2c4fbb76.sqlite 'select address,advertisementKey from advertisementKeys inner join macAddresses on advertisementKeys.macAddress == macAddresses.macAddress'


The result was the MAC and the advertisement key ;-).

Best regards

oldEdDy

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Jake Baldwin avatar image
Jake Baldwin answered ·

Wow! I'm so happy Victron is trying to actively work with the DYI/Developer community. It's no surprise I have only seen Victron controllers in camper van conversions.

I'd like to provide some additional context as the original document on how to decrypt the BLE advertisement data leaves quite a bit to be desired: https://community.victronenergy.com/storage/attachments/48745-extra-manufacturer-data-2022-12-14.pdf

Here is a practical example with some pitfalls I encountered:

1) Advertisement Key Retrieval: As far as I could tell there is no way to get the advertisement keys from the Windows App, iPhone app, or Android app. I was able to get the key from my friends Mac and I've read you can also get it from Linux but haven't tried it myself.

2) Raspberry Pi: I wasn't able to obtain the key from a Raspberry Pi either. The current Raspberry Pi OS BT driver won't pair or connect to the Victron Controller so you can't get the keys. The Victron maintained Raspberry Pi image, VenusOS, doesn't allow connecting to their devices via BT, only serial, which validates the issues I was having; even Victron couldn't figure it out!

However, the BT driver in Raspberry Pi OS can listen for the the BLE advertisement messages which is great if you've already obtained the keys.

3) BLE Advertisement Data: This was particularly unclear from the provided documentation. Here are 6 BLE advertisement data packets I got from one of my SmartSolar MPPT 100/50 charge controllers:

10 02 57 a0 01 00 02 20 b0 08 09 0f 73 44 a4 e6 6f 7d 00 24
10 02 57 a0 01 10 02 20 e4 c5 b2 08 2a b8 8d 83 33 7e 4f cc
10 02 57 a0 01 20 02 20 17 a5 b2 d8 02 bf ca 8c 4a c3 ad 39
10 02 57 a0 01 30 02 20 d2 de 94 01 9e 56 6e 03 d7 23 f1 b2
10 02 57 a0 01 40 02 20 58 bd 6b 2c ea ab 78 6d 7a 5b f1 4f
10 02 57 a0 01 50 02 20 b9 40 bc c1 66 e5 38 97 66 dc 4a d8

Byte [0] is the Manufacturer Data Record type and is always 0x10.
Byte [1] and [2] are the model id. In my case the 0x02 0x57 means I have the MPPT 100/50 (I forget where I found this info but it's always 2 bytes and it's not really needed for decryption)
Byte [3] is the "read out type" which was always 0xA0 in my case but i didn't use this byte at all

The first 4 bytes aren't mentioned in the provided documentation so it was difficult to figure out where the "extra data" started. Now we get into the bytes documented:
Byte [4] is the record type. In my case it was always 0x01 because I have a "Solar Charger"
Byte [5] and [6] are the Nonce/Data Counter used for decryption (more on this later)
Byte [7] should match the first byte of your devices encryption key. In my case this was 0x20.

The rest of the bytes are the encrypted data of which there are 12 bytes for my Victron device.

4) Decryption: If you have the Encryption Key and the Nonce/Data Counter (from bytes [5] and [6]) in the BLE advertisement data you can decrypt the packet. The description in the provided documentation is not the best. I had to use reverse engineer the Python library referenced above in this thread because I'm using C#. Here is the Python library function in question:

https://github.com/keshavdv/victron-ble/blob/841f5601800a70b35df910526f174427a5e39c12/victron_ble/devices/base.py#L388

Now there are a few issues here.
1) The Python library pads the data bytes to get a full AES block of 16 bytes, and the documentation refers to using "1 AES operation" to decrypt the data if less than or equal to 16-bytes. This suggests a block cipher algorithm. However, the AES-CTR is not a block cipher algorithm, it's a stream cipher algorithm. This means no block extension should be necessary because the data is decrypted byte by byte, that's what a stream cipher means. In my case 12-bytes in gives me 12-bytes out.


2) The "Nonce/Data Counter" referred to in the documentation is often referred to as the "Salt" or the "Initialization Vector" in encryption lexicon which made a google search difficult


3) Now this was the worst part! The "Nonce/Data Counter" (bytes [5] and [6]) are not only transmitted in the advertisement data LSB first (little endian), but must also be used in the decryption as a little endian value! The provided documentation doesn't mention this fact at all!!! I only figured this out from looking at the referenced github Python library. Now what does this actually mean in practice if you aren't using Python? Because if you run the python code in a debugger it sets up a new counter object in little endian format but it puts it in an obscure object where you don't have access to the underlying byte array. Take this BLE advertisement packet as an example:

10 02 57 a0 01 40 02 20 58 bd 6b 2c ea ab 78 6d 7a 5b f1 4f

Byte [5] and [6] are the Nonce/Data Counter, in this case it's 0x40 0x02. Put this Nonce/Counter/Salt/Initialization Vector a 16 byte array in little endian format it should look like this:

{ 0x40 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 }

4) AES counter mode isn't supported in C# but you can see this stack overflow answer for an algorithm that works: https://stackoverflow.com/a/51188472. Just use the counter as described previously.

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
thank you @Jake Baldwin !! we'll use this when improving the document.
0 Likes 0 ·
philspain avatar image philspain mvader (Victron Energy) ♦♦ commented ·
How can you get the Advertisement Key without linux or Mac? I bought this Victron product only to get data into HA I feel well and truly ripped off. Because you have a Youtube video starting you can integrate into HA. But you seem to make it hard to get the data needed to do that.
0 Likes 0 ·
tayl avatar image
tayl answered ·

Hi @mvader (Victron Energy) , I looked through the docs for alarm codes for the battery monitor (0x02), but can't find them.. maybe I'm overlooking them? The integration is otherwise perfect. I'd just like to give some context to these alarms. The remark in the docs for the 16-bit alarm field is shown as "VE_REG_ALARM_REASON".

5 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Hi @tayl ok I see. Did you already check the VE.Direct Text protocol document, or the public VE.CAN HEX document?
0 Likes 0 ·
tayl avatar image tayl mvader (Victron Energy) ♦♦ commented ·

I see a table in those documents, but I'm not sure it relates to the battery monitor we're using (smart battery sense), as the alarms we're seeing don't make sense for the product. In fact, across 100 of these sensors, we're seeing every one of these bits set in various cases. I'm wondering if the field is not actually used on these devices and the bits are just initialized with junk? For instance, this simple sensor can't measure AC, nor should it communicate info about a BMS lockout, yet both of those bits are set in several instances of packets we've pulled off them. All the voltage and temperature readings are fine though.


1691115408321.pngtable or alarm reasons from VE.Direct-Protocol-3.33.pdf


1691115864997.pnglist of values received from the 16-bit field defined in extra-manufacturer-data-2022-12-14.pdf as "VE_REG_ALARM_REASON"


Thanks

0 Likes 0 ·
1691115408321.png (62.1 KiB)
1691115864997.png (15.7 KiB)

Hi @tayl

This field is indeed not used on the Smart Battery Sense, but then it should be initialized to 0. We will look into this. In the mean time, is the other data decrypted properly? And the values in your table above are after decryption?

Kind regards,

Thiemo van Engelen


0 Likes 0 ·
tayl avatar image tayl Thiemo van Engelen (Victron Energy staff) ♦ commented ·
Hi Thiemo, thanks for the confirmation. Yeah, all of the other data is fine after decryption and matches the documentation. The alarm/status bits are typically 0, as I'd expect them to be. It's only on occasion that they get filled with garbage. We thought the packet these values are coming over with might be bad data all around, so we checked voltage etc.. in those packets, but the rest is fine. Seems the alarm field is missing/skipping initialization in some cases.


In any case, we will ignore the alarm field on this sensor from now on. Thank you.

0 Likes 0 ·
tayl avatar image tayl mvader (Victron Energy) ♦♦ commented ·
Hi @mvader (Victron Energy) just checking in, did you have a chance to read my last message?
0 Likes 0 ·
philspain avatar image
philspain answered ·

Any news on App update to be able to get advertisement Key easy

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hi all, it took a while, but per VictronConnect v5.93, released last Friday, you can see the keys. Here two screenshots:


1693142132046.png


1693142210358.png



1693142132046.png (413.3 KiB)
1693142210358.png (445.0 KiB)
4 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

zbindenp avatar image zbindenp commented ·

Hi mvader

Unfortunately I cannot see the instant readout data.

I have a "VE.Direct Bluetooth Smart Dongle" on a BlueSolarCharger MPPT 75/10. App is VictronConnect 5.95, BlueSOlarCharger is on Firmware v.1.61 and BT Smart Dongle is on Firmware v2.26.

According to the compatibility list, this should work. Before I tried also to find out the encryption key on the mac, but the sqlite table is always empty.

What am I doing wrong?

Regards

Patrick

0 Likes 0 ·
Thiemo van Engelen (Victron Energy staff) avatar image Thiemo van Engelen (Victron Energy staff) ♦ zbindenp commented ·

Hi @zbindenp

You have an old revision of the VE.Direct Bluetooth Smart Dongle for which we have not (yet) released a firmware version that supports instant readout.

Kind regards,

Thiemo van Engelen

0 Likes 0 ·
zbindenp avatar image zbindenp Thiemo van Engelen (Victron Energy staff) ♦ commented ·

Hi Thiemo

Thank you for the fast answer. I think it would be helpful (also for others) to mention, from which revision of "VE.Direct Bluetooth Smart Dongle" it is supported.
And it would be nice to have an idea, if and when a firmware with instant readout support is released

Regards,
Patrick

0 Likes 0 ·
Thiemo van Engelen (Victron Energy staff) avatar image Thiemo van Engelen (Victron Energy staff) ♦ zbindenp commented ·

Hi @zbindenp

>I think it would be helpful (also for others) to mention, from which revision of "VE.Direct Bluetooth Smart Dongle" it is supported.
I totally agree.

>And it would be nice to have an idea, if and when a firmware with instant readout support is released
The only thing that I can say is that it is very likely that a new firmware version will released somewhere in the future, but I cannot say anything about when. The only thing I can say about that is that it is not planned in the coming weeks.

Kind regards,

Thiemo van Engelen

0 Likes 0 ·
olen avatar image
olen answered ·

Has anyone made any effort to try to get the encryption key without using the official apps (e.g. reverse engineer how the key is fetched from the device?)
Since the official apps can do it, it should be possible for other as well.

Getting the key using for instance a RPI Zero is impossible today, as they only release the appimage for x86_64 not ARM.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Chris Jackson avatar image
Chris Jackson answered ·

Hi all

I've been following this thread with interest for some time, and finally, it seems all the pieces have come together. So I'm embarking on developing an Arduino/ESP32-based solution to access realtime Advertisement data from my two 100/30 MPPT controllers.

Apologies for the length of what follows....


I have started with the "BLE-Scan" example in the Arduino IDE, and running it in my ESP32 recognises my two MPPT controllers:

Example output from BLE-Scan:

Advertised Device: Name: MPPT Port, Address: ff:01:6f:13:e7:3b, manufacturer data: e10210024aa001870b9009f6f5e5cecf272b95552e6b, rssi: -93

Advertised Device: Name: MPPT Stbd, Address: cc:18:68:b6:4f:d4, manufacturer data: e102100276a001bd073651a0cc59bb4d94d966b29049, rssi: -95


Some successive "manufacturer data" records for the Stbd device (from advertisedDevice.getManufacturerData()), eg:

MPPT Stbd: e102100276a001ca6e363ae821389810e0ee0315568c

MPPT Stbd: e102100276a001d16e367a16705dbfc2fd72b1cfedde

MPPT Stbd: e102100276a001ec6e365a7d0892e2b974afe10cb98c

MPPT Stbd: e102100276a001f76e36e2f52ad7bd23912f5f4c0d4e

MPPT Stbd: e102100276a001fa6e365bd3fb3c6cd3a343d225048d

MPPT Stbd: e102100276a001326f367a286066dc43595b827d15b2

MPPT Stbd: e102100276a0013b6f365e9d850ad6964623559abfb8

MPPT Stbd: e102100276a001416f36963d7871dd70f1e5c320733d

MPPT Stbd: e102100276a001476f36f5543d0966d0a36647bafa54

MPPT Stbd: e102100276a0014f6f368f5ab68b399ed92e0c66dd2f

MPPT Stbd: e102100276a0015d6f3633615c8132fa6c7eb2e6bc86


My hope was that the "manufacturer data" above would contain the data I want in encrypted form, plus a 7-byte preamble as described by @Jake Baldwin above. But I can't reconcile my preamble with what @Jake Baldwin describes, ie:

Byte [0], e1, expecting 10 (record type)

Byte [1-2], 0210, feasible (device ID; mine is 100/30, 100/50 is 0257)

Byte [3], 02, expecting a0 (readout type)

Byte [4], 76, expecting 01 (record type, Solar Charger is 01)

Byte [5-6], a001, expecting an incrementing value (nonce)

Byte [7], variable, expecting 33 (the first byte of my Encryption Key "36b...")


I've also had a look at the Payload, which similarly doesn't seem to correspond with expectations.


For info, here are the public properties of the class I'm using:

class BLEAdvertisedDevice {

public:

BLEAdvertisedDevice();

BLEAddress getAddress();

uint16_t getAppearance();

std::string getManufacturerData();

std::string getName();

int getRSSI();

BLEScan* getScan();

std::string getServiceData();

std::string getServiceData(int i);

BLEUUID getServiceDataUUID();

BLEUUID getServiceDataUUID(int i);

BLEUUID getServiceUUID();

BLEUUID getServiceUUID(int i);

int getServiceDataCount();

int getServiceDataUUIDCount();

int getServiceUUIDCount();

int8_t getTXPower();

uint8_t* getPayload();

size_t getPayloadLength();

esp_ble_addr_type_t getAddressType();

void setAddressType(esp_ble_addr_type_t type);

bool isAdvertisingService(BLEUUID uuid);

bool haveAppearance();

bool haveManufacturerData();

bool haveName();

bool haveRSSI();

bool haveServiceData();

bool haveServiceUUID();

bool haveTXPower();

std::string toString();



I have a suspicion that what I'm seeing labelled as "manufacturer data" is not the same as "Extra Manufacturer Data", which is apparently what I need. Can anyone please shed some light on what might be going on? TIA!!

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi @Chris Jackson,

I'd suggest creating a new post in the modifications space with this question for more attention.

0 Likes 0 ·
Chris Jackson avatar image Chris Jackson Guy Stewart (Victron Community Manager) ♦♦ commented ·

Thanks for the suggestion, however I now see that if I discard the first two bytes (e1 02), the rest of the data matches my expectations perfectly. So I'm now moving on to the decryption task....

1 Like 1 ·
theterminalman avatar image
theterminalman answered ·

If anyone is interested in a basic standalone application to read this data using an ESP32 module, I've put together a simple (well... kinda) example that receives, decrypts, decodes, and outputs the Bluetooth Low Energy advertising beacons from Victron SmartSolar charge controllers.

Credit where credit is due: Thanks, Victron for enabling and documenting these beacons, and to @FabianS for his ESPhome code that helped me figure out how to do the AES decryption.

Feel free to give it a try to see if it works for you. I've tried this on several different ESP32 boards, including some basic three-for-$20 ESP32 dev modules from Amazon.

https://github.com/hoberman/Victron_BLE_Advertising_example

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

theterminalman avatar image
theterminalman answered ·

And extending on this I've also written code for the very nice M5StickC and M5StickCPlus modules that uses their integrated graphics hardware to display data from the BLE beacons.

https://github.com/hoberman/Victron_BLE_Scanner_Display

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Daniel Hrynczenko avatar image
Daniel Hrynczenko answered ·

I am very happy about getting this data from my battery monitor and solar chargers into my home assistant instance aboard nowadays

Are there any plans to extend this feature to the smart chargers as well? It would be a great way to quickly see if it has been disconnected from shore power

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ashdash avatar image
ashdash answered ·

Thanks @Jake Baldwin for your detailed post which helped me make sense of what I was seeing using command line and online tools.

This post is for a 500A SmartShunt operating on the test bench, powered by a mains power supply connect to the VBatt and Aux inputs. No battery and zero amps. A Raspberry Pi 5 is communicating with the shunt using the pi's onboard bluetooth.


Open 2 pi terminals. On the first terminal execute "sudo hcitool lescan --duplicate".

On the second terminal execute "sudo hcidump --raw". You should see something resembling the image below.

Look at the second long line. Find "02 81 04 42". From the Victron "Extra manufacturer Data", "02" indicates battery monitor. "81 04" is Nonce/Data counter in LSB order. "42" is the first byte of the encryption key. The rest of the block is encrypted data.

hcidump.png


This post https://devzone.nordicsemi.com/f/nordic-q-a/86096/decrypting-secret-hex-output-from-aes-ctr-example-in-3rd-party-application-python from a Nordic Semi forum led me to this https://cryptii.com/pipes/aes-encryption super useful online utility.

So, looking below, if we copy and paste from the pi terminal window into the cryptii decryption utility we can start to see some sensible looking decrypted data.

Into the left dialogue box is entered the encrypted data. In the middle dialogue is entered the key from the Victron Connect app on my phone. IV (initialization vector) is "81 04" padded with zeroes out to 16 bytes. Thank-you @Jake Baldwin.

cryptii-screenshot-8104.png


And for the image below, the encrypted data is completely different, key the same, Nonce is now "82 04" padded and the data is the same except for the last byte. I haven't investigated that last byte yet.

cryptii-screenshot-8204.png


Looking at the decrypted data and referring to Victron "Extra manufacturer Data", the first 2 bytes (16 bits) is TTG (time to go?) and NA value is 0xFFFF.

Next 2 bytes (16 bits) is battery voltage. A signed 16 bit integer in little-endian format (low byte first, reads back to front). So 0x04e0 is decimal 1248 which is 12.48 volts.

Next 2 bytes (16 bits) is alarm reason. No alarms so I guess 0x0000.

Next 2 bytes is aux voltage. So 0x04e1 is decimal 1249 being 12.49 volts. The two inputs are reading differently by 0.01 volts.

The next 3 bytes (24 bits) consists of 2 bits representing how the aux input is configured. For voltage in this case. The remaining 22 bits is the current. Zero in this case because nothing is connected to the shunt.


Thank-you fellow contributors for your help. I hope this post will help others.

Regards.



2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Bruno M avatar image
Bruno M answered ·

Although documentation specifies a value of 0x10 for the field "Manufacturer Data Record type", I received frames with a different value.

Sources:

- https://community.victronenergy.com/questions/187303/victron-bluetooth-advertising-protocol.html

- pdf doc "Byte [0] is the Manufacturer Data Record type and is always 0x10".


I'm thus expecting the third manufacturer data byte to be 0x10. Here are 3 frames received from Victron product, I see in my logs:


Orion DC/DC

e1 02 10 00 d1 a3 04 60 5a f7 11 18 14 62 e3 e9 f9 d8 0e 5c

> manufactuter bytes are (0x10, 0xe1) > Victron

> fine we, have 0x10


SmartSolar

e1 02 02 41 ff 5e 79 97 0c d6 75 b0 8a 87 fc f1 d6 70 0c 79 bf a5 3a 93 f2 a0 49 fc 7b

> manufactuter bytes are (0x10, 0xe1) > Victron

> what is this 0x02


Phenix AC charger

e102 02 41 b7 dd 26 99 3e ed be 81 94 81 9c 29 40 fc d8 c3 17 2b 24 3a

> same 0x02 type I don't understand


Any idea why I get 0x02 and not 0x10?

Could be an error on my receiving code, but I suspect a different or older protocol for these devices, too.

Thanks !



3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi @Bruno M

Message with the value 0x01, 0x02 or 0x03 there are used for VE.Smart Networking. They do not contain instant readout data. Only messages with the value 0x10 there are Instant Readout packets.

Kind regards,

Thiemo van Engelen

1 Like 1 ·
Bruno M avatar image Bruno M Thiemo van Engelen (Victron Energy staff) ♦ commented ·
Thanks for this information, I may have captured the wrong frames indeed.

Is there any documentation that specifies these frame types (0x01/2/3/) ?

Can you confirm advertisement frames (0x10) are sent permanently without any need to "trig" them ?

Thank you

0 Likes 0 ·

Hi @Bruno M

We do not provide any specification on the VE.Smart networking packets.

And I can confirm that the 0x10 advertisement frames are sent permanently (unless bluetooth is turned off on the product) and they do not need to be triggered.

Kind regards,

Thiemo van Engelen

0 Likes 0 ·
VAN THUAN avatar image
VAN THUAN answered ·

Bonjour,

I am working on retrieving data from a SmartBMV 712 using an M5STACK Core 2 in C++. I have used the extra-manufacturer-data document, which allowed me to extract certain values such as the State of Charge (SOC) and the voltage. However, I am unable to extract the 'Battery Current.' Has anyone succeeded in decoding the 22 bits of this data? I am getting huge, completely erroneous values.

Merci !

4 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi @VAN THUAN

I would help if you could add some information, for example the (unencrypted) data that you are trying to parse and the value you think that you are reading. Perhaps even some code that you use to decode the data.

Kind regards,

Thiemo van Engelen

0 Likes 0 ·
petaramesh avatar image petaramesh commented ·

Hello,

Dunno for C++, but you can find the way it's done in Python here, it might help you to get the correct structure :
https://github.com/keshavdv/victron-ble/blob/main/victron_ble/devices/battery_monitor.py

I have ported it myself to micropython that has less C data structure abilities, some code is there :
https://github.com/keshavdv/victron-ble/issues/44#issuecomment-2041409051

HTH

0 Likes 0 ·
ashdash avatar image ashdash commented ·

Bonjour @VAN THUAN,

I agree it would be helpfull to have more information. I think your problem will be one or more of the following. The data is little-endian format (low byte first, reads back to front). Do you have the correct endianess? Endianess is platform and transmitted data dependent, not C++.

The data is 22 bits packed with 2 bits of aux data. When the data is in a signed integer have you shifted out the 2 aux bits? You need to arithmetically (preserve the sign) shift the whole 24 bits?


Looking at and from the git python example:

from construct import GreedyBytes, Int16sl, Int16ul, Int24sl, Struct

## Python supports 24 bit signed integers.


"current" / Int24sl,

## "current" is defined as a signed 24 bit integer within a structure.


"current": (pkt.current >> 2) / 1000,

## Shift out the aux bits and divide by 1000 for amps. Raw data is milliamps.


I hope this helps. Regards.

0 Likes 0 ·
VAN THUAN avatar image VAN THUAN ashdash commented ·

Thank you very much for this information. I haven't fully grasped all of your explanations yet, but I understand that the encoding of the "current" value I've used isn't correct despite the valuable help from @theterminalman. The entire code is too lengthy and isn't on Github, so here are some portions to try to explain better. Here's the structure I'm using:

/ Structure des données envoyees par le controleur Smart BMV 712
typedef struct {
  uint16_t bmvTTG;
  uint16_t bmvBat1Voltage;
  uint16_t bmvAlarmReason;
  uint16_t bmvBat2Voltage;
  uint8_t byte0;    // byte 0-2 , 3 octets qui, selon de Extra_manufacturer_data de Victron, contient: Aux input, Battery current
  uint8_t byte1;    // on n'utilisera que battery current.
  uint8_t byte2;
  uint8_t byte3;    // byte3, 1 octet contenant le MSB du Consumed Ah. Cet octet sera extrait mais pas affiché, car nécessiterait aussi le LSB
  uint32_t bmvData2;// 32 derniers bits des données contenant entre autre la valeur SOC et le LSB de Consumed Ah.
} __attribute__((packed)) victronBMVData;


The 24 bits of "Aux input" and "Battery current" are the variables byte0, byte1, and byte 2.

Here's the data extraction: (Hoberman will recognize is own code for SmartSolar, modified. Thank you)

uint16_t bmvTTG = uint16_t (victronData->bmvTTG);
          float bmvBat1Voltage = float(victronData->bmvBat1Voltage) * 0.01;
          uint16_t bmvAlarmReason = uint16_t(victronData->bmvAlarmReason);
          float bmvBat2Voltage = float(victronData->bmvBat2Voltage) * 0.01;
          uint8_t byte0 = (victronData->byte0);
          uint8_t byte1 = (victronData->byte1);
          uint8_t byte2 = (victronData->byte2);
          uint8_t byte3 = (victronData->byte3);
          uint32_t bmvData2 = uint32_t(victronData->bmvData2);

This extraction works well to obtain bmvTTG, bmvBat1Voltage, bmvAlarmReason, bmvBat2Voltage, and the SOC (Stored in bmvData2) whose values are accurate.

And finally, here are the raw datas I obtain for the 3 bytes. Zeros at left are not displayed:

11:32:51.472 -> byte0=10100
11:32:51.472 -> byte1=100101
11:32:51.472 -> byte2=0
 
11:33:02.184 -> byte0=1000100
11:33:02.184 -> byte1=100101
11:33:02.184 -> byte2=0
 
11:33:06.335 -> byte0=1110100
11:33:06.335 -> byte1=100101
11:33:06.335 -> byte2=0
 
11:33:14.350 -> byte0=10111100
11:33:14.350 -> byte1=100010
11:33:14.350 -> byte2=0
 
11:33:17.022 -> byte0=1001100
11:33:17.022 -> byte1=100000
11:33:17.022 -> byte2=0



0 Likes 0 ·
VAN THUAN avatar image
VAN THUAN answered ·

Thank you very much for this information. I haven't fully grasped all of your explanations yet, but I understand that the encoding of the "current" value I've used isn't correct despite the valuable help from @theterminalman. The entire code is too lengthy and isn't on Github, so here are some portions to try to explain better. Here's the structure I'm using:

// Structure des données envoyees par le controleur Smart BMV 712
typedef struct {
  uint16_t bmvTTG;
  uint16_t bmvBat1Voltage;
  uint16_t bmvAlarmReason;
  uint16_t bmvBat2Voltage;
  uint8_t byte0;    // byte 0-2 , 3 octets qui, selon de Extra_manufacturer_data de Victron, contient: Aux input, Battery current
  uint8_t byte1;    // on n'utilisera que battery current.
  uint8_t byte2;
  uint8_t byte3;    // byte3, 1 octet contenant le MSB du Consumed Ah. Cet octet sera extrait mais pas affiché, car nécessiterait aussi le LSB
  uint32_t bmvData2;// 32 derniers bits des données contenant entre autre la valeur SOC et le LSB de Consumed Ah.
} __attribute__((packed)) victronBMVData;

The 24 bits of "Aux input" and "Battery current" are the variables byte0, byte1, and byte 2.

Here's the data extraction: (Hoberman will recognize is own code for SmartSolar, modified. Thank you)

          uint16_t bmvTTG = uint16_t (victronData->bmvTTG);
          float bmvBat1Voltage = float(victronData->bmvBat1Voltage) * 0.01;
          uint16_t bmvAlarmReason = uint16_t(victronData->bmvAlarmReason);
          float bmvBat2Voltage = float(victronData->bmvBat2Voltage) * 0.01;
          uint8_t byte0 = (victronData->byte0);
          uint8_t byte1 = (victronData->byte1);
          uint8_t byte2 = (victronData->byte2);
          uint8_t byte3 = (victronData->byte3);
          uint32_t bmvData2 = uint32_t(victronData->bmvData2);

This extraction works well to obtain bmvTTG, bmvBat1Voltage, bmvAlarmReason, bmvBat2Voltage, and the SOC (Stored in bmvData2) whose values are accurate.

And finally, here are the raw datas I obtain for the 3 bytes. Zeros at left are not displayed.

11:32:51.472 -> byte0=10100
11:32:51.472 -> byte1=100101
11:32:51.472 -> byte2=0

11:33:02.184 -> byte0=1000100
11:33:02.184 -> byte1=100101
11:33:02.184 -> byte2=0

11:33:06.335 -> byte0=1110100
11:33:06.335 -> byte1=100101
11:33:06.335 -> byte2=0

11:33:14.350 -> byte0=10111100
11:33:14.350 -> byte1=100010
11:33:14.350 -> byte2=0

11:33:17.022 -> byte0=1001100
11:33:17.022 -> byte1=100000
11:33:17.022 -> byte2=0

11:33:25.037 -> byte0=1000000
11:33:25.037 -> byte1=100101
11:33:25.037 -> byte2=0

11:33:34.505 -> byte0=10110100
11:33:34.505 -> byte1=100011
11:33:34.505 -> byte2=0

11:33:38.442 -> byte0=11000100
11:33:38.442 -> byte1=100110
11:33:38.442 -> byte2=0

11:33:39.895 -> byte0=111000
11:33:39.895 -> byte1=100011
11:33:39.895 -> byte2=0

11:33:41.114 -> byte0=11001100
11:33:41.114 -> byte1=100101
11:33:41.114 -> byte2=0

If I understand correctly, I should start by concatenating Byte0, 1, 2 to obtain a 24-bit word, then shift 2 bits to the right to remove the "Aux Input" value, which is actually on the 2 bits at the far right and not on the left. Then, this is where I encounter a problem in reversing the remaining 22 bits, then identifying and interpreting the sign bit, and finally decoding the absolute value in milliamps.

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ashdash avatar image ashdash commented ·

Hello VAN THUAN,

The extra information you have provided will help resolve your issue.

My apologies, I have introduced some confusion talking about endianess. All major systems (Arm etc) are little endian and the data is also so that is not a concern. When I was looking at the bluetooth data I was also looking at VE-Direct and how to implement a project on an 8 bit PIC so I was very focused on bits and byte order without floating point and string libraries.


Looking at your data:

11:32:51.472 -> byte0=00010100 (right hand two zeroes are aux config)
11:32:51.472 -> byte1=00100101
11:32:51.472 -> byte2=00000000

I decode that as 0b00|0000|0000|1001|0100|0101 = 0x945 = decimal 2373, so plus
2.373 amps. Does that look correct? Battery is trickle charging?


I don't know C++. I did a quick google and it looks like C++ does not support int24_t so you won't be able to handle this data as simply as python or C can do it. Even the XC8 compiler for the PIC handles 24 bit long integers.

I would look closely at the C++ IDE for your M5STACK Core 2. I think you may have to bit bang the data into a 32 bit signed integer. Maybe there is a library that can do this for you? Or maybe 24 bit is supported on these small processors. Remember this is signed data so you need to preserve the sign as the highest order bit. Your current can be positive or negative.

I also note that in the "Extra manufacturer data" document the "Battery Voltage" and "Aux voltage" fields are plus/minus so that data should be int16_t not uint16_t. I cannot think of a situation when the battery voltage will be negative, but maybe Victron has.

Does any of this help?

0 Likes 0 ·