question

fabian-lauer avatar image
fabian-lauer asked

Shelly 3EM SmartMeter with VenusOS / Cerbo GX

Hi,

my name is Fabian - I am new to the whole topic with Victron and VenusOS / GX devices.
But I created a small script/module for the devices to connect Shelly 3EM smart meter.

You can find all details on GitHub: https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter

venus-os-shelly3em-smartmeter.pngvenus-os-shelly3em-smartmeter-devicedetails.pngHope that this is helpful for someone out there.

I would be very happy to get some feedback and input how to improve.


Cheers


cerbo gxVenus OSEnergy Meterpower meter
8 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Awesome! Thanks very much, Fabian.
0 Likes 0 ·

You, Sir, are a legend!

0 Likes 0 ·
fabian-lauer avatar image fabian-lauer Warwick Bruce Chapman ·
Thx - but I am still alive ;-)
0 Likes 0 ·

You can have living legends, lol

0 Likes 0 ·
Thanks, this is great, I have been waiting for something like this. Main reason is to replace the EM24 in my installations. EM24 has caused so much pain - plus it needs cabling and is more expensive while having less features.
0 Likes 0 ·

I have ssh:d in to the pi and run your script. How do i do this section. Im a newbie...

/Mattiasskarmavbild-2022-05-04-kl-172637.png


0 Likes 0 ·

Hi Trying to get it to Work . is it possible for you to make a more detailed instruction how to att hook up an SHELLY EM to Venus OS? Im a bit on thin ice with everything .... Thanx /MAttias @fabian-lauer

0 Likes 0 ·
Hello, i use your Skript and my Venus know the State of the Grid but my Multiplus do mit Work. I Mean the Multiplus send the PV from AC Out to AC in but not from Batterie. What i do wrong.
0 Likes 0 ·
19 Answers
willi9974 avatar image
willi9974 answered ·

thanks for the topic, will check this if i have my hardware

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

baxter avatar image
baxter answered ·

Would it be possible do add the support for Shelly EM to measure the production of a 1ph pv inverter (micro inverters)? Tia Mike

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

I think so.

I don't have a Shelly EM, so hard to test.
But I have Shelly Rgbw 2, which can also measure power consumption.

When I call http://192.168.2.104/status on this device I get a JSON output with a node "meters" there are some information in - how does yours look like

1647623346979.png

0 Likes 0 ·
1647623346979.png (11.0 KiB)
christian-graf avatar image
christian-graf answered ·

Really a great thing. Maybe someone can help me. I always get aborts with the following error:

Traceback (most recent call last):
File "/data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py", line 16, in <module>
import configparser # for config/ini file
ImportError: No module named configparser

4 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi - not 100% sure, because I am new to Python, but the module "Configparser" is not installed in your Python environment - needs to be installed.


What setup are you using?
I have only tested with Cerbo GX and latest firmware

0 Likes 0 ·
I struggle with the same issue. I try to run it on an MultiplusGX, and there seems no configparser installed, and I got no idea how to install it.

Is there an easier way to "hardcode" the Shelly's IP address (and this is all I change in config.ini) into the python script so we don't need the configparser?

0 Likes 0 ·

Hi - is possible...just do the following:


  • Remove "import configparser # for config/ini file" (Line 16)
  • Remove function _getConfig(self) (line 76)
  • Update function def _getSignOfLifeInterval(self)
    to only return a fixed value: return 0
  • Update function def _getShellyStatusUrl(self)
    to only return the url: return "http://url/path"

That should be all

0 Likes 0 ·
Thanks Fabian. I found that the mistake was on my side. The OS was not up2date. I did an update, and configparser was included then. So your script is working fine now! :) But it's good to know a way around - just in case.
0 Likes 0 ·
teemischer avatar image
teemischer answered ·

FYI, I just briefly looked at your code on Github and I think you are counting the amount of Energy Forward/Reverse in the “wrong“ way. There were discussions about the way different energy meters count, like ET340 vs EM24. Can you please check?


Die Shelly 3EMs saldieren nicht so, wie es ein Zähler Deines EVUs macht. Du scheinst in Deinem Plugin aber davon auszugehen und übernimmst diese Werte in den DBUS.

Das ist ein Shelly Problem, siehe hier https://www.shelly-support.eu/forum/index.php?thread/8977-shelly-3em-phasen-saldieren-in-home-assistant/ (Post #16 rechnet es schön vor)

3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

That is a good tip - maybe I did some logic wrong :-(

Will check your article and provide a new solution...

code changes are always welcome :-)

0 Likes 0 ·

Hi @teemischer,

I checked the article and I guess I know what you mean - in the cloud-app from Shelly the diagram data is "wrong" - see attached screenshot yellow-box

For the calculation I use only the real time data - red-box.

I think that should be fine, also for the official German grid meters.
I know that I can't say what was really consumed on Phase A (in my case the pv-inverter is connected there and off course some loads).
So when I produce 700W of solar-/pv-energy and shelly says -668,3W on that phase I have currently no idea of the consumed 31,7W.


Maybe I didn't get something and my explaination is wrong.
So I would be happy to get more feedback and correct the software if needed.


shelly3em-datasource.png

shelly3em-json.png


By the way - I just noticed some strange consumption data in the VRM portal - maybe this related to the this issue - but lets discuss

0 Likes 0 ·

Hi,

yes, I think you got it.

The current values ("Power") are okay and seem to get processed correctly.

But the aggregated values over time ("Energy") counters are used incorrectly.

This is not directly your mistake because the Shelly aggregates them "differently" from the regular grid meters.

1648477368421.png

Those are the values which cause "wrong" PV generation/electricity consumption values.

A solution would be to calculate the real Energy Values yourself. And then pass those on to dbus.


br,

TM


0 Likes 0 ·
1648477368421.png (2.8 KiB)
mskovdk avatar image
mskovdk answered ·

This is EXACTLY what I'm looking for. 3 x Multiplus II in order, and already have the Shelly 3em installed. If it works, I owe you a beer!

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

fabian-lauer avatar image
fabian-lauer answered ·

FYI - deployed a new version with fixes crashing of script

https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter/releases


In detail: the regular loop was called by timeout_add() function, which is cancelled when function is not returning True at every call - this could happen in case of an exception.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

vikt0rm avatar image
vikt0rm answered ·

Hi Fabian, i am also a software developer and would like to contribute on your project. I already fixed the issue on energy in my local setup. Furthermore, based on your project, I created a new project for using a Shelly 1PM to use it as a PV inverter in VRM. You can find my project here:

https://github.com/vikt0rm/dbus-shelly-1pm-pvinverter

8 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi @vikt0rm,


thanks again for support - new version of script is now on GitHub - https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter/releases/tag/v0.2


Maybe this fixes also the point that @teemischer mentioned before.

0 Likes 0 ·
I am still thinking about the total values for /Ac/Energy/Forward and /Ac/Energy/Reverse


I think the calculation is wrong - it is according to @teemischer comment.
Because when I check latest statistics for today - I fulfill a lot of energy from the grid (according to VRM portal), which can't be true, because of sun is shining and PV is doing a good job - grid meter is not "moving" and shell is also saying that I produce more than I consume, based on having a balancing meter (which we have in Germany)

@vikt0rm maybe you can check with your installation (don't know where you are located).

But currently I think I have to update the script a bit....not 100% sure how

0 Likes 0 ·

You are right, your current implementation is the "kundenunfreundliche" variant. To make it "kundenfreundlich" you have to adapt your over all Forward to:

consuptionOrFeedback = (L1_f + L2_f + L3_f) - (L1_b + L2_b +L3_b)
All_Forward = consuptionOrFeedback > 0 ? consuptionOrFeedback : 0

All_Backward = consuptionOrFeedback < 0 ? consuptionOrFeedback * -1 : 0


0 Likes 0 ·

Yeah - I thought of another implementation as I had a look in the vrmlogger which is collecting the data and uses "delta" loads and send to VRM.


Attached you find an Excel how I tried to calculate the "correct" values - also attached the script where I added a function _getBalancingMeterResult()

Maybe you can double-check I currently run the script in my local environment....with battery usage it looks ok....guess need to test a bit.

Hole idea is to do a realtime balanced meter calculation based on Shelly "total" values. Therefore I take a value and point in time "t0" and compare later values "t1", take the difference on input and output energy and do some magic (seeExcel)

https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter/blob/develop/docs/calculation-shelly-balancing-meter.xlsx

https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter/blob/develop/dbus-shelly-3em-smartmeter.py

0 Likes 0 ·

On second try with a fresh brain this morning I was able to follow your intention :) Looks good and plausible. The new method in the branch does what it should do. I will test it in my setup and see if my consumption looks better when sun is shining.

0 Likes 0 ·
Yeah maybe my explanations where not that great.


But I checked yesterday at night and there is still a gab for me - official grid meter shows 3kW and VRM ~6kW for last 24h - a bit difference would be ok, because the official meter has not decimal place.

I did an update on the development branche on GIT.
In the Excel I added a sample with 3 time-slots.
And I changed the function a bit, because I just assume that on runs >1 the basis is wrong - should use the result of the previous run.

But I will check now
Would be great @vikt0rm if you can have a look - seams to be a topic for more than just 1 pair of eyes

0 Likes 0 ·

I did some corrections to the script and the balancing calculation yesterday at lunch time - checked this morning and values are ok now - offical grid meter says 2kWh and VRM is a bit more precise.

Will check for some more days and push to main branch

1648706002446.png

0 Likes 0 ·
1648706002446.png (38.2 KiB)

Hi Fabian,

thank you for your great work. I have this running with my Multiplus as well and this saved me an EM24!

Regarding the summed backward and forward energy calculation: I solved this differently some time back in home assistant and just copied my code over to your implementation.

You get the correct summed values, if you sum up each watt (power) and convert to kWh.

Just exchange the two lines for forward and backward calculation with:

if (self._dbusservice['/Ac/Power'] > 0):

self._dbusservice['/Ac/Energy/Forward'] = self._dbusservice['/Ac/Energy/Forward'] + (self._dbusservice['/Ac/Power']/14400000)

if (self._dbusservice['/Ac/Power'] < 0):

self._dbusservice['/Ac/Energy/Reverse'] = self._dbusservice['/Ac/Energy/Reverse'] + (self._dbusservice['/Ac/Power']*-1/14400000)

The division by 14400000 derives from 60min x 60sec x 1/4sec x 1000watt.

You take a reading from the shelly every 250usec. I'm wondering, if the shelly provides data more often than once per second. Are you positive, this is really required or switching to 1sec is fine as well? (in that case the 1/4sec in the code above need to be removed)

Christian


0 Likes 0 ·
vikt0rm avatar image
vikt0rm answered ·

Well I think wrong depends on your expectation. From what I can see from todays logs (I reset my data around 10am, since I am still in development) it looks promising. By the way, I am also located in Germany (Munich).

From what I know right now there exist "Phasen-saldierend" and something like "Summen-saldierend". Here is a good explanation https://www.messstellenbetrieb-bock.de/saldierende_Zaehler.php what can be the difference.

This is that my data shows in VRM:

bildschirmfoto-2022-03-28-um-193925.png

And this is my graph from home assistant.

bildschirmfoto-2022-03-28-um-193857.png

For me it looks good.

Edit:
Wallbox value is currently not part of the VRM implementation. goe-charger needs to be connected to VRM. That is my next task. :-) And yes the calculation needs to be adapted as stated in my direct answer.


1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

How do you plan to connect the go-e charger to VRM? Is there some kind of integration already?
0 Likes 0 ·
highlander avatar image
highlander answered ·

Hello Fabian

Great! I've been trying to integrate an em3 into Venus for a long time. If I understand correctly, your python-script runs on Venus or does it have to be on a rasperry? Or have i do a script outside for example on a arduino? Sorry i am a little confused...

5 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi @highlander - I run the script on a Cerbo GX from Victron - so should work for you - never tried on a Raspberry, but that should work out.

It has to run on the Venus/GX device hat is e.g. connected to the VRM because the script uses the system DBUS.


Hope my answer will help you

0 Likes 0 ·

Thanks Fabian, now I understand a few things. But how the heck do I bring the directory with the files from Windows into the Venus. I tried putty. I can get in too, but I can't transfer the files. Unfortunately, I cannot find the correct syntax for the commands. Create directory, transfer file etc... The idea was to adapt the config.ini and transfer everything afterwards. It might work with your code, but how do I then adapt the config.ini in the venus. Thank you for your help...

0 Likes 0 ·

Morning @highlander - if you don't want to use the script mentioned in the readme on the GitHub repo, use WinSCP for SFTP (basically SSH) file-transfers.


Change the config.ini to your needs.
Run the install.sh script.
That should work

1648706231026.png

0 Likes 0 ·
1648706231026.png (68.4 KiB)

Yes! Now it works with my multiplus 48-1600.

remote-console.png

Soooo cool!
If you're ever in Lucerne, you'll get a fine home-brewed beer!

0 Likes 0 ·
remote-console.png (61.9 KiB)
Have got it installed directly on the Multiplus? I ask because I tried this, and got some issues with missing configparser python stuff.
0 Likes 0 ·
flailingfox avatar image
flailingfox answered ·

Thank you for your idea of using a shelly, I have plenty of them dotted around the house just not a em. But it did get me thinking, I use a self made "Home Assistant Glow" meter reporter (https://github.com/klaasnicolaas/home-assistant-glow), and you can also fetch power and energy parameters from that through REST, so I've forked your project and have it working on my own setup nicely at the moment, the latest code hasn't been committed yet as I'm still testing through a shell...I'll keep you posted on updates if you like, but just wanted to say thanks @fabian-lauer

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

kevner avatar image
kevner answered ·

top work. Thanks for the script.

Can anyone help here? instead of querying a shelly how can get the values directly from iobroker?

I'm not a developer, but it should be possible, or not?

https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter/issues/6


@FlailingFox

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hundlos avatar image
hundlos answered ·

victron.pngHey,

great work but i am confused why my system shows under ac loads some strange values - looks like it doesnt like the negative value -

for example i have a load of 1700 on L1 - and -1700 comes in via L3 -

The Grid Value show correct but the AC Loads not and Overall Consume is also to high then.


I am having a Multiplus II - 3000 // 3 Solar Chargers / Canbus Battery / Shelly 3EM


please help :)


victron.png (65.9 KiB)
3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hmmm....got your point, but never had this situation - can only guess that the problem is based on the fact that the multi (or other device you use for battery discharge) is not connected to L1 - for me it is the case
0 Likes 0 ·
madtv avatar image madtv fabian-lauer ·

Hi, I have the same issue like hundlos . is there a way to fix this? I am using a Multiplus II 48/3000 Thx Josef

0 Likes 0 ·
Hi,

can you give more details - updated screenshot and how wiring is done for your setup.

In case that Shelly app is also showing the value on the wrong phase, maybe the wiring of shelly is not correct.

0 Likes 0 ·
fgingat avatar image
fgingat answered ·

THANKS !!!!!!! i was stuck with a faulty EM24 and i cannot find a replacement anywhere ..

Luckily for me i already had a 3EM to monitor input/output ... you saved me a lot of headaches and money, THANKS AGAIN....

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

cwaxe avatar image
cwaxe answered ·

Hello @fabian-lauer ,
great job! The script is a real benefit for all shelly users. I'm using a shelly em to log and monitor my pv. The status output looks as attached. I tried to install your service via the script provided by you on a Multiplus II GX but i cannot see the device in the device list. Are there any additional steps to do? Thank you in advance for your help.

status.png


status.png (22.1 KiB)
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Elimac avatar image
Elimac answered ·

Hi, I hope you don't mind I ask this here...

Does anybody know if Shelly EM/ 3EM can be powered from a different AC line (both Neutral an Live) than the Grid it is measuring? Although the device consumption is low, my goal is to put the Shelly device powered by an Offgrid installation (both consumption and reliability/ availability issues). The Shelly however, is supposed to measure Grid AC energy spenditure on another grid installation (close by). Can this be done?

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

It CAN be done, and you will get measurements, but the power values will be not correct as the voltage you're measuring is not in phase with the voltages the 3EM is connected to. How much off? I can't say. It should be ok for a rough estimate.
0 Likes 0 ·
jw1971 avatar image
jw1971 answered ·

FYI if using the "large" Venus image you can also use Node-Red to fetch directly from the Shelly either via their REST API or MQTT. This might be easier for some depending on what you plan to use those measurements for.


1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

I looked to see if you could do this in NodeRed but couldn't see an obvious way. Can you describe it please?
0 Likes 0 ·
tobi-k avatar image
tobi-k answered ·

The Skript Work and Venus Know the State of Grid but my Multiplus Dont Work. They can send Power from AC Out to AC in. What i do Wrong. When He can send Power to Grid?


Victron Multiplus 24/1200

Batterie 4x Longlife 6v 225Ah

Shelly 3EM

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

bonzai avatar image
bonzai answered ·

Hello,

this skript is running very well for me. but one question: in the remote desktop it looks all rigth but in the portal dashboard i`am missing the total: what can id do?

1656315928734.png


1656315928734.png (12.7 KiB)
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mmm avatar image
mmm answered ·

How do I adapt this to work for Shelly EM?

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.