question

Ulrich avatar image
Ulrich asked

reload past data from VRM to local influx db

I use Grafana to display data from my installation. (https://www.zoomsoft.de/grafana)

Sometimes it happens that the connection is lost and the data is not currently transferred to the local database.

Is there a script with which I can reload the data from VRM?

VRMgrafana
1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Ulrich avatar image Ulrich commented ·
In influxdb.js i found this.lastWriteTime = Date.now()

But this is not the starttime to load data from vrm.

Where/how exactly could i set it?

​​​
0 Likes 0 ·
1 Answer
joanspark avatar image
joanspark answered ·

Do you mean something like this?

https://docs.victronenergy.com/vrmapi/overview.html#get-the-energy-readings-for-a-given-site-installation

"Get the energy readings for a given site/installation
The energy readings consist of various data attributes, which can be combined to give a total amount. The values of the different data attributes for the given period and interval are returned, and the sum of the attributes is also included in the response of the endpoint.

Endpoint:
https://vrmapi.victronenergy.com/v2/installations/{installation ID}/stats?type=kwh
Required parameters (GET):
start: Unix timestamp of the starting timestamp for which the stats should be retrieved
end: Unix timestamp of the ending timestamp for which the stats should be retrieved."

note this:

"There is a maximum allowed time period for each interval, which is:

  • 31 days for: 15mins
  • 31 days for hours
  • 180 days for days
  • 140 days for weeks
  • 24 months for months
  • 5 years for years

Requesting a larger time period will result in an error .."

Looks like the VRM db is a Grafana db as well, if one looks at the resolution of historical records and 15 min for the last 30 days is the highest resolution you will be able to acquire.
Your graphs seem to have a datapoint for every minute, so you'll be out of luck if you want this kind of resolution fromt he VRM as a sort of backup.

3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Ulrich avatar image Ulrich commented ·

Unfortunately, I have dropouts for several hours. Filling these with 15 minute data would be better than having no data.

The description must be out of date, because when I request data from December 1st, 2020 in the VRM portal, I get values per hour. That would also be ok, especially for the annual overview, in which I don't show any more.

Unfortunately I cannot adapt the JS program so that the missing data is loaded.

It would be very useful if you could reload old data. Everyone who starts with Grafana would be happy if they had a lot of data right away.

Who can help me?

0 Likes 0 ·
joanspark avatar image joanspark Ulrich commented ·

More reading from the VRM API wiki:

"There are two use cases that you could have in mind when implementing a system that calls this API:

  1. To build your own front-end; that uses our database & API as a backend.
  2. To replicate the data into your own backend.

The API is not suitable for that last use case; and there is no support offered for such implementations by us either. One of the issues is that you can never know what data you have replicated, and what data not yet. Combined with the backlogging function in the GX devices this means that you'll have to continuously query quite far into the past in order to really make sure to have data replicated. We might some day design a proper replication function; but at the moment there are no such plans and also we will not offer support of such setups; unless the involved directly commercial value is extremely high.

If you really do want to host the data; we recommend to implement your own data transmission system on the GX device. Or change the URL of the existing data-transmission system on the GX device to your own URL. Note that this will (obviously) break all the advanced features such as Remote Firmware updates, Remote Support, Remote Console, and-so-forth."


See, that's why I go VE.Direct <> RS232 <> Ethernet <> my own cloud system(s), completely circumventing vendor lock-in.. ;-)
By using VRM you have to get your VE device online anyway and pulling the data from the device with your own software-stack isn't much harder than relying on VRM IMHO.

PS: I take it your device(s) aren't reachable from the cloud? You can't pull data from them? How is your hardware setup.. don't you use a modem router where they are located? Can't they push to zoomsoft.de directly? And failing that, setting up a VPN which would allow you to PULL data from there is possible too with most modern modem routers. Alternatively running a local instance of VenusOS on some SBC which collects the data without interruptions and as fine grained as you need which then pushes it onto zoomsoft.de?

Sorry for not being more helpful. Good luck.

0 Likes 0 ·
Ulrich avatar image Ulrich joanspark commented ·

Your path sounds very interesting. Could you briefly describe it to me? That would be a nice Christmas present from you. :-)

0 Likes 0 ·