question

mvader (Victron Energy) avatar image
mvader (Victron Energy) asked

SOLVED - VRM Portal performance issues 2023-05-22


UPDATE: this thread is closed. For latest status, see here https://community.victronenergy.com/questions/209545/vrm-portal-performance-issues-2023-06-01.html.


Good day. Since a few hours, and same as past few Mondays (...), the VRM Portal is having performance issues.

And as a temporary measure we have disabled the state graphs on the Advanced page, since they cause for an extraordinary high load, compared to all the other graphs.

We're working hard on this; and I expect it to be solved in the coming hours.

My apologies for the inconvenience.

Matthijs Vader

VRM
19 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

cpt-pat avatar image cpt-pat commented ·
Considering the portal is a free service, I'm just pleased it is being provided. Best of luck in troubleshooting!
25 Likes 25 ·
mattysolow avatar image mattysolow commented ·
I live off of the grid for 6 months a year. I depend on solar and batteries with an inverter. I rely on the information in advanced Cerbo to balance my use. I would not mind paying a small monthly fee to offset additional servers and storage.
3 Likes 3 ·
nwac avatar image nwac commented ·

Hi I am noticing my 2 MPPT Charge controllers (1 x 100/20; 1 x 150/35) being intermittent in the VRM dashboard. Just wondering if this is this related to the performance issues, or whether I need to troubleshoot locally (only started happening in the last few days - maybe coincidental with firmware upgrade to v1.61). Installation name is "GarageProject". thanks

(ps I second the appreciation offered by other contributors about the provision of VRM in the first place - I wish you well in rooting out the cause of the problem)

1 Like 1 ·
fasterthinking avatar image fasterthinking commented ·
We appreciate your updates! Nothing can be smooth ALL the time, but good communication keeps everyone happy! Great products, great service! Thanks Victron!
0 Likes 0 ·
marekp avatar image marekp commented ·

@mvader (Victron Energy)

I wonder if the problem with VRM, I reported here:

https://community.victronenergy.com/questions/197273/historical-data-graph-on-vrm-is-not-updating-autoa.html

Was just the first indication of the current problem.

0 Likes 0 ·
pbeelien avatar image pbeelien commented ·

Hello, I am able to access the VRM and have it show real time data when I am on the boat through my boat/marina wifi on any of my devices, but when I leave the boat wifi area, it will only show the data up to when I left the boat either at my home computer or any of my mobile devices ... why? I've tried logging out and back in, but no change. screenshot-2023-05-25-at-73520-pm.jpg

0 Likes 0 ·
screenshot-2023-05-25-at-73520-pm.jpg (424.1 KiB)
gbppower avatar image gbppower commented ·
The last update id worse then before, loosing connection more then before
0 Likes 0 ·
gbppower avatar image gbppower commented ·

This is what i get every hour or so some time faster

1685463039691.png

0 Likes 0 ·
1685463039691.png (33.9 KiB)
Roy Taylor avatar image Roy Taylor commented ·
Portal should not be free maybe charge a small fee but add it onto the Cerbo or add it into some products to make life easy. or allow for X advanced graphs then charge for more .


-3 Likes -3 ·
gbppower avatar image gbppower Roy Taylor commented ·
So you want to make it more difficult for already Struggling people?
0 Likes 0 ·
Roy Taylor avatar image Roy Taylor commented ·
How many active VRM ids are there ?

also wipe all old data older than 1 year.

-4 Likes -4 ·
Frank avatar image Frank Roy Taylor commented ·
I really like seeing many years of data helps understand each year and power usage
3 Likes 3 ·
paul-campbell avatar image paul-campbell Frank commented ·
They are using influx cloud for data.

If they have not set the retention policies and continuous queries up correctly problems will only continue to get worse.

Disabling access to older records competely should give short term performance gains in that is the case.

I didn't so the above and on my home system the DB hit over 14Gb in 6 months. I now reduce data from high res to 1 minute average after 7 days, 5 minutes average after 2 months and so on. This reduces the size of the "in memory" db.

The reason influx sucks (certainly V1.8) is it's an in-memory file mapped DB. This means if someone should access a wide time range query like "Last 10 years", the ENTIRE database has to be remapped back through memory if its not already entirely there. This creates a massive amount of disk access and memory churn, destroys the file cache and other undesirables leading to, in the worst case massive load spikes, and very poor performance system wide (disk, cpu and memory).

1 Like 1 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ paul-campbell commented ·

Hi @Paul Campbell

You're very close to the mark here.

From my reading of the engineering notes, It would indeed appear that it is these long time range queries that are causing the most issues (and likely to remain disabled the longest during recovery from this event).

Each site is now producing more data than ever, we have more sites than ever, and new sites growing faster than ever.

The priority now is to get everything stable again as it is.

4 Likes 4 ·
pkkrusty avatar image pkkrusty paul-campbell commented ·

@Paul Campbell This is a very helpful breakdown of the issue and for me, running my own Influx instance for IOT. Do you know if InfluxDB 2.x is better about this? Or if changing to TSI indexing vs inmem will solve the problem (for a home user)?

0 Likes 0 ·
paul-campbell avatar image paul-campbell pkkrusty commented ·

Honestly I have no idea about V2. For a home user the retention policies should be fine with good reduction continuous queries.


One thing I am still procrastinating about is "States". How do you reduce states. As a software engineer I thought, well, you record the entry state, the exit state and the transition count within. That preserves the meaning of the state graphs in some way. However, doing that kind of reduction in influxdb queries ... well I gave up. The effort did not make it seem worth the outcome. So all my state graphs run for 1 month and that's it.

And what about states which are tri or quad or 5 or a dozen states?


Another approach would be to reduce it like a histogram by day. Hours in one state, hours in another. Again that may need to be done outside of influx though.

"Text" type and basically any "non numeric" type in influx is difficult to work with.

Best idea is to succcumb to the beast and feed it more memory and more concurrent SSD/SSA disks arrays and non-commodity hardware.

Alternatively, give up and state a retention policy of just 1 year in VRM portal and provide an export so people can take their data offline and keep it longer if they want.

0 Likes 0 ·
pkkrusty avatar image pkkrusty paul-campbell commented ·
I would imagine the developers over at Influx hav thought about this. But who knows. Bi-state up to 8-state isn't very many bits compared to storing int_16 or int_32 or float.


I'm going to read up on TSI indexing and maybe clone my raspberry pi and switch to see what happens. I get occasional lockups where Influx is hogging all my processors, probably due to similar reasons as this thread. Then all my queries start backing up and making it worse.

0 Likes 0 ·
paul-campbell avatar image paul-campbell pkkrusty commented ·
The raspberry pi is not exactly a match for Influx. I can't speak for the RPi4, but the first 3 have poor bus bandwidth. So memory or IO intensive processes (influxdb is a multiplying compound of both) are at a significant disadvantage.
0 Likes 0 ·
ronski avatar image ronski Roy Taylor commented ·
Some data is already wiped after six months.
1 Like 1 ·
26 Answers
anrich avatar image
anrich answered ·

These things happen - thank you for the transparency.

Good luck to the teams working on this!

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

UPDATE 5


Dear all,

Good news, right now all functions are working and no issues.

More news later.

All the best, Matthijs Vader

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Jim avatar image Jim commented ·

The overview summary page is not auto-refreshing. If this is on purpose, that is okay, but wanted to make sure it was not missed if everything was re-enabled to operate as before. This is the page: https://vrm.victronenergy.com/installation-overview

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ Jim commented ·
Hey @Jim , good one! I missed that. And indeed, auto refresh of the data, while keeping the page open in your browser, is still disabled.


Note that that doesn’t apply for the numbers shown in the diagram on the Dashboard: real time data will keep updating real time.


I’ll can’t yet say when we’ll enable the auto-refresh again.

0 Likes 0 ·
ikeakayke avatar image ikeakayke commented ·

Hi,

There's many things that could be done, like improving hardware, but don't use ssd, they're obsolete and very very slow compared to nvme arrays with nvme protocol. ram and scalable processing units will also help.

But no hardware will be the ultimate future-wise solution if not a tuned workflow is set in place, and that has 2 parts: 1- avoid common mistakes done by various admins and 2- add optimization layers and linear scalability. 1 is of paramount importance as working from a flawed baseline up is never rewarding, while proper 2- architecture will ease up things as they grow indefinitely.

For 1 look for things like:
- make db give direct answers instead of feeding you data that needs processing elsewhere. most db can do in-flight almost binary-like calculations on their own
- use data-in-transit transports meaning don't buffer them up in ram for subsequent queries, but serve them directly to clients
- check write optimisations, ram buffers, give priority to db, not file cache. (file cache should take place at storage level, not db processing level)
- check proper and indexing refreshing (some can be scheduled so not in real-time)
- check individual data type in relations to its individual storage needs. many admins define much too permissive long fields that are never used but take space and flux time.

for 2:
- separate clients (those asking for db data) from db, from storage.
- never allow clients to talk directly to db. they should query db proxies which should be load balanced themselves, then ask in a load balancing way the db nodes for info. they should also be able to intercept and cache whatever data-flow seems hot in your usercase. purpose to avoid touching db for puerile queries. these can further be augmented with out-of-band pre-caching based on stats. (ie don't wait for a predictable client to ask in order to generate cache)
- obviously proxy caches use different storage from db.
- db should be clustered and if possible split into time-frames say main db with last 365 days, other nodes for older data. proxy or db broker should be in charge of redirecting the queries.
- use out-of-band granularity based cloned dbs. meaning keep native granularity for the common time-frame as 'hot' but also pre-generate other granularities say averaged daily, hourly etc based on the time-frame of the long report asked. if user zooms in, then serve a different granularity. these should almost never need db to parse actual data. these should be pre-generated on cloned dbs.
- try to separate long running queries (but beware most should be resolved by the above already) from short ones at cluster level. ie ask long one to different proxies/brokers/ db shadows to not impact fast short snappy ones.

Finally if there is a need to delete older data (as I've seen people proposing) to keep things running smoothly then something is wrongly implemented.
Regards,

0 Likes 0 ·
Guy Stewart (Victron Community Manager) avatar image
Guy Stewart (Victron Community Manager) answered ·

1st Update -

Due to the server load issue there is now a backlog of new VRM data that needs to be posted to the database.

We are disabling some additional features to allow this incoming VRM data from systems to be posted.

During this time you might see that your system data is not updating, or out of date. This is to be expected, as the vrm site is temporarily paused from requesting the latest data from the database.

Once the backlog is cleared, we will open the site again, and you will see the up to date data again.

Some of the high-load elements will remain disabled until we have a permanent solution.

Apologies again for this - it is receiving full attention.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

M. K. avatar image
M. K. answered ·

thank you for the explanation...


2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mark-a avatar image
mark-a answered ·

Thanks for the notice. I temporarily reduced the log interval to 1 hour on my installation.

EDIT: or rather increased, it's the log frequency I have reduced ;-)

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·
Thanks Mark, a noble gesture!


Just to be clear, regardless of that setting all data will find its way into the database, there is just a delay until it is shown on the VRM page again.

4 Likes 4 ·
thanar avatar image
thanar answered ·

Any chance this is linked to enabling notifications recently on the iOS app? I see a strong connection between the VRM server issues and the latest VRM version that was released for iOS.

2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·

It is too soon to say for certain, however so far the notifications have not been identified as one of high load queries.

There are several different things that are running, some continually - like the dashboard spark graphs, and battery data, others occasionally like some state graphs in advanced widgets, and some on-demand like the reports.

Normally no one feature is to solely blame, and overall they can contribute to a tipping point.

In this case it appears there might be a specific query that is causing most of the issues, the clue being the periodic Monday morning timing.

But this is only my 2nd hand early speculations on the engineers investigative process.


I am certain they will identify the catalyst, improve some efficiencies along the way, and next time have a more effective process for how to manage these load spikes so that normal user experience isn’t so affected.

VRM is a fast growing tool on many fronts, and we already have long term projects underway for scaling ourselves out of these issues. Excuse the techno buzz word babble, it’s a bit hard to avoid sometimes :)

10 Likes 10 ·
thanar avatar image thanar Guy Stewart (Victron Community Manager) ♦♦ commented ·

Techno buzz word babble is always highly appreciated. Keep up the good work.

1 Like 1 ·
daniel20222 avatar image
daniel20222 answered ·

Drück euch die Daumen das ihr den Fehler findet.

Danke für den Service

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hey all,

2nd update: just now we re-enabled the graphs on the advanced page, except for the “state graphs”. Full status:


- Backlog of ingress fully recovered

- Dashboard and installation overview is operational; except that some graphs on the dashboard are disabled.

- Advanced page operational; except for state graphs

- Generator run time data is disabled (from the various downloadable reports and any other place where it normally shows).


In the coming hours we’ll one by one re-enable those features again; while monitoring the load.


Thank you for the encouraging words! Appreciated by the engineers working now to resolve it.

All the best, Matthijs

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

superjoe avatar image superjoe commented ·
Would battery SOC percentages be one of the disabled graphs?
0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ superjoe commented ·
Yes. But it will be enabled soon; if it isn’t already
0 Likes 0 ·
paulcupine avatar image paulcupine commented ·

Can you give us an idea of the size and rate of data you are dealing with? How big is the data store, what is the rate of injest and how many retrieval queries per second are you dealing with?

Does the load vary during the day? I presume, having systems all around the world that it is reasonably constant...

0 Likes 0 ·
regan avatar image
regan answered ·

This might be a good time to consider branching off the VRM into self-hosted supported installs.


Ubiquiti, for instance, has their UISP which is a CRM and cloud management portal for all of their UISP branded wireless gear. They offer a cloud-hosted service as well as a self-hosted install of the identical portal.

Ie: https://help.ui.com/hc/en-us/articles/115012196527-UISP-First-Time-Setup-Installation

This also works great for customers that are in remote spotty connected parts of the world and the like where a local copy just makes sense. But at the same time, it alleviates a huge amount of burden on having thousands upon thousands of devices dumping it's stats into one central location.

Their cloud hosted service is also segmented, they put so many customers onto a cloud instance/piece of hardware and the portal links you to the one you are on when you log in. This allows smaller fault domains and easier scalability. UISP specifically, gives you a subdomain for every customer "instance" at <customer_created>.unmsapp.com. That is then the domain you plug into every device connecting to your cloud instance.


For me, the real shock is just how much I rely on this data to be available, and how utterly inaccessible it is in any readily-available way locally without the VRM. When I started out, I was using a tablet always connected to the Cerbo's remote console, now I have to display both the VRM dashboard and and split screen to my node-red dashboard. The VRM dashboard is incredibly feature rich, but it feels "wrong" in a way to have to pull a real-time dashboard like that from a cloud resource to see status from my install 150ft away.


Anywho, thank you for the updates and working on finding a solution. Just keep in mind that such a great product has also created a large dependency that many of us are eager to see fully operational again!


On a side-note, the main VRM dashboard murders any browser I've tried to keep it running in after about a day. I've always had to use timed tab-reloading addons to keep it functional as a standalone info-wall.

7 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jon-lijnes avatar image jon-lijnes commented ·
I agree fully with regan

Please look at some other system that allows the user to save the data locally and not have to rely on the Cloud

We, here in South Africa are having enough problems with all the power load-shedding that causes the internet to go down regularly.

I would however not be prepared to pay for a VRM system as all other suppliers provide monitoring as part of their service.

Despite it being the most expensive option, I selected Victron because it appeared the best as regards quality and reliability, but if this continues for much longer, I need to consider changing when I need to replace or upgrade my current system.

I hope you are able to resolve the issue ASAP as I use it extensively to program my activities and use of heavy loads.

3 Likes 3 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ jon-lijnes commented ·

Hi @Jon Lijnes and @regan,

Victron has definitely and extensively discussed local solutions for alternatives to cloud VRM for a long, long time.


There are a few main issues that stick;


For those who are really data heavy and not satisfied by VRM, we already have the local version of Grafana, which doesn't require an internet connection, but DOES require a much higher level of technical knowledge. I imagine for someone who is committed to Victron, but who cannot rely solely on VRM could use this, and it would be worth the time and knowledge it takes to get working.


It is possible to run a local Grafana, or private server Grafana, in parallel with cloud VRM all simultaneously as well - for the best of both worlds.

https://www.youtube.com/watch?v=IkNuadRbANA



For those that want the nice and consistent UI, and plug and play nature of VRM (so not Grafana) but without the internet demand we have looked at creating offline self-hosted versions.


Our main issue with this path is that the developers who would need to work on this project are the same ones that are currently working on VRM for our 100,000s of customers who would not want or not be able to run this. So until now the development priority has been on how we can improve VRM for the most people.


The other issue is more speculative, but that it would create a 'many servers' support issue. We presume that problems would arise with the various details of running people running their own servers on different kinds of hardware, different operating system versions, and dependancies. Anyone who has any experience in open source software development will know the tangled knot of support this can create, and ultimately would be a distraction for the VRM developers who are currently able to focus on only ONE type of system configuration.


There would be advantages to that path, and ultimately it may get built out, but I don't think it will happen before we have done everything we can to make the out-of-box experience everything that we want it to be for the vast majority of Victron customers.


On VRM cost - while ultimately Matthijs' decision, for as long as I have been involved in some way with Victron, which has been since the very beginning of VRM (as a user), VRM has been seen as more valuable to be a free service, and part of what makes Victron a valuable product.


For some nice history nearly 7 years to the day on May 21st 2016, I made a post as someone who was considering going all-in Victron at my solar company level on the (then) public Venus-dev mailing list about my concerns around VRM Sustainability.


I got an almost immediate direct reply from Matthijs, and he has been unwavering from that position ever since.

vrm-sustain.jpg

3 Likes 3 ·
vrm-sustain.jpg (528.2 KiB)
tazer avatar image tazer Guy Stewart (Victron Community Manager) ♦♦ commented ·
I would also like to run VRM locally on my own hardware. I have a small server with some Linux containers in my home network, which already runs a bunch of web accessible services (EVCC, Traccar, CheckMK, Rspamd, Nextcloud, MotionEye, Guacamole, RoundCube and many more) and it would be easy to add another.

I understand however that supporting a self hosted version would be a lot of work, it needs documentation, testing on different hardware and operating systems, different release configurations, installation instructions and so on.
Maybe it would be possible to make a "community version" of VRM, with no official support from Victron, but suitable for local installation and special use cases, maintained and supported by volunteers from the community?
Just an idea, I like the VRM portal as is and use it frequently, but also noticed the performance and display problems here and there as others have mentioned.

1 Like 1 ·
ronski avatar image ronski commented ·
I'd be happy with keeping my data local, that would mean I could retain as much data as I wanted, could be as simple as plugging in a USB drive to the Cerbo, or storing it on a local file server. Would require suitable backup options in both cases.
0 Likes 0 ·
paul-campbell avatar image paul-campbell ronski commented ·
The data storage is not my concern. It's easy if you know how to far exceed what VRM provides. Like what about data with a 5 second resolution for 6 months for ALL metrics, not just a select few.


My concern is how it seems to be the only way to configure victron hardware over the network is by some funky file transfer protocol through the cloud, VRM and back, just to configure a device a SINGLE network hop away.

It's just not acceptible in todays world of IoT eco systems.

0 Likes 0 ·
paul-campbell avatar image paul-campbell commented ·

Absolutely.

I would pay for a local VRM... VLM?

I will however not be paying for VRM, because, insert long rant.



0 Likes 0 ·
epuhs avatar image epuhs paul-campbell commented ·
I wouldn't rule out paying for VRM, but I'd love VLM.
0 Likes 0 ·
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hey all,

3rd update: enabling any graph data on the vrm portal; even only the ones on the advanced tab, brings the database to its knees again.

So we’re back one step:

- data ingress works (and is backlog free or will soon be)

- installation overview works.

- dashboard works, including real time data, and including the kWh bar graphs; but excluding battery SOC or any other graphs.

- advanced page doesn’t work at all.

- reports will either not work at all; or if they do then quite limited.


We’ll continue to work on it.


Best regards, Matthijs



4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ikeakayke avatar image ikeakayke commented ·
Hi,

Maybe due to all the improvements in venus, some recently added columns to database are not indexed properly? I remember a recent post for collecting more info into vrm and new columns are usually easily overlooked from tuning.

Regards,

0 Likes 0 ·
will-morgan avatar image will-morgan commented ·

Hey Matthijs, thanks for the update. FWIW I think you guys are doing a great job considering everything that needs supported, and really appreciate the transparency and updates. Good luck and don't burn the midnight oil!

0 Likes 0 ·
stef81 avatar image stef81 commented ·

Hi Matthijs,

If the problem resides in the too big amount of data stored (multiple years, for my installation), will you give us a chance to download it before deleting it, if this is the plan ?

Best regards, SteF

0 Likes 0 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ stef81 commented ·

Hi @SteF81,

This issue is specifically caused by an abnormal load on the database, some small number of queries are causing excessive demand.

The size of the database has its own issues, but it is not the cause of this specific issue (so removing old data will not be part of the solution).

VRM is already (and has always) pruned advanced data details after 6 months.

We have committed to the most common used, and useful, dashboard data for 5 years at least, and that policy is not going to change (except to improve if technically possible).

2 Likes 2 ·
ramdor avatar image
ramdor answered ·

None of the graphs showing on advanced view at the moment (20:36gmt). Showing 'Warning: Something went wrong error' bottom right of web page in browser. I guess there are still issues?

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Neil Swinton avatar image Neil Swinton commented ·
Truth
0 Likes 0 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·
Hi @Ramdor,

With the exception of the state widgets, all other advanced view widgets are now working for me again.

There are still issues, but that shouldn't be one of them right now.

0 Likes 0 ·
ramdor avatar image ramdor Guy Stewart (Victron Community Manager) ♦♦ commented ·
many thanks Guy+team. I must have posted just as you posted about the advanced view being turned off. Cheers.
0 Likes 0 ·
Dean Perry avatar image
Dean Perry answered ·

Some thoughts:

First, the VRM is simultaneously amazing and frustrating on its "bad days". I'm sure you are all working very hard behind the scenes with the scaling issues.

Since the VRM is so amazing it's become a selling point and a significant way for my company to differentiate Victron products and support of them after they have been sold.

Also, it's become an important debugging tool. Any reduction of functionality - particularly in the advanced view, will be felt. Many of our customers are totally off-grid and literally a thousand kilometres from skilled technical support of any kind. Satellite internet has enabled us to support these people and lets us keep their systems operational.

So, if it is a matter of requiring money to scale up on resources (Amazon or Azure or whatever) then by all means charge for it.

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·

Hi @Dean Perry,

Events like this are indeed frustrating for us too when we want VRM to be the best in class remote system monitoring and support tool.


Please see my post above regarding Matthijs' vision for VRM and its economic sustainability.


I don't want to preclude any Victron business decision to allow for a premium service tier, it may be something that we do offer in the future.


BUT


I can say with confidence that such an offer would take VRM development time to deploy, and may even undermine a larger value proposition that VRM provides.


Right now VRM isn't something you need to 'up-sell' to a customer, which in some cases will cause a cost conscious customer to say no.

Then everyone is really worse off because VRM is great value and super useful, the marginal cost to Victron of supplying VRM to that customer is relatively small, but the value that would be lost for that customer, and their installer is relatively large.

We love VRM and we want everyone to use it and love it too.

The fact that it is available for almost all large Victron system customers now (as most will also need a GX device for another reason) means that you can be very efficient in using it as a consistent feature across your fleet, without worrying if this one has this feature or not.


Now that said we DO WANT to encourage VRM dealers and installers to sell and charge their customers for providing premium service!

Your teams ability to take the data that you know is there for all connected customers, analyse it, and support the customers system through issues is valuable and time consuming.


Victron has committed to providing the VRM platform from our system sales (GX+Multi+MPPT).

Matthijs and I then think that the ongoing monitoring and support service that Victron is asking installers to provide to their end-customers should be sustained via ongoing service contracts to fund that support.

Over time the support load on good installers will continue to grow, and their business model needs to grow with it.

4 Likes 4 ·
Rob Duthie avatar image Rob Duthie Guy Stewart (Victron Community Manager) ♦♦ commented ·
Hi Guy


Well said, it is the most useful tool for fault finding and remote assistant to my end clients, So far to date it is the best solar tool in use to date the Chinese and European, Except Victron of course cloud remote site are just terrible in control and very basic and buggy.

Regards

Rob D

NZ

Installer

1 Like 1 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Hi @Dean Perry , if money could solve this, then it would have been solved already. I can assure you that that is not the problem.
3 Likes 3 ·
acama avatar image
acama answered · 
Hello to all interlocutors.
Hello to all the VICTRON team.
I come to this forum to congratulate the entire VICTRON team and encourage you in the wonderful work you are doing.
When I see the potential and all the functionalities that you give to users in the VRM platform, I think that your engineers are simply geniuses.
Your VRM platform is so powerful that we use it constantly to support our customers. This is why when there is a problem, we feel the lack but given the power of the VRM tool, it can be understood that errors can occur.
Thank you for everything you do to repair and good luck to the whole team. You are the best.
Hervé FIBLEUIL.
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

baronner avatar image
baronner answered ·

Thanks for the red banner on the homepage, the link to this forum and the updates in this message! Good luck with solving the issue!

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

petajoule avatar image
petajoule answered ·

We do appreciate the transparency of how Victron is handling this, also we're not pointing fingers as these things happen and it's more important to work on fixing rather than blaming. Besides, it's not that similar things haven't happened elsewhere or to us in the past.


How.Ever.


This is a premier example why one does not want to be dependent on a cloud service (rather centralistic as vendor-provided) and have some local solution instead. I can remember well how Matthijs Vader stated clearly that Victron would not support any private "VRM like" solutions.


IIRC it was in the context of VenusOS providing logging data locally, or opening up the GX <-> VRM communication protocol. Well, after this issue is sorted out it would be about the time to rethink that position - yes?

Because personally, currently I feel that if Victron is not going to make it easier to set up "ones own VRM", I will have to go hard on the reverse-engineering DIY route soon.


2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Matthias Lange - DE avatar image Matthias Lange - DE ♦ commented ·
@PetaJoule A working internet connection or VRM-Portal is not needed for a working Victron System, it's just a bonus for monitoring.

You can run monitoring locally with Grafana for example and with a VPN you can access you remote console without the VRM-Portal.

Only accessing the settings of VE.Bus, VE.Can or VE.direct devices can be difficult without the VRM-Portal.

1 Like 1 ·
petajoule avatar image petajoule Matthias Lange - DE ♦ commented ·
Will check it out - thanks for the pointer.
0 Likes 0 ·
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

4TH UPDATE


Dear all,

After having enabled most of the graphs on the advanced page some 6 or more hours ago, and making various improvements, we're now re-enabling graphs on the dashboard again.


Current status:

- data ingress works (and is backlog free)

- installation overview works.

- dashboard works, including real time data, and including the kWh bar graphs; and in a few minutes will also include battery SOC or any other graphs.

- advanced page works.

- reports will either not work at all; or if they do then quite limited.


All the best, Matthijs Vader

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mark-a avatar image mark-a commented ·

Thanks for the great service! Just for the record: on May 11th I spotted an irregularity which persisted until the advanced graphs were disabled and has disappeared since they are available again. With my custom widget the curve styles as defined on the edit page (left side) were different from the styles as displayed in the main view (right side).

vrm-graph-prob-20230511.jpg

0 Likes 0 ·
vrm-graph-prob-20230511.jpg (1.8 MiB)
nonlinear avatar image
nonlinear answered ·

I personally am grateful for VRM and the work done by the Victron team and thus do not wish to fan the flames further; surely some or all of the requirements for local data that are raised in the posts above must be available via the MQTT facility provided?

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

tuxedo0801 avatar image
tuxedo0801 answered ·

To anyone complaining about not having a local VRM alternative:
Just enable MQTT and subscribe to what you need. Then push it to an own instance of InfluxDB and use Grafana.
If you're skilled to install a "local VRM" (if it would exist), you're also skilled to push the information you need to your own influx/grafana.

All in all: VRM is great. Thanks a lot for this easy to use and free solution. If you want more: Just do it yourself. This is no rocket-science ...

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

pkkrusty avatar image pkkrusty commented ·

Agree with this post. I use the grafana/influxdb solution and it works well. Local logging, I can control the granularity (up to 2 seconds!) and how long data is stored, when VRM is down, I can VPN into my network and view my pretty graphs.

screenshot.png

Plenty of guides on the internet how to put everything together. Victron even made a nice docker solution:


https://github.com/victronenergy/venus-docker-grafana/blob/master/README.md

VRM is a complex beast, and even if Victron made a local hosted option, any updates or mal-formed queries would put the local administrator in the same boat as Matthew has been in the last day. Being a database administrator is tough.

4 Likes 4 ·
screenshot.png (607.9 KiB)
ako avatar image
ako answered ·

I agree with the other poster , the red banner informing everyone of the situation with VRM showed consideration for Victron Clients and saved a lot of headbanging . Unfortunately i didn't see it for several hours until i logged on again on a different Laptop , until then i was scratching my head wondering if the problem was at my end , was such a relief to see that it wasn't .

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

polonioli68 avatar image
polonioli68 answered ·

Hola, tengo un problema para acceder a nodered a través del portal vrm, el acceso es demasiado lento o no. ¿Alguna idea? muchas gracias.

p.d for access nodered dashboard its ok but nodered for deploy not

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

polonioli68 avatar image
polonioli68 answered ·

Hello, I have a problem to access nodered through the vrm portal, the access is too slow or not. Any ideas? thank you so much. nodered dashboard accesso fine but nodered to program no

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ionut avatar image
ionut answered ·

In the last 2 days I am unable to login to VRM due to missing 2FA prompt.
It is a known issue?

vrm-login-issue.png


vrm-login-issue.png (28.2 KiB)
3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ionut avatar image ionut commented ·
@Guy Stewart (Victron Community Manager) Shoud i be worried about my login issue, or is a general situation?
0 Likes 0 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ ionut commented ·
I’ve just asked the VRM team to get in touch with you via the email address you’ve used to register here
0 Likes 0 ·
ionut avatar image ionut Guy Stewart (Victron Community Manager) ♦♦ commented ·
Perfect! Thanks a lot for quick support and agility.
0 Likes 0 ·
polonioli68 avatar image
polonioli68 answered ·

Good morning ,now the problem with nodered acces via Vrm is solved (I think it was the support people who repaired it because I didn't do anything from here)

Now the other problem not possibel connected via mqtt to server ,


thanks a lot for quick support .

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ionut avatar image
ionut answered ·

My issue is solved too now. After login I see the missing field "SMS verification token".
Great job! Long Live VRM :-)

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Brian Brown avatar image
Brian Brown answered ·

@mvader (Victron Energy) Are you still having problems with state graphs on the Advanced page? I'm happy to not load these if you still need time to work things out.


1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·
Hi @Brian Brown


There are currently no issues.

The VRM team is making a few changes to help reduce the extra load created from the most demanding elements - like you’re seeing there with the state graphs.

Feel free to use VRM exactly as you would like.

1 Like 1 ·
gbppower avatar image
gbppower answered ·

My system is down again


2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jetlag avatar image jetlag commented ·

My VRM is working absolutely fine, and no one else is reporting problems with VRM again this day... So maybe a problem on your side?

0 Likes 0 ·
Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·

Hi @gbp.power

Currently there are no known issues or other reports, so you’re going to need to be a bit more specific with the report.


As this specific performance issue is considered resolved, it would be best to create an entirely new question with some screenshots and more details about what exactly isn’t working.

0 Likes 0 ·