500 Internal Server Error Venus GX device - maybe a memory leak?

Goodmorning,

When trying to connect to a Venus GX device by using a local ip-number I receive the response ‘500 Internal Server Error’ nginx/1.24.0 (doesn’t matter if I want to connect to the old or new UI).
VRM is still getting ‘Real time’ information from the device except the values from the tanks.

Given the fact temperature and tank values are starting the problem I have the idea something is wrong there. Connecting to the console from VRM still works.
This is how I reset the site by using the ‘Reboot’ function within the console. After a Reboot the site works for 2-3 days than the problem starts again.

If I leave the Venus GX device to long in this state it will eventually lock up, impossible to reach from VRM and no data coming in. I then need to go to the site and power down the device. I now reboot the device every 3-4 days.

Does the victron staff have an idea?

I have a Cerbo GX device ready to replace the Venus device. But before I swap it I love to know if this is a hardware failure or a software memory leak problem? Regards,
Robert

This situation is going on for a few months now. I tried multiple firmware version but currently I am on 3.73-1.

I forgot to mention that the network settings also change after the look up. If the network setting was on a fixed IP it changes to a DHCP configuration.

3.73~1 is a beta. 3.73 is released now.
Try connect using venus.local instead of the IP.
You can also try to factory reset the GX.
Do you have any modifications or drivers loaded?

Hi Nick, thanks for your reply.

3.73~1 is a beta. 3.73 is released now. I know I am having this problem for months now (since the 3.6 version). Beta or not, makes no difference. I am now on the 3.73 version, same thing happens.
Try connect using venus.local instead of the IP. Same problem, 500 internal server error.
You can also try to factory reset the GX. I can try doing this and leave the tank settings out.
Do you have any modifications or drivers loaded? Nope, nothing, all standard.

I will try the factory reset and leave out the tanks settings. Will let you know.

It happens in stages:
first stage - not able to connect 500 internal error - happens in a few hours.
second stage - tank and temperture doesn’t get send to VRM - 1 day
third stage - complete connection lost with vrm - 2-3 days.

Robert

@robert.casadoouro Just another angle. The 500 error is rather generic and resource limitations (storage, cpu, timeours etc) could be also be a cause especially when you say it started at 360. With the Venus now a discontinued product, if you have the luxury of time, roll back to maybe 355 (updates for CCGXs are no longer pushed after 355 ) after trying a full factory reset to also clear the storage. If these solve the problem, which seems to be an isolated instance at this point, great. Otherwise the Cerbo GX MK2 is the way to go as with so many new os integrations and enhancements, the Venus probably is showing its limitations after many years of service. PS - Worth installing this widget to see how the Venus is coping.

Are you using a micro sd card for extra memory in the gx device?

If the card is corrupt, things freeze and lose communication, especially with the cerbo devices.

You could try removing the card if one is fitted?

Thanks for your reply. The Venus GX is still supported (see the beaglebone releases) and there is no reason to revert to a older version nor to buy new bigger and better. The victron staff have done some changes on the way tank information is processed. Besides some of the changes done in the latest releases are needed. Not long after these changes the errors started. As mentioned there is a structure in how the errors evolve.

Take care ,
Robert

Thanks for your reply. I am not using any sd card for extra memory.

Thanks,

Robert

@robert.casadoouro Curious that yours appear be an isolated case so far? Did a factory reset yield any different result?

Goodmorning,

I am a isolated case because I use the tanks and tankreaders not inside a boat but for agricultural use (furtilizer). The tank readers are used from bottom till top, using the full length of the wema tank sensor.

On the 3.73 version I switched off the tank devices and up until now no more errors and no 500 Internal server error. I have done No factory reset or anything else, just stopped taking information from the tank readers. The devices where on a ‘custom’ tank setting, see picture.

Sensors values can be below or above the entered Empty and Full setting.

To me it looks like there is a ‘bug’ when using ‘custom’ tank settings. I will try to confirm this in the next few weeks.

Robert

Just adding, I am using Wema tank senders hard wired to the ekrano (3 tanks) and I am not having the problems you describe. I dont think its tanks in particular. And yes I am running v3.73

You could try to enable the superuser settings and then SSH and log in to the cerbo via SSH to monitor system processes and memory.

https://www.victronenergy.com/live/ccgx:root_access

You haven’t posted your dbus round trip time widget as was requested above, this will provide a guide to load.
The more devices (sensors) the greater the load on the GX and the old ones, while supported are constrained.

You can also turn off GUI animations in the UI settings, this frees up some CPU.

Goodmorning,

A few days ago I performed a factory reset on the venus gx device (chapter 19.1 of the manual). After the factory reset the device can’t be reached anymore. No repond on any ip-number (lan or wifi), nor with victron connect, nor does it communicate with VRM. It asks for an IP-number from the DHCP stack, that’s it.

Having the correct measurements of the tanks is very important for me. Therefore I spend some time replacing the Venus GX asap with a Cerbo GX. The Cerbo GX is running on default Tank settings (Europe) regarding measurements. All works fine for now (3.73 OS version). In a week or so I will try the custom Tank device settings to see if I stay out of troubles.

Regarding the used and broken venus gx, I will try to reinstall the OS (19.2 manual) to see if it comes back to live. To be honest, I think this venus GX device is probably dead :skull: .

To Nick: thanks for the advice. More than a few months ago I already turned off the GUI animation, didn’t make much of a difference. In the beta version of the OS there is a need option ‘Enable frame-rate visualizer’ that gives a need presentation of how busy the CPU is. Can’t remember how high the value was but it responded quick enough during a period of time. I also compared it to other GX device I have. I looked at the frame-rate to see if I should increase the reading interval on the tanks values (default 10 seconds). During the three stages I described before there wasn’t much of a difference.

Thanks,
Robert