question

benc-ipind avatar image
benc-ipind asked

Gaps in logging on RPi Venus OS

I have an RPi 3B+ running v2.87 with GuiMods on a 32GB SD card.


The partition size is the default, so there is ~11GB of available space for the OS to write to.


I am finding that when Venus cannot see the internet for a day or so, it stops logging to the card intermittently.


Rebooting, full reinstallation and even using a different SD card does not resolve the issue.


Logging is set at 1min intervals.


Any assistance is greatly appreciated.


Ben.

Here is a month worth of data to show the random gaps:


1658094452899.png

VRMVenus OSRaspberry Pi
1658094452899.png (81.9 KiB)
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

9 Answers
Louis van der Walt avatar image
Louis van der Walt answered ·

Check that you have an large enough power supply for the Pi. If you use a few USB connections or perhaps a hub you need at least 2.5A PSU.

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image benc-ipind commented ·
Thanks.


It's on a 5A 12V to 5V PSU.

0 Likes 0 ·
johanndo avatar image
johanndo answered ·

Did you auto-reboot the pi when connection to vrm is lost or for any other reason?

The pi has no RTC, so the timestamp of your logs are then 1970, and your data transferred to the VRM has this time stamps (epoch data). You may add a gps receiver or a rtc as time source when the internet is unavailable and you need to reboot your device. Without getting time from a ntp server over the internet or another local source you will loose the logs.

The logs will be fine as long as the pi remains running and is not rebooted , it counts the time internally during the offline phases.

6 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image benc-ipind commented ·
Hi @JohannDo,


Thank-you for coming back to me.

The 'reboot on lost connect' option is turned off.

I suspect you may be on the money about the RTC thing though. The RPi is probably only getting it's time via NTP when I have the boat close enough to my home WiFi or when I turn on mobile hotspot on my phone.

If it gets rebooted without either of these internet sources available, then that would explain the 'gaps'.

Are you able to assist with getting a DS1307 or DS3231 RTC module working with Venus OS?

1 Like 1 ·
johanndo avatar image johanndo benc-ipind commented ·

Sorry, we are sailing in Colombia and later to San Blas with no access to parts and very little Internet access. But it shouldn't be a big issue to add a RTC to the Pi I think, there are lots of examples on how to do it.


I have solved the issue the other way around (i hope), I have a U-blox GPS mouse connected to my RPi / Venus that is supported out of the box, it provides location and speed data to the Venus OS. A great benefit / side effect is, GPS is a very precise time source too. I installed a ntpd deamon on the pi and configured it to gather the time from the GPS receiver and propagate it to the local network (and use it itself of course). So I should have a reliable rtc this way, but I still avoid rebooting my pi, it hangs / halts sometimes instead of rebooting and I have to temporarily unplug the sensors to get it back running. Have not found a workaround for this yet

To install any missing services (dnsmasq, ntpd, hostapd or whatever) you can use the opkg package manager, most essentials are there.

.

1 Like 1 ·
laurenceh avatar image laurenceh johanndo commented ·
I have a shell script running from rc.local that collects the time and date from the connected GPS module and sets the Rpi time and date on every reboot. A simple fix and easy to install an no extra hardware or tinkering required. I’ll post the script tomorrow.
1 Like 1 ·
Show more comments
benc-ipind avatar image benc-ipind johanndo commented ·

Thanks.


I just went and picked up a DS3231 RTC module and have got it working. You can also use a DS1307, but the DS3231 is more accurate.


Now to see if it resolves the issue.


Here are the instructions on how to make the RTC module work on Venus OS. The DS3231 modules work with the DS1307 kernel module.


Download the required overlays:

ds1307 rtc overlay download: https://github.com/PiNet/PiNet-Boot/raw/master/boot/overlays/ds1307-rtc-overlay.dtb

i2c-rtc overlay download: https://github.com/PiNet/PiNet-Boot/raw/master/boot/overlays/i2c-rtc-overlay.dtb


Copy the .dtb files to /u-boot/overlays

Add the following to [all] section of /u-boot/config.txt

#ds1307 rtc interface - lives on the i2c bus
dtoverlay=i2c-rtc,ds1307-rtc

Save the changes

Go to /data and create and rc.local file then edit it and add the following line:


echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device
hwclock -s

Save and exit

Make the file executable

chmod 755 rc.local


Install the RTC kernel module

opkg install kernel-module-rtc-ds1307


Reboot and run the following command to check the RTC time

hwclock -r







0 Likes 0 ·
Show more comments
johanndo avatar image
johanndo answered ·

Well done, easy peasy...

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

laurenceh avatar image
laurenceh answered ·

So this is an answer of sorts:

If you have a connected USB GPS module, and most do seem to work. One of the features of GPS is that is has to know the time quite accurately to work, and it get the time from the GPS satellites.

And this data is available in the messages being transmitted to the Venus device serially over the USB. You can see the data as it arrives:

cat /run/serial-starter/gps/ttyACM0

The data format for each line is well documented if you search online.

So here is what I have done.

/data/rc.local is easy, add this line:

/data/gps.sh > /var/log/gps-date &

Now the bit that took time to write and debug. Don't ask me how it works I did it a couple of years ago and shell scripting is arcane.

The script gps.sh is:

#!/bin/bash
#add this line to /data/rc.local
#/data/gps.sh > /var/log/gps-date &
N=0;
FILE=/run/serial-starter/gps/ttyACM0
echo "Waiting for GPS data"
while [ ! -e $FILE ] && [ $N -lt 50 ];
do
    sleep 1;
    ((N++))
    echo -n "."
done;
sleep 5;

if [ $N -ge 17 ]; then
   echo " GPS data not available is the device plugged in"
   exit 0;
fi

echo
echo "Device attached waiting for data"

input=$FILE
N=0

# check if GPS has FIX
LINE1='^\$..GGA,[0-9\.]+,[0-9\.]+,[NS],[0-9\.]+,[EW],([0-2])'
# Catch lines with date and time info
LINE2='^\$GPRMC'

FIX=""

# extract date and time from GPS packet

function extract {
[[ $1 =~ ^\$GPRMC,([0-9]{4})([0-9]{2})\.[0-9]{2},.,[0-9\.]+,[NS],[0-9\.]+,[EW],[0-9\.]*,[0-9\.]*,([0-9]{2})([0-9]{2})([0-9]{2}), ]]
  echo "found this"
  COMMAND="20${BASH_REMATCH[5]}${BASH_REMATCH[4]}${BASH_REMATCH[3]}${BASH_REMATCH[1]}.${BASH_REMATCH[2]}"
  echo command is
  date -s $COMMAND
exit 0
}

while IFS= read -r line && [ $N -lt 2000 ] 
do
#  echo "$line"
  [[ $line =~ $LINE1 ]] && FIX=${BASH_REMATCH[1]} && echo "GPS device has a fix after $N lines"
  [[ $line =~ $LINE2 ]] && [[ $FIX -eq 1 ]] && extract $line
  ((N++))
done < "$input"
echo "Timed out no date/time information in 2000 lines from GPS receiver" 

The log output looks like this:

Waiting for GPS data


Device attached waiting for data
GPS device has a fix after 1 lines
found this
command is
Sun Jul 24 19:25:48 UTC 2022

The fix after n lines will take longer (than 1) on a system that has just been switched on.





2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image
benc-ipind answered ·

Hi all,


Just an update on this.


The time/date has been fine with the installation of the RTC module.


What has not improved it the gaps in the logging. I have now confirmed, across multiple RPi 3B+ running the last few releases of VenusOS; that the logging will stop after ~2,900-3,200 records are written to the SD card.

1665179252841.png


1665179335536.png


Am I missing something? There is no storage issue with ~11GB available.


I did see this in a post relating to the Venus GX:

Data logging: 
When connected to the internet, all data is sent to the VRM Portal. When there is no internet connection available, the Venus GX will store the data internally, up to 48 hours. By inserting a micro SD-card or USB stick, more data can be stored. These files can then be uploaded to the VRM Portal, or offline converted with the VictronConnect app, for analysis.


The 48hrs lines up with what I'm seeing.


I do not see this issue on real Cerbos with a USB storage device plugged in. I've got customer installations that have had 3 months+ of data stored.


Ben.


1665179252841.png (718.1 KiB)
1665179335536.png (780.0 KiB)
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image
benc-ipind answered ·

I believe I've resolved this by running:

/opt/victronenergy/swupdate-scripts/resize2fs.sh


The location where the logs are written has gone from 403.5MB to 9.2GB.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image
benc-ipind answered ·

Unfortunately, this has not resolved the issue. :(

The partitions prior to running the script:

1665435299075.png


and after:

1665435321609.png


but the logs are still topping out at ~3,000 records/2 days.


Can anyone please assist? @mvader (Victron Energy)


1665435299075.png (91.0 KiB)
1665435321609.png (90.5 KiB)
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

bathnm avatar image
bathnm answered ·

@benc_ipind the VRM temporary store is not held in / but in /data/db directory. You have loads of storage there, so that is not the issue.

Looking at one of your screen shots you are getting error 153, that should give you an idea of what is happening. Take a look at this part of the VenusOS manual.

It says

Error #153 Connection error: this could indicate a routing problem. For details, check the shown error message:

So you need to scroll down on the VenusOS screen and see what the error message shown is. That will give you some indication of the issue. It could be a DNS issue, routing issue, etc. The manual has two suggestions on what an Error #153 could be.

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image benc-ipind commented ·

@Bathnm thanks, but Error 153 is expected because there is no permanent internet connection.

The device could go for weeks or a month without having access to the internet.

The purpose of this query is to attain logging beyond 2 days. If it had a full-time (or at least once every 1-2 days) internet connection; there wouldn't be an issue.

0 Likes 0 ·
jeroen avatar image
jeroen answered ·

The logger has a concept of "Internal storage" to /data (flash normally) and it will only keep logs there for 2 days and "External storage" to a sd-card or pen-drive. Since on a RPI the sdcard is the "Internal Storage" logs will be limited. An easy way to make it work is just insert a pen-drive (not tested).

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

benc-ipind avatar image benc-ipind commented ·

@Jeroen yes - that was the line of thinking I was going down now. There appears to be a hard coded limit to only allow 48hrs of internal data collection. So, it's a time based limit, not a space based limit.


Whilst plugging in USB storage would likely be a solution, it's completely unnecessary.


It would be nice if Victron could provide an option in the VRM settings to set the location for the logging (Internal/External) so that the massive amount of (otherwise wasted) available space on the SD card could be used.


An alternative would be to set a space limit instead of a time limit. Something like "Until available =100MB".

0 Likes 0 ·