Data upload issues

Probably discussed before but I cant find any related post.

Issue 1:
My Rasberry Pi has a relay hat I use one of the relays to controll a much larger 24v 100a relay as a battery protect device, wont go into the details but gives me way more flexability that the victron battery protect doing it this way with nodered. The Pi is independantly powered not via these relays and stays on when there is a low battery state. The router does get turned off and as such no data is uploaded to the VRM Potal during the low battery condition. This only really happens when I have a week of cloufy days and its not enough to keep the battery topped up.
I was expecting the Pi to cache the data and upload it once the router turns back on but I’m left with huge gaps in my data where I have no idea what was going on. This makes it very dificult to troubleshoot and find out why the system was off for a week. Is there a flag or something I can change in a config file to modify this behavior?

Issue 2:
The Pi has suddenly started consuming 3x the power. It’s gone from about 10w to 30w. Is this a firmware update thats turned some more stuff on? or more likely the Pi has gone faulty.
I have the wifi turned off but left the bluetooth on for other sensors or emergency access.
Nothing has changed in the config since this problem started occuring.

Thanks in advance

Greg

re Issue 1: there’s a ‘reboot device when no contact’ option in one of the config menus and you can set a delay time (from 1 to 23 hours). I’m off-grid, using a RPi 3 permanently powered and have a router on a time switch, operating for 1h per day. Cache offline and upload has been working well. On two occasions the RPi has stopped recording, and needed a reboot. Data is still lost but only 23 hours (in my case) instead of many days when this first happened.

How do you turn the cached upload on?
My router comes on when the battery is over 50% using a relay but the system never uploads data for the times when the router is off.

AFAIK this is automatic, but there are a few reports of it not working on Pi.

Have a look in Settings / VRM Online Portal , scroll to the bottom

Free disk space is your SD card, but AFAIK this is not where data is logged when the VRM connection is down, because this is the OS drive - just a small amount of data (a few hours?) is stored in RAM.

“No storage found” means no additional USB or SD is available (the SD card that VenusOS is on can’t (or wont) be used by VenusOS to buffer logs - it has to be a USB port or SD slot, like you would find on a Cerbo, Cerbo-S, Ekrano etc. Pi doesn’t have a 2nd SD slot unless you add a hat, so USB is your only option.

So … I think you might have to populate one of your usb ports with a usb drive formatted as FAT.

RP>

FAT as in FAT16 or FAT32?
I cant help but think this is a linux kernel and would likely support a ext3 file system rather than a filesystem invented by Microshaft?

Can anyone clarify? There is a fair bit of travel involved to do this and I don’t really have any other reason to go there in the winter.

This solution works. I’m now datalogging all night after the battery protection cuts in. I’m not getting some odd occasional drop outs. This doesn’t make much sense to me as if it can log all night when the router is 100% off, why would I get a drop out between 12:00 and 13:00 when the router is 100% on when the system now logs to USB. Weird!!!. I can confirm FAT(16) 2GB partition on a USB stick works.

VenusOS is certainly built on a linux kernel (64bit) - I can’t think of a device built on a windows kernel that has been even moderately successful, maybe WindowsCE but that only survived about 10 years before it was on the endangered list, or WindowsPhone which lasted about 5 years. Products using a MS kernel just don’t perform in terms of speed, stability, security, and longevity in the developer space.

You can usually measure the life of a product that MS have written in months or years. However, products they stole, bought, or acquired, or where MS were only part of the development of, have sometimes done well. FAT is a good example, where there were about 5 companies involved in developing FAT at the start (FAT12) which probably diluted the poisoning effect of “the MS way”. Once FAT became a de facto industry standard, it was entrenched, and despite much better file systems since then that are less prone to corruption and issues like fragmentation, FAT remains the go-to for most removable media.

The issue with running an ext3 or ext4 on the card is that hostages from windows won’t have the tools to work with the cards - *nix has supported FAT and NTFS for decades, windows still doesn’t natively support ext3 or ext4, and that is purely a marketing decision, which loops back nicely to the first paragraph as the reason MS has the reputation for crap, buggy, half thought-out software - it’s absolutely in their financial interests to put more work into lock-in, lawyers, and patents than into first class software, so enshitification and 2nd rate software is the natural result.
It hurts their users to not have a tiny ext3/ext4 tool natively, but it doesn’t hurt their bottom line.

With the logging;
I think there is a period where the device logs to memory if the connection is down, and only after a certain length of time does it start logging to non-volatile memory (i.e. the USB stick). You have to remember that these devices have to work for decades, so you must keep the number of writes low where the media is the sort that wears out - which the onboard media on a Raspberry Pi or Cerbo does.
Memory such as that used by a BIOS only gets written when you change settings, so you can expect to have maybe 0 to 50 writes per year, most machines will be writing to the non-volatile BIOS memory (CMOS) maybe 5 times in their whole life.
Memory that only gets written on boot will typically be written up to 5k or 10k per year - think of a mini computer board in a vehicle like a taxi, maybe it gets booted at every engine start.
Memory that gets written whenever there is a logging event can range from near zero to tens or hundreds of thousands of writes per year, depending on the logging level.
Memory that gets written to log things like solar performance will typically be writing something every 1 to 60 seconds, so for 1/sec that might be 30 million writes per year - this would wear out some types of memory very quickly.
So most small boards (like Cerbo or any GX devce, Pi, ESP32) tend to write to DRAM memory (where the wear rate of the memory is so low it might not be able to be calculated) and then if necessary, flush to disk/SD/USB every now and again.

With VenusOS devices, the rule goes something like;

  • log to fast, volatile memory
  • When the upload timer expires, upload the logged data (with the upload timer you can upload larger packets less frequently, which suits some systems that are on 4G or other expensive data plans)
  • If the connection is not available at this point, write to non-volatile memory (if available) for later uploading
  • If the connection is available, and there is data on the non-volatile memory, upload all of it.
  • Once data is uploaded, it is cleared from the non-volatile memory (or marked as uploaded)

I think this is what you are seeing - some data is logged to (fast, volatile) memory, and before it gets a chance to log to non-volatile memory, a reboot occurs. At this point, all data that was in volatile memory is lost.

So if you want to debug this, you will need to confirm that the GX is indeed staying powered up 24/7, and then see if that “Stored records” counter is doing. You might be able to graph that by regularly pulling data from MQTT if it is published in a queue.

RP>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.