question

kivanov avatar image
kivanov asked

GX device logging to ram (tmpfs)?

I am setting up my GX device (Raspberry Pi2 with Venus OS 2.87) and I am curious to understand how does this device handle logging and the constant and frequent write operations to the file system, t.e. to the micro SD memory card. I had bad experience with embedded systems running frequent or/and intense write operations like logging on flash memories and I have always tried to move as much as I can to tmpfs.

For what I see, services work files seems to be kept running in tmfs, however the system logging is written to /var/log which is mounted under /data (separate partition of the media).

Since I am new to venusos, is it possible to move the logging entirely to tmpfs and whether this will be ok with the venusos daemon mechanisms like dbus, etc, which I am still not familiar with?
Probably a script as a service like log2ram can be implemented for this? It uses systemd, but I believe this can be easily integrated to venusos.

I hope that someone else (like @Kevin Windrem with his super useful modifications scripts) will get interested in this.

[root@raspberrypi2:~]# cat /proc/mounts
/dev/root / ext4 rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=469540k,nr_inodes=117385,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /var/volatile tmpfs rw,relatime 0 0
/dev/mmcblk0p1 /u-boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 0 0
/dev/mmcblk0p4 /data ext4 rw,noatime 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
overlay /service overlay rw,relatime,lowerdir=/opt/victronenergy/service,upperdir=/run/overlays/service,workdir=/run/overlays/work/service 0 0
[root@raspberrypi2:~]# ll /run/overlays/service/gui/log/supervise/
drwx------    2 root     root         120 Jul  2 12:53 ./
drwxr-xr-x    3 root     root          60 Jul  2 12:53 ../
prw-------    1 root     root           0 Jul  2 12:53 control|
-rw-------    1 root     root           0 Jul  2 12:53 lock
prw-------    1 root     root           0 Jul  2 12:53 ok|
-rw-r--r--    1 root     root          18 Jul  2 12:53 status

[root@raspberrypi2:~]# ll /var/
drwxr-xr-x    9 root     root        1.0K May 31 13:40 ./
drwxr-xr-x   21 root     root        1.0K Jul  2 07:57 ../
drwxr-xr-x    2 root     root        1.0K May 31 13:32 backups/
drwxr-xr-x    4 root     root        1.0K May 31 13:40 cache/
drwxr-xr-x    7 root     root        1.0K May 31 13:39 lib/
drwxr-xr-x    2 root     root        1.0K May 31 13:32 local/
lrwxrwxrwx    1 root     root           9 May 31 13:40 lock -> /run/lock/
lrwxrwxrwx    1 root     root           9 May 31 13:32 log -> /data/log/
lrwxrwxrwx    1 root     root           4 May 31 13:40 run -> /run/
drwxr-xr-x    4 root     root        1.0K May 31 13:39 spool/
lrwxrwxrwx    1 root     root          17 May 31 13:40 tmp -> /var/volatile/tmp/
drwxrwxrwt    5 root     root         100 Jul  2 12:53 volatile/
drwxr-xr-x    4 root     root        1.0K May 31 13:39 www/
[root@raspberrypi2:~]# ll /data/log/ssh-tunnel/
drwx------    2 root     root        4.0K Jan  1  1970 ./
drwxr-xr-x   28 root     root        4.0K Jul  2 12:53 ../
-rwxr--r--    1 root     root       14.9K Jul  2 12:53 current*     # -> this is the actual log for this service
-rw-------    1 root     root           0 Jan  1  1970 lock
-rw-r--r--    1 root     root           0 Jan  1  1970 state
gx device
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

2 Answers
kivanov avatar image
kivanov answered ·

I found something interesting related to the topic.

There is a startup scrips that actually checks the state of the /data partition verifying that it is ok, and if it finds problems with it (ro state, failed to remount, etc), it leves it as ro and mounts exactly the logs with tmpfs which prevents failure of most services to start.

[root@raspberrypi2:~]# cat /etc/init.d/test-data-partition.sh 
#!/bin/sh
### BEGIN INIT INFO
# Provides:          data partition check
# Required-Start:
# Required-Stop:
# Default-Start:     S
# Default-Stop:
# Short-Description: monitor if data doesn't becomes ro
### END INIT INFO

# Monitor that the data partition doesn't get remounted as ro, as in filesystem
# trouble, mainly intented for ubi. Note: is-ro-partition depends on a small
# linux patch to notify this script that auto ro remount has occured. If that
# happens somewhere early in init the machine won't boot since many processes
# keep spinning and most of the cpu time is spend in "sys". The watchdog will
# kick in before something is shown and it creates a continues reboot loop. For
# completeness, if it happens later on, the machine keeps running, but fails in
# odd manners since files / settings can no longer be stored. So instead,
# directly reboot when this happens and store that it occured. Since writing to
# data is no longer possible, u-boot env is used instead.

# There seems to be two different cases. One is recoverable with a reboot, the
# other one, ECC failures, is persistant. So the following is done, if data
# becomes ro the first time, it is stored in u-boot env and the machine is
# rebooted. If after a reboot the data parition doesn't become ro for quite
# some time, u-boot env is updated that the device had an incident, but
# recovered from it. If it becomes ro again, the device is marked as broken
# and whatever is needed to report that event to the user and vrm is done.

if [ "$1" != "start" ]; then
exit 1
fi

sf=/run/data-partition-state
uvar=data-failed-count

# first, lets see if data is mounted at all
mounted=1
line=$(cat /proc/mounts | grep " /data ")
if [ -z "$line" ]; then
echo "data partition is not mounted!"
echo failed-to-mount > $sf
mounted=0
fi

mkdir /run/lock
count=$(fw_printenv "$uvar" 2>/dev/null)

if [ "${count}" = "$uvar=2" ]; then
echo "data partition failed!"
echo failed > $sf
fi

# If the data partition became read-only more then once, it is really broken
if [ $mounted -eq 0 ] || [ "${count}" = "$uvar=2" ]; then

# make sure logs can be written so process don't block when trying to log.
# The device cannot boot without it, since the watchdog will reset it.
mount -t tmpfs shmfs -o size=12m /data/log/

# Just leave the data partion read-only, broken device...
exit
fi

# If data became ro once, check if this happens again
if [ "$count" = "$uvar=1" ]; then
echo "data partition failed once!"
echo failed-once >  $sf

{
sleep 86400
echo "data partition failed once, but seems recovered"
fw_setenv "$uvar" 0
echo recovered > $sf
} &

else
if [ "$count" = "$uvar=0" ]; then
echo "data partition is recovered"
echo recovered > $sf
else
echo "data partition is fine"
echo fine > $sf
fi
fi

{
if is-ro-partition /data; then
# if data became ro again, just mark it as failed and reboot to report it
if [ "$count" = "$uvar=1" ]; then
fw_setenv "$uvar" 2
else
# first time, lets see if rebooting helps
fw_setenv "$uvar" 1
fi

reboot
fi
} &

This startup script is another option to force logging to tmpfs, but I believe that log2ram will be the better option, since it will provide integrated mechanism to sync with the disk periodically.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

kivanov avatar image
kivanov answered ·

Here is a step-by-step procedure of my implementation of log2ram for venusos. This can be easily automated with a script if someone is willing to do it. It can also be implemented it as a package (similar to Kevin Windrem packages).
I will paste file content in-line and attach a zip with them to this post.

1) Downloading:

mkdir -p /data/my_files
cd /data/my_files
curl -L https://github.com/azlux/log2ram/archive/master.tar.gz | tar zxf -
cd log2ram-master

2) Install/copy and create the needed files to a systems dir in /data:

First creating log2ram system directory in /data so we can have all needed files to be easily copied to the needed places even after upgrade.

mkdir -p /data/log2ram_system
cp log2ram /data/log2ram_system/log2ram
chmod 755 /data/log2ram_system/log2ram
cp log2ram.conf /data/log2ram_system/log2ram.conf
chmod 644 /data/log2ram_system/log2ram.conf
cp log2ram.logrotate /data/log2ram_system/log2ram.logrotate
chmod 644 /data/log2ram_system/log2ram.logrotate

2.1) Init script

Creating the init script (SysV Init). This will provide proper start and sync on reboot and gracefull shutdown as well.
Content of /data/log2ram_system/log2ram-init.sh

#!/bin/sh
#
# log2ram SysV Init script
# Developed by kivanov for VenusOS
#
### BEGIN INIT INFO
# Provides:          log2ram
# Required-Start:       
# Required-Stop:        
# Default-Start:     S
# Default-Stop:      0 6
# Short-Description: Provides ramdrive for system logging
### END INIT INFO

# Init start
START=06
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
LOG2RAM_SCRIPT=/usr/local/bin/log2ram
CONFIG=/etc/log2ram.conf

# Check if all is fine with the needed files
[ -f $CONFIG ] || exit 1
[ -x $LOG2RAM_SCRIPT ] || exit 1

# Source function library.
if [ -f /etc/init.d/functions ]; then
      . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ]; then
      . /etc/rc.d/init.d/functions
fi


case "$1" in
  start)
    echo -n "Starting log2ram: "
    #touch /data/log2ram.started.$(date +"%Y-%m-%d_%T")
    ${LOG2RAM_SCRIPT} start
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        echo "OK"
    else
        echo "FAIL"
    fi
    ;;
  stop)
    echo -n "Stopping log2ram: "
    #touch /data/log2ram.stopped.$(date +"%Y-%m-%d_%T")
    ${LOG2RAM_SCRIPT} stop
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        echo "OK"
    else
        echo "FAIL"
    fi
    ;;
  sync)
    echo -n "This operation is generally provided by cron."
    while true; do
    read -p "Continue (y/n)?" choice
    case ${choice} in
        [Yy]* ) break;;
        [Nn]* ) exit 1;;
        * ) echo "Please answer yes or no.";;
    esac
done

    echo -n "Force log2ram write to disk on-the-fly from the cli: "
    #touch /data/log2ram.synched.$(date +"%Y-%m-%d_%T")
    ${LOG2RAM_SCRIPT} write
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        echo "OK"
    else
        echo "FAIL"
    fi
    ;;
  status)
    cat /proc/mounts | grep -w log2ram > /dev/null && { echo -ne "log2ram is running\n"; };
    cat /proc/mounts | grep -w log2ram > /dev/null || { echo -ne "log2ram is NOT running\n"; };
    exit $?
    ;;
  restart)
    $0 stop && sleep 1 && $0 start
    ;;
  *)
    echo "Usage: /etc/init.d/$(basename $0) {start|stop|sync|status|restart}"
    exit 1
esac

exit 0

next:

chmod 755 /data/log2ram_system/log2ram-init.sh

2.2) Cron file

Creaging cron script which can be copied to cron daily/weekly/ etc folder or set manually in cron to be execuded for particular periods of time in order to sync the log ramdisk to the flash memory.

Bellow is the content of /data/log2ram_system/log2ram-cron.sh:

#!/bin/sh
#
# cron.daily/log2ram -- daily write/sync ramlog to disk
#
#

LOG2RAM_SCRIPT=/usr/local/bin/log2ram
CONFIG=/etc/log2ram.conf

# Check if all is fine with the needed files
[ -f $CONFIG ] || exit 1
[ -x $LOG2RAM_SCRIPT ] || exit 1
cat /proc/mounts | grep -w log2ram > /dev/null || { echo -ne "log2ram is NOT running\n"; exit 1; }

exec ${LOG2RAM_SCRIPT} write

next:

chmod 755 /data/log2ram_system/log2ram-cron.sh

Finally fix ownership (if some of the copied filed are not owned by root)

chown -R root:root /data/log2ram_system

3) Configure log2ram setting:

Set the bellow parameters in /data/log2ram_system/log2ram.conf (editing the file using nano for example). The edited file is in the attached zip as well.

SIZE=100M
USE_RSYNC=true
MAIL=false
PATH_DISK="/data/log"
ZL2R=false

It Highly recommended to install rsync if it has not been installed yet:

opkg update
opkg install rsync

IMPORTANT: Install findmnt - needed to log2ram core tool:

opkg install util-linux-findmnt

if not installed, log2ram should be modified to use other mount point cheching mechanism (the easiest would be a grep over /proc/mounts).

4) Installing to system:

cd /data/log2ram_system
mkdir -p /usr/local/bin/
cp log2ram /usr/local/bin/log2ram
cp log2ram.conf /etc/log2ram.conf
cp log2ram-init.sh /etc/init.d/log2ram-init.sh
# we create daily cron for syncing
cp log2ram-cron.sh /etc/cron.daily/log2ram-cron.sh
[ -d /etc/logrotate.d ] && cp /data/log2ram_system/log2ram.logrotate /etc/logrotate.d/log2ram

Remove a previous log2ram version (if present for some reason):

rm -rf /var/log.hdd
rm -rf /data/log.hdd
# Make sure we start clean
rm -rf /var/hdd.log
rm -rf /data/hdd.log

5) Setting up init and check

update-rc.d log2ram-init.sh start 06 S . stop 80 0 6 . 

If we want to disable:

update-rc.d -f log2ram-init.sh remove

Checking correct symlinks are installed:

ls -alh /etc/rcS.d/    # symlink should be present
ls -alh /etc/rc3.d/    # symlink should NOT be present
ls -alh /etc/rc6.d/    # symlink should be present

6) reboot

sync
reboot

After reboot - there should be log2ram tmpfs filesystem already mounted and log hdd:

cat /proc/mounts
df -h


ZIP archive of files: log2ram_venusos_files.zip


2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jeroen avatar image jeroen ♦ commented ·

You shouldn't do it like this, if you want to log to RAM just mount /var/log as such, that is all that is needed.

0 Likes 0 ·
kivanov avatar image kivanov jeroen ♦ commented ·

Yes, I agree and this is the easiest and most logical solution, but this way non of the logs will survive reboot/shutdown. In some cases (depending on what is placed to be moved to ram) there may be some data exposed to vrm (I suppose there is such) or other platform, which will be lost if reboot/shutdown.

0 Likes 0 ·