Did you set the date and time in the remote console? Have you set the location on the VRM portal? These settings do not change, but if not set will default to UTC.
It seems ntp on venus is handled by connmand, I found there is no way to manually trigger it, and it only seems to try to sync every 18 hours or so.
Maybe your internet connection is unstable, and when it is offline when that sync should occur, you lose time? Because I also notice my RPi 4 adjusts about 1 second every sync, which can be seen in the log:
root@raspberrypi4:~# grep ntp /var/log/messages
Mar 1 20:24:08 raspberrypi4 daemon.info connmand[817]: ntp: time slew -1.019392 s
I also notice there is a “ntp” package available in opkg, so maybe that’d be the way to go f you’re so inclined.