Has anyone developed a data logger for Cerbo GX using node-red

[SimonGrey]

Hello world!
Has anyone managed to use the node-red file write node on the Cerbo GX?
I am trying to implement a data logger, by writing to a file on the SD card on my Cerbo, and although the write appears to be successful (no error and input of write node written to output), the file is not updated.
I have checked paths are correct by creating a file on the SD card and reading it into node-red, so the issue is with writing.
Does anyone have any ideas?

Update: On node-red forums, it seems that permission issues arise when node-red is installed as root user apparently “this will cause all sorts of problems”. Does anyone @victron have any idea?

Here is the code for reference:

[
{
“id”: “ba8aa8c078a8b619”,
“type”: “file”,
“z”: “3938045e480f9c41”,
“name”: “”,
“filename”: “/run/media/mmcblk0p1/logs/testlog.txt”,
“filenameType”: “str”,
“appendNewline”: true,
“createDir”: true,
“overwriteFile”: “false”,
“encoding”: “ascii”,
“x”: 1130,
“y”: 140,
“wires”: [
[
“3d30281f2ee4265e”
]
]
},
{
“id”: “e084a4ae48fc8987”,
“type”: “inject”,
“z”: “3938045e480f9c41”,
“name”: “”,
“props”: [
{
“p”: “payload”
},
{
“p”: “topic”,
“vt”: “str”
}
],
“repeat”: “”,
“crontab”: “”,
“once”: false,
“onceDelay”: 0.1,
“topic”: “”,
“payload”: “Hello world”,
“payloadType”: “str”,
“x”: 800,
“y”: 140,
“wires”: [
[
“ba8aa8c078a8b619”
]
]
},
{
“id”: “3d30281f2ee4265e”,
“type”: “debug”,
“z”: “3938045e480f9c41”,
“name”: “debug 5”,
“active”: true,
“tosidebar”: true,
“console”: false,
“tostatus”: false,
“complete”: “false”,
“statusVal”: “”,
“statusType”: “auto”,
“x”: 1480,
“y”: 140,
“wires”:
}
]

Thanks!

[jrhaus (JohnH)]

I’ve had this problem on RPi3 running Venus OS. It was fixed by logging in via ssh, creating an empty file then setting permissions (chmod 777 filename.txt). The default permissions only provide root access which is why node red can’t access the file.

[SimonGrey]

Thanks
I have done this, but for some reason the chmod command isn’t working. I checked by setting permissions to various things (e.g. 222 and 766) but when I check the file attributes, they have stayed the same. Here is the SSH sequence:

root@einstein:/run/media/mmcblk0p1/logs# chmod 222 testlog.txt
root@einstein:/run/media/mmcblk0p1/logs# ls -l testlog.txt
-rwxr-xr-x 1 root root 53 Feb 11 18:21 testlog.txt
root@einstein:/run/media/mmcblk0p1/logs# chmod 777 testlog.txt
root@einstein:/run/media/mmcblk0p1/logs# ls -l testlog.txt
-rwxr-xr-x 1 root root 53 Feb 11 18:21 testlog.txt
root@einstein:/run/media/mmcblk0p1/logs#

The attributes are OK if the node-red user was root which is the only one with write permissions.

I wonder if I have to go back to root@einstein to set the permissions for the mmcblk0p1 directory (the SD card).

Will keep messing about but any further ideas would be really helpful!

[SimonGrey]

I have found the below in the Victron documentation for node-red.

Using the /data/home/nodered directory allows me to write the files, but I still cannot access teh SD card!

See Venus OS Large image: Signal K and Node-RED [Victron Energy]

Q1: Write/read file gives "Error: EACCESS: permission denied’

Since Venus OS 2.90 and on wards, Node-RED no longer runs as user root, but as user nodered. This means that the Node-RED flows are only allowed to access files and directories that are owned by the nodered user. These files typically start below /data/home/nodered/. So modifying the storage location to something like /data/home/nodered/storage.txt ought to work.

[Marc_HD (Marc)]

chown nodered filename.extension
the command above in the directory of the file: filename.extension

have fun

[jrhaus (JohnH)]

The node red folder would be a better choice for user data.

Just curious, what data are you logging?

cheers,

John

[response edited: I’ve realised the Raspberry Pi mount point is ‘/u-boot’, and the Cerbo mount point is different.]

[SimonGrey]

Thanks
I am getting an ‘Operation not permitted’ response when trying to chown the file

The logging works OK if I use the nodered folder on the Cerbo. I just wanted to put it on the SD to make it portable and also to make sure I didnt run out of space!

Thanks again

[SimonGrey]

I am logging data from a domestic system with a small wind turbine, battery storage and (to be added) solar. Only logging hourly and no more than 10 variables, so not a heavy data load.

[Marc_HD (Marc)]

oops that’s odd. As root a “operation not permitted”.

Well maybe because of card failures. I killed one sd-card an a raspi with venusOS.

Unfortunately no further ideas from me.

[JFischer (Jeremy Fischer)]

Hello! That is because the SD card is formated as FAT (In my experience Linux can’t change the permisions on this format). You should format the SD as ext4, then change permisions of the directory or the file via “chmod”.

[SimonGrey]

Thanks!
I have found a different and much more simple way of achieving what I need.

That is to write the files to /data/home/nodered/ and then use Filezilla to transfer them over the LAN to my PC.

I will post a reply to my original post to give others a ‘how to’ to implement a simple data logger solution.

[SimonGrey]

OK, so after a lot of help from this community and learning on my part, I have successfully implemented a data logger on the Cerbo using node-red and can recover the data over LAN to my laptop PC.

Here are the basic steps:

1. Set up what you want to log as either flow or global variables e.g. “global.totalKwh” or whatever
2. Use a function to create a js object containing the various variables to be logged
3. Send this object to a CSV parser and onwards to the file write node
4. Use Filezilla (or similar) to retrieve the file onto your PC
5. Open with Excel or whatever and … Happy Days!

Some more detail, notes and do’s and don’ts

1. I use global variables just for ease of porting around the various flows. There might be a better or less memory-hungry way of doing this.
2. I trigger the function every hour with an inject node, but you can chose the frequency you want to log. Be careful though as a higher frequency will generate a lot of data. I am just looking at longer term power trends from a turbine so hourly is fine.
3. Within the ‘Create logfile line’ function I am generating an object to send to the CSV parser. I could have created a text line of comma-separated variables in the function, but node-red can do that automatically, so why bother. Here is my function:

var d = new Date();
payload = { “Date”:d,
“TotalKwh”:(Math.round(global.get(“lifeTotalKwh”)*10))/10,
“DayKwh”:(Math.round(global.get(“dayTotal”)*10))/10,
“SoC”:global.get(“soc”),
“DayMaxW”:global.get(“dayMaxKw”),
“LifeMaxW”:global.get(“lifeMaxKw”),
“AgaLifeKwh”:(Math.round(global.get(“lifeTotalAgaKwh”)*10))/10,
“AgaDayKwh”:(Math.round(global.get(“dayTotalAgaKwh”)*10))/10
}
msg.payload = payload;
return msg;

I have done some maths to round values before storing them. I should really do that elsewhere in the flow, but wanted to avoid cumulative rounding errors.

4. In the CSV parser I have specified the column headings, but you don’t need to do this. It just sorts the columns into whatever order you need, and I did it just to check that I could change the order. Probably leave this out.
   

   

5. In the write file node, specify the full path to /data/home/nodered//<yourlogfile.txt> and don’t check the ‘add newline’ box, otherwise you get blank lines in your data file which you later have to delete.

And that is that for creating the logfile!

I use Filezilla to access the logs. You need to know the IP address of your Cerbo on your LAN and the password to log in with username ‘root’. See the Victron documentation for root access and setting the password.

Once logged in, you can then navigate to /data/home/nodered/ and drag files up to your PC. Keep an eye on file sizes and adopt good discipline of deleting files on the Cerbo so as not to run out of space.

I hope this helps and thanks to all who have guided me to work it all out.

And, if there are better, quicker or safer ways to do this, then let me and the community know!