Home Uniquely NZ Travel Howto Articles Pauline Small Firms Settings Search

Home

Uniquely NZ

Travel

Howto

Pauline

Small Firms

Advanced Search Search of pcurtis.com by Freefind

Upgrading Corinna's Electrical System With Remote Access and Management via a Raspberry Pi

The initial document on the Upgrade to Corinna's Electrical System has got far too long so I have decided to cover this addition of a Gateway to allow remote monitoring using a Raspberry Pi on new pages. There are currently two new parts: the straightforward addition of The Raspberry Pi using the Venus OS and The Extension to Venus OS Large with Node-RED to allow the addition of a dedicated Dashboard and additional smart management.

Introduction

The Victron Smart devices I am using on Corinna (and some of their earlier devices) offer multiple ways to control, configure and view their data. Initially I just used the Victron SmartConnect App on an Android Phone and the in-built Bluetooth in my Smart devices - that is very effective but the combination has some shortfalls, namely the bluetooth is short range (<10m), only one device can be viewed at a time and the App can not run in the background. Victron however offer several more ways to connect their devices for different purposes which I am going to explore here. The earlier devices and most of the new devices can connect directly via Victron's VE Direct which is basically a serial connection using a small standard connector ( JST PH 2.0 4pin terminal connectors) on the devices and connections, generally in a star configuration to a 'host'. I am only going to use such connections in this article but for completeness I should mention some other alternatives.

Some Victron devices also have VE.Can Bus connections which are mainly used for interchanging real time information between devices to keep them synchronised - a battery may need to control the charge rate from a charger and several inverters may need to be synchronised to give a 3 phase output. They are generally daisy-chained together. To quote Wikipedia: "A Controller Area Network (CAN bus) is a robust vehicle bus standard designed to allow micro controllers and devices to communicate with each other's applications without a host computer. It is a message-based protocol, designed originally for multiplex electrical wiring within automobiles to save on copper, but it can also be used in many other contexts. For each device, the data in a frame is transmitted sequentially but in such a way that if more than one device transmits at the same time, the highest priority device can continue while the others back off." NMEA-2000 which is used for integrating marine instruments and other marine systems in a one example of CAN bus. There is yet another daisy-chained connection used by Victron Devices called the VE.Bus. The Smart devices that I need to connect all have VE.Direct and Bluetooth connectivity and that is all this article will consider.

Victron make a series of what they call GX devices which in their words "Operate as a communication centre of an installation. All the other system-components - such as inverter/chargers, solar chargers, and batteries - are connected to it. Monitoring can be carried out locally and remotely - via the free-to-use Victron Remote Management portal (VRM). The GX-device can also provides Remote firmware updates and allows inverter/charger settings to be changed remotely."  I have not yet found what the GX stands for but Global or Gateway Xconnection seems to sum it up! There are at least 6 Victron GX devices with varying processing power, number of connections, types of connections, sensors monitored and local display options. They are all Linux based and all run the same customised Linux distribution called Venus. It all seems very well thought out and compatible and one of the most important features is that they all act as a Gateway to the Victron VRM portal which you can log into via any browser or their apps anywhere in the world and see all the current and historic data. GX devices can store data locally for long periods if their internet connect is broken and then upload when connectivity is restored. Once you have a GX device live you can use the existing SmartConnect Apps through the VRM and GX device to monitor and configure just the same as locally via bluetooth or a VE.Direct cable. The VRM portal gives very powerful display options and allows you to monitor for any errors and set alert levels on key parameters - you can then have emails sent to warn you of problems. The VRM portal is a free service for registered Victron users.

The GX devices are however not cheap, the lowest prices are for the Venus GX and its recent replacement the Cerbo GX are ~£287 with an extra £215 if you have a local display on the Cerbo and cables are an extra ~£15 per device. However Victron has committed to make certain of its projects open source including the Venus OS, the Linux distribution used to run all the GX devices. In fact Victron note that while being a very successful project in its standard configuration, the Venus OS, and compatible hardware platforms can also be the perfect platform for Open Source contributions. They give examples such as adding drivers for extra products and, if interesting enough, they offer to add transmission and readout of that data to the VRM Portal. In particular, adding readouts from generator, start/stop, and temperature sensors would be very interesting to them as would be add logic on a product, for example to control the relay, or do other things.

Most important for me is that they offer full information about Venus OS; and compatible hardware including how to install on a Raspberry Pi at https://github.com/victronenergy/venus/wiki/raspberrypi-install-venus-image which contains several other links worth following. The story behind this can be found in their blog at https://www.victronenergy.com/blog/2017/09/06/raspberry-pi-running-victrons-venus-firmware/ . To cut a long story short I found this to be sufficiently encouraging that within a couple of days of finding the paper above I had ordered various bits including a Raspberry Pi 3B+ , taken delivery and amazingly commissioned the complete system running it locally and via the portal. I will however go into more detail for those who wish to follow in my footsteps but before doing so consider the complexity of your system - the Victron Cerbo GX can handle dozens of devices with many different types of connection without converters, whilst the RPI 3B+ just has 4 USB ports and even the VE.Direct inputs need converters rather than cables. It is a very cost effective and fun solution for a small system like mine but less so for a complex system.

The Full(er) Raspberry Pi Setting-up Story

The Raspberry Pi is a small single board computer designed to have many accessible outputs and inputs to allow easy interfacing to hardware and as teaching tool for coding. It has proved very popular and over 40 millions have been sold making it the best-selling British computer. Most Pis are made in Wales. There have been a number of versions and the hardware now includes good Wifi and Bluetooth as well as USB ports, an ethernet input and an HDMI output for a monitor. The Raspberry Pi uses a SoC (System on a Chip) as an almost fully contained microcomputer and the 3B+ version has a custom Broadcom 1.4 GHz 64-bit quad core ARM Cortex-A53 processor with 1 GB RAM. This SoC does not contain any kind of data storage, which is common for a microprocessor SoC, but there is a socket for a microSD on the board.

The latest version of the Raspberry Pi is version 4 but the Venus OS, at the time I started, worked best on version 3B+ . The Raspberry Pi 3B+ was still in production but on long delivery in many places including Amazon. So I was precipitated into ordering before I had done as much homework as usual when I found The Pihut had the Pi 3B+ in stock at the standard £34. Rather than mess around I also ordered the official 2.5 Amp mains power supply for development and a rather nice and very flexible official case for £6 to protect it as well as an extra 16 Gbyte Sandisk Edge microSD card (class 10) at £6. It all came by courier in under 24 hours leaving me time to get it working before ordering the expensive VE.Direct to USB cables to be delivered. They came from OnBoardEnergy by courier the following day from enabling me to have the end to end system running in well under 48 hours from locating a source of a Pi 3B+ which must be a record!

I am always very suspicious when people say anything involving computers is quick and easy but setting up the Pi was. I followed the procedures at https://https://github.com/victronenergy/venus/wiki/raspberrypi-install-venus-image fairly closely and I recommend you do the same so I will not go into a lot of detail. I downloaded the latest Venus OS which at the time was 2.73 and unzipped it (double click then drag the image out to the same download folder or anywhere you can remember). I used BalenaEtcher https://www.balena.io/etcher/ to burn it onto the microSD card rather than my standard Mint software on my home machines. BalenaEtcher is one of those pieces of software which just seems to work, including verifying the resulting image on the card - it comes as an Appimage so it is self-contained and just needs a double click to run under Linux or most operating systems. Once flashed the MicroSD card slips into a socket under the board which was still accessible after the case is assembled. I followed the recommended route and started with a network cable plugged into board and connected power and opened my browser and went to http://venus.local and it was all running just like that. It was about 30 mins from unpacking and examining the bits to having everything running via a network cable!

The only problem was that the WiFi was not working. It quickly became obvious the WiFi was not enabled on the Pi 3B+ board by default so I added a HDMI cable to a monitor and plugged in a keyboard/pad giving me command line access to see what was going on. I tried the connmanctl command line utility as it was already installed. See https://wiki.archlinux.org/title/ConnMan#Wi-Fi . I confirmed that wifi was disabled by running connmanctl in a terminal:

connmanctl technologies

and checked the line that says Powered: True/False to confirm it was disabled.

To power the wifi on you run 

connmanctl enable wifi

I checked again giving:

$ connmanctl technologies
...
...
/net/connman/technology/wifi
Name = WiFi
Type = wifi
Powered = True
Connected = True
Tethering = False
$

After that I could return to browse venus.local and used the Settings menu to find and connect to my WiFi. I initially used the house WiFi for testing. While I was using a terminal I also set up SSH so I had Remote terminal access which reuires a root password which I also set. I will cover that in the next section. Once it was all checked I unplugged the monitor and keyboard and went to the Corinna where I changed the Wifi to my old tethered phone which provides my internet access on the boat. This was about two hours after DHL rang the doorbell with my box from the Pihut.

This was the point I ordered my VE.Direct to USB cable. They are expensive at £26.99 and instructions exist on the Internet on how to make your own but the proper ones have an isolated connection which none of the home made ones have. Victron insist that they need to be isolated to avoid interference and earth loops and to protect one's expensive equipment - it has been reported that it can be zapped by lightening strikes which is a risk on sailing boats but less so on canals. I spent a while exploring the use of the VRM portal and various ways of connecting, monitoring and configuring. I was surprised how easy and flexible it all was. The only thing to note is that the the VRM portal is very powerful and flexible for monitoring and setting alerts etc but the SmartConnect App is the best way to configure ones devices, either locally using Bluetooth or remotely using the VRM tab in the App.

Setting Up SSH Access (Useful to explore within the system)

If you have to ask what SSH is you should ignore this section at this time. If you know what it is you probably realise that it is nice to have Shell (Console or Terminal) Access over your local network for simple jobs especially on a machine with no keyboard or display (headless in the jargon) and I wanted to be able to explore the system. The SS stands for Secure Shell as it is an encrypted connection, the H goes back to an earlier less politically correct definition. It also enable secure access via an FTP programme.

The procedures are different to what I am used to in Debian based distros such as Ubuntu or Mint which is what I normally use - the Venus OS does not seem to use sudo or su for superuser access but uses an actual login as root which can be extremely dangerous so take care. This is sort of covered in https://https://github.com/victronenergy/venus/wiki/raspberrypi-install-venus-image which I followed at the time.

You first need to set the root password and to do so you must first set your access level to Superuser. A Superuser is Godlike and can do anything including trashing the whole system but I suppose that does not matter if it only takes a few minutes to flash a new SSD.

Go to Settings -> General and set the Access Level to User and Installer. In my case it was already User and Installer, if not the password is, I understand, ZZZ which is at least less common than many defaults.

• Now Highlight Access Level and Press and hold the right button of the center pad until you see the Access Level change to Superuser or
• if working from the Remote Console (as I was), you need to hold the right key on your keyboard. Pressing and holding the right button with your mouse won't work.
• and when using a touch-screen drag the menu down and hold it down for five seconds or use Remote Console to switch to Superuser.

Now you have access to the Superuser features and can create a root password

Go to Settings -> General -> Set root password and create a root password.

Note that, the root password will be reset by a firmware update. The reason is that the passwd file is on the rootfs, which is fully replaced by an update.

Before you can login via ssh, you must enable Remote Support (Settings -> General). Besides enabling the reverse tunnel it also enables the sshd daemon - all very impressive sounding but you do not need to understand the details.

To login, enter the ip address of the GX device in a ssh client. Most users will simply do this from the command line in a Linux system by:

ssh root@venus.local

or use a ssh client app if you are using android or windoz. The first time it will require extra conformation to add you to a list of known hosts as below - ignore the gobbledygook and just answer yes (not Y).

:~$ ssh root@venus.local
The authenticity of host 'venus.local (192.168.1.92)' can't be established.
ECDSA key fingerprint is SHA256:rSpOAXHFgBKNBzuhB0SHxxxndjdCSpyQas46e0SrVzY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'venus.local,192.168.1.92' (ECDSA) to the list of known hosts.
root@venus.local's password: ************
Last login: Tue Nov 9 10:59:22 2021
root@raspberrypi2:~#

I usually login in as a normal user rather than root in Linux desktop systems but in this case you need to to be the Superuser (ie logged in as root)

The advice is to create a root password. But use it to login only the first time, and then install a public ssh key(s). Thereafter login with the keys. Again ignore this if you do not understand public and private keys and settle for the procedure above as I am doing. Using a ssh key for authentication, instead of a root password, has the advantage that it isn't lost during a firmware update as the keys are stored on the /data partition. The Victron article has details.

SFTP: An additional advantage of enabling SSH access is that you are actually doing more than that as other access routes also become available through the SSH daemon including SFTP (Secure File Transport Protocol) which means that you can use an FTP program such as Filezilla on your machine which enables you to view a remote file system as easily as a local one, upload and download files and carry out simple actions on the files such as rename, delete and setting permissions. SCP (Secure Copy Protocol) is also available for command line transfers which can be faster but less robust. Filezilla is a easy to use but very flexible programme which is free and runs on all common operating systems.

Android SSH Apps: I also want to be able to access from my phones and pads over SSH. I already had JuicySSH on some of them and I added an extra Keyboard overlay as the standard touch keyboard is missing many keys needed for programming or terminal use. In particular the standard editor available in the Venus OS seems to be vi which makes use of Escape to change modes. I am using the popular Android 'Hackers Keyboard' App which has everything I need. I use the AndFTP App to provide FTP access.

More about The Venus OS (for power users and the curious!)

The following is mostly notes from my early reading and has been augmented when I have done some more exploring:

The Venus OS is a customized Linux. It uses a dual file system boot mechanism: two bootable partitions that swap between active and backup with each software update, plus a data partition that survives a software update. Anything extra one installs will generally need to be reinstalled after a Venus OS update. In versions higher than 2.80 the root filesystem is read only

The underlying Linux distribution is openembedded-core (currently Yocto Dunfell in v2.90). As the name implies it is optimised for embedded systems, normally headless with no keyboard or monitor, with minimal utilities and users like me who have used desktop distributions such as Debian and its derivatives Ubuntu and Mint will find it very different.

It uses BusyBox so some common commands may have reduced functionality.

The apt package manager is not installed. One needs to use opkg to install things from the Victron repository which is comparatively restricted in the packages it provides. wget is however available in Busybox for downloads.

Adding extra functionality to Venus should be possible as long as there are no conflicts between the hardware used by Venus and the apps you install and provided you have enough processing power.

You may need to tweak partition sizes in order to fit any added functionality.

The microSD card is partitioned during the first install and the two partitions containing the alternate filesystems are only formatted with sufficient space for the read only file systems. They can be expanded to fill the space but there is a reason as the unused space and read only file system has been done to extend the life of the microSD cards.

Raspberry Pi versions 2B and 3B are currently fully supported. Raspberry Pi 4 support was first implemented in version 2.90 and is solid.

More on opkg: Most Linux distributions (or mobile device operating systems like say Android or iOS) allow the functionality of the system can be upgraded rather significantly by downloading and installing pre-made packages from package repositories (local or on the Internet). The opkg utility is a lightweight package manager is designed to add software to stock firmware of embedded devices and serves as a full package manager for the root file system, including kernel modules and drivers. opkg attempts to resolve dependencies with packages in the repositories - if this fails, it will report an error and abort the installation of that package. Missing dependencies with third-party packages are probably available from the source of the package. See https://openwrt.org/docs/guide-user/additional-software/opkg for more information. I understand the Victron Venus Repository is restricted in the applications it offers. opkg list-installed lists all installed packages

More on Busybox: BusyBox is a software suite that provides many Unix utilities in a single executable file. It runs in a variety of environments such as Linux and Android although many of the tools it provides are designed to work with interfaces provided by the Linux kernel. It was specifically created for embedded operating systems with very limited resources. The authors dubbed it "The Swiss Army knife of Embedded Linux", as the single executable replaces basic functions of more than 300 common commands. In fact, looking at the latest version as used in the Venus OS, there are over 370 commands crammed into 2.1 Mbytes. The command busybox --listall provides a useful list of the commands currently implemented in a form you can pipe into grep.

BusyBox initially aimed to put a complete bootable system on a single floppy disk that would serve both as a rescue disk and as an installer for the Debian distribution. Since that time, it has been extended to become the de facto standard core user space toolset for embedded Linux devices and Linux distribution installers so it is no surprise to find it as part of the Venus OS.

More on the Linux Kernel. I have been logging into the Venus OS using SSH from my pad using the JuicySSH and Hackers Keyboard Apps. The Hackers KeyBoard App provides a full set of keys including Escape and cursor up and down. and this is what I found for the Kernel Version and folder structure

Last login: Wed Nov 17 13:15:20 2021 from 192.168.43.98
root@raspberrypi2:~# uname -a
Linux raspberrypi2 4.19.81-v7 #1 SMP Mon Aug 30 18:57:52 UTC 2021 armv7l GNU/Linux

The choice of kernel is interesting, it is relatively old and very specific in the v7, the only other OS which I have found which used it was Tiny Core Linux v11 . However this may shortly be outdated as there is an upgrade in the pipeline to bump the kernel to 5.10.81 for some RPI versions.

More on the partitioning and folder structure: The remainder of the output serves as an example of the use of ssh access (and busybox) and shows the partition and file structure implemented in the Venus OS v 2.73. This was done for my own interest and is not necessary to follow or understand any of this for normal use of the Venus OS and setting up a Raspberry Pi system.

root@raspberrypi2:~# ls / -l
drwxr-xr-x 2 root root 1024 Jan 1 1970 Settings
drwxr-xr-x 2 root root 3072 Aug 30 19:04 bin
drwxr-xr-x 2 root root 1024 Aug 30 19:00 boot
drwxr-xr-x 13 root root 4096 Nov 9 11:42 data
drwxr-xr-x 17 root root 3500 Nov 9 10:59 dev
drwxr-xr-x 50 root root 4096 Nov 9 13:31 etc
drwxr-xr-x 3 root root 1024 Aug 30 19:03 home
drwxr-xr-x 7 root root 4096 Aug 30 19:04 lib
drwx------ 2 root root 12288 Aug 30 19:04 lost+found
lrwxrwxrwx 1 root root 10 Aug 30 18:57 media -> /run/media
drwxr-xr-x 2 root root 1024 Aug 30 18:57 mnt
drwxr-xr-x 3 root root 1024 Aug 30 19:02 opt
dr-xr-xr-x 251 root root 0 Jan 1 1970 proc
drwxr-xr-x 14 root root 820 Nov 17 13:20 run
drwxr-xr-x 2 root root 4096 Aug 30 19:04 sbin
drwxr-xr-x 3 root root 1024 Nov 9 10:59 service
dr-xr-xr-x 12 root root 0 Jan 1 1970 sys
lrwxrwxrwx 1 root root 8 Jan 1 1970 tmp -> /var/tmp
drwxr-xr-x 3 root root 16384 Jan 1 1970 u-boot
drwxr-xr-x 11 root root 1024 Aug 30 19:03 usr
drwxr-xr-x 9 root root 1024 Jan 1 1970 var

root@raspberrypi2:~# df -h -T
Filesystem Type Size Used Available Use% Mounted on
/dev/root ext4 4.8G 288.2M 4.3G 6% /
devtmpfs devtmpfs 458.5M 4.0K 458.5M 0% /dev
tmpfs tmpfs 463.0M 252.0K 462.8M 0% /run
tmpfs tmpfs 463.0M 428.0K 462.6M 0% /var/volatile
/dev/mmcblk0p1 vfat 43.7M 23.7M 20.0M 54% /u-boot
/dev/mmcblk0p4 ext4 4.9G 22.2M 4.6G 0% /data

root@raspberrypi2:~# cat /etc/fstab
/dev/root / auto defaults 1 1
proc /proc proc defaults 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0
tmpfs /var/volatile tmpfs defaults 0 0
/dev/mmcblk0p1 /u-boot vfat defaults 0 0
/dev/mmcblk0p4 /data ext4 noatime 0 2

root@raspberrypi2:~# fdisk -l
Disk /dev/mmcblk0: 15 GB, 15931539456 bytes, 31116288 sectors
486192 cylinders, 4 heads, 16 sectors/track
Units: sectors of 1 * 512 = 512 bytes
Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/mmcblk0p1 * 64,0,1 764,2,20 8192 97875 89684 43.7M c Win95 FAT32 (LBA)
/dev/mmcblk0p2 768,0,1 1023,63,32 98304 10268375 10170072 4965M 83 Linux
/dev/mmcblk0p3 1023,63,32 1023,63,32 10268672 20537343 10268672 5014M 83 Linux
/dev/mmcblk0p4 1023,63,32 1023,63,32 20537344 31115263 10577920 5165M 83 Linux

As you can see, the tools in BusyBox are not ideal - utilities I would normally use like blkid and lsblk are missing or lack options like df. fdisk shows the permanent partitioning in a less than friendly way and I was tempted to edit out most of the columns as below!

Device         Boot Size  Id Type
/dev/mmcblk0p1 *    43.7M c  Win95 FAT32 (LBA)
/dev/mmcblk0p2      4965M 83 Linux
/dev/mmcblk0p3      5014M 83 Linux
/dev/mmcblk0p4      5165M 83 Linux

So it seems we have a basic LBA partitioning scheme with 4 partitions partition holding a small initial FAT32 boot partition and the remainder of the micro SD is divided into 3 approximately equal Linux ext4 partitions, one holding the /data partition (/dev/mmcblk0p4) which survives updates and two partitions used in turn to give a backup during and after updating which is a very robust way to proceed. I often do a fresh install into an extra partition when doing major version upgrades ~ every two years on my home systems. There seems to be extensive use of temporary file systems which makes the outputs above difficult to interpret.

What else can we learn: The df output shows disk is much bigger than we need and even a 4 Gbyte SD would be more than adequate.

If you want to learn more about ways to examine partitioning on Linux systems in general I found the article at https://www.binarytides.com/linux-command-check-disk-partitions/ clear and useful, the comments contain some additional suggestions. An alternative is to remove the card and plug it into a another machine with partitioning software such as gparted.

Safe ways to turn off the system.

When you want to turn off your Raspberry Pi, simply pulling the power cord is effective but not ideal. This is because the Raspberry Pi may be writing data to the SD card, in which case simply powering down may result in data loss or, even worse, a corrupted SD card. I should stress this is extremely rare but even so some users keep a spare SD card flashed with a Venus OS.

The problem occurs because the Venus OS does not offer a shutdown option for the Pi on their own units. It does offer a reboot option and some users suggest using that and switching off immediately after the power light flickers off in the belief that is a safe time as no writing is taking place.

I favour using ssh which enables me to use a proper shutdown command namely:

shutdown -h now

the now can be replaced by a time or delay if required - the following command will shut down the Raspberry Pi in 20 minutes time:

shutdown -h 20

or halt at 1730:

shutdown -h 17:30

The -h (for halt) can be replaced by -r (for restart) if required. This can be useful if there has been an anomaly and avoids pulling the plug. Option -c (for cancel) will cancel any 'timers' and operations.

Introduction to the Venus OS Large and Node-RED

There is an extentension to the Venus OS called Venus OS Large which adds Node-RED and an optional Signal K server. To quote Victron Energy:

Node-RED is a tool for connecting hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows. With it, one can for example program a functionality such as activating a relay based on a temperature measurement. Or make far more complex algorithms, tying relays, measurements, or other data available from Venus OS or elsewhere together. All without having to write real source code, as Node-RED calls Low-code programming for event-driven applications.

Node-RED features a fully customisable dashboard, viewable in a web browser - both locally but also remotely, via the VRM Servers.

The Signal K server is aimed for yachts, and multiplexes data from NMEA0183, NMEA 2000, Signal K and other sensor inputs.

Venus OS Large runs with Node-Red on my Raspberry Pi 3B +. It is ideal for those with some programming knowledge who wish to tinker or customise their systems and have a sufficiently powerful processor in their Victron controller or Raspberry Pi. The idea of an improved dashboard and extra control from Node-Red was very attractive to me and I tried it out as soon as I had got the basic system up and running.

To cut the story short for this introduction: The Raspberry Pi 3B+ has adequate power for what I have tried so far and a new page on the use of the Venus OS Large which includes Node-RED describes my journey of exploration. In general the use and testing of the Node-RED extension has not affected the basic operation of the Venus OS, data gathering of the Pi and transfer to the Victron Portal and once it was installed I have been able to mostly work remotely through the VRM Portal rather than on a very cold Corinna over the winter.

Conclusions - How is it all working in practice?

It was still relatively early days when I initially wrote this but even then I could tell it is all working on the Raspberry Pi as it would with a Victron Venus GX (which has no display), in fact it is better in many ways at a much reduced cost. The major cost has been the VE.Direct to USB isolated cables, which are required, and are more expensive (£27) than the VE.Direct cables to Victron's own Gateways. The Raspberry Pi 'gateway' also has Bluetooth so it can be accessed directly by the SmartConnect App for setting up, a feature which most of the Victron GX devices lack.

So we now have several routes to control and monitor the system and can do it locally and remotely. As far as I can tell there is no difference in functionality between using the SmartConnect App via Bluetooth or the VRM portal, except perhaps in smoothness as there are always some lags through the internet. I find the WiFi gives better coverage through the boat than Bluetooth despite being initially provided by an elderly tethered Samsung Galaxy S3 mini (3g) which lives on a bracket in a window. So we have everything we had before with more local range even before we started to use the portal for world wide coverage. I am yet to check the data use fully but I do not need to worry too much in the UK and EU with a GiffGaff Goodybag providing 15 Gbytes of 4g data a month for £10. Initial impression is that it may be ~2Gbyte/month.

I have not finished exploring the Portal (and it is frequently getting updated and improved) but I find it very easy to watch the Solar and Battery status real time, daily and historic data from the comfort of the house. The views can be fully customised and the configuration is saved on the Portal. There is an App for Android devices as well as access through a browser. It does not seem to be possible to simply control my devices to say switch the inverter to and from Eco mode or configure any parameters directly from the Portal but that can all be done by the SmartConnect App and both can be open at the same time. I can see the logic for separating monitoring from configuration but a few simple switches would be convenient, in particular inverter mode.

I have checked that the data continues to be stored in the Raspberry Pi if the internet connection is broken and then uploads to the portal as soon as it is reconnected. However, if the Pi is rebooted local data not already uploaded does seem to be lost. Even if the internet connection is temporarily broken one can still access the basic dashboard via any machine on the local wifi network (or via a network cable for ultimate reliability onboard) at venus.local.

Overall I am delighted with progress so far and I am gradually building up a comprehensive set of data which is stored on the VRM Portal for at least 6 months and can also be downloaded for additional processing in addition to that gathered earlier via Bluetooth.

The use of The Venus OS Large which includes Node-RED covered in the next page has added an extra dimension and is showing great promise. Node-RED allows much greater flexibility including 'intelligent' control of the devices via your own configurable dashboards. Node-RED can be accessed and programmed via the VRM portal and I have done all the Node-RED programming and 99% of testing remotely. I implemented remote control of the inverter including an option to automatically return the Inverter to Eco (Search) mode from Continuously On after an hour to save power if one forgets. I have also added an extra 'Tab' to the dashboard to monitor the Pi CPU loads and processor temperature and shut down and restart the Pi and Node-RED itself. And this is early days!

Appendix A - Simple cloning of the microSD card as a backup

The following method provides as perfect a clone as one can get between identical microSD cards and leaves you with a compressed backup on a hard drive. This is not done on the Pi - the card is removed and cloned on a separate Linux based computer. The only requirement is a matching microSD card socket with or without an adapter. It uses a command line utility dd. See https://opensource.com/article/18/7/how-use-dd-linux

dd is a simple and powerful image-copying tool that's been around since the early days of unix and nothing's come along that does the job of cloning better. Using dd, you can make perfect byte-for-byte images of almost anything digital. But there's some truth to that old Unix joke: "dd stands for disk destroyer." If you type even one wrong character in a dd command, there is a possibility you can instantly and permanently wipe out an entire drive of valuable data including your system drive. Check and check again before writing to a disk with dd.

The magic incantations below use dd to read an entire SD card bit by bit and compress it and save in an archive as backup by piping the output of dd into gzip. You need to be very sure where the microSD drive is mounted hence the initial fdisk -l and note you need to enter the drive not the individual partitions. In the case of a 16 Gbyte card with a Victron Venus OS the final archive was only 136 Mbytes. The Venus OS large gives a larger archive but still only 530 Mbytes including all the Node-RED flows.

The archive is unzipped and written byte by byte back to preferably an identical card. If the card is bigger you will have unallocated space after the partitions, if it is smaller it have at least one unreadable partition. The following is the important bits of a clone to a larger card of the same type. You will see all the sizes of the partitions are exactly the same and much of the extra space in the root partitions are unallocated. Once you have ejected using a system facility so it is shut down properly you can increase the partition sizes using gparted if you need to use the extra space. Always make sure you are writing to the correct devices as dd destroys disks.

 

Update: I have found it easier to flash the archive back onto another microSD card using BalenaEtcher which is very fast, has a lot of error checking and, most import, verifies the result. It also runs under Linux, Windoz and Mac and can be run from an appimage under Linux without even installing and is opensource. What more can one ask?

There is futher on the subject of cloning in a the next section which looks more deeply at the partitioning before discussing how to shrink the clone if part of the card is unallocated, ie not formated and used and reduce the size of card needed.

Appendix B - Venus OS Partitioning and File Systems

The details of the drive Partitioning and File Systems have evolved over time but some features have remained constant. Victron have always tried to make the updating of file systems as safe and robust as possible so updates have always been based on the information being completely available from whatever source before the existing system was lost so, in particular, an update could never fail because a download from the internet was interrupted. This includes Venus OS Large. The drive has two identical partitions for the OS files so the previous working system can be maintained until the update is completed and one can swap back to the old system if problems arose. This is very sensible and the way I already work for major upgraded to my desktop Linux systems, in fact in some cases I have three partitions available for the root file system as well as a common system for users and and two additional partitions for unencrypted and encrypted data shared between users and backed up between machines.

The Venus OS has 4 partitions, one small boot partition which also handles the switching between root file systems, two root files systems of which only one is ever used, the other is a backup of the previous install and a data partition which gives persistence between updates. The main space is basically partitioned equally between two root file systems and a data partition and symbolic links bring the parts together into a more conventional layout. For example whilst the home folder would normally be below root it is actually at /data/home/ and user root's home folder is /data/home/root so, for example, the node-Red configuration is in folder /data/home/root/.node-red. Whilst it may seem confusing the big advantage is that because /data is on a different partitionand filesystem this gives complete independence of the two and the filesystems can be swapped or updated completely independently of users and data. With careful linking there is no reason why the root file system can not be read only for even greater stability and security.

My early exploration was via the console and built in commands, I have now looked at the microSD cards in my desktop where I can use tools such as gparted to see the structure in more detail whilst I was backing them up. This has revealed some interesting information about the differences in the deployment between various versions of the Venus OS but it has now stabilised so I am only showing the latest deployment here.

Venus OS Large after updating (versions 2.80 or higher)

You will note that one of the two partitions used for root have tiny filesystems only using a fraction of the available space in the partition and occupies 91% of the alocated space. The lack of headroom is not a problem as the basic root file system is now read only so does not need to grow between version updates. Now there is obviously more than enough space on my huge 64 Gbyte cards for the data partition but it is surprising that so much of the card is unformatted. The spare space and read only system partitions are for a reason - the spare space and reduced write operations should greatly extend the life of the microSD cards. Solid state drives have a large but finite number of life cycles and complex ways the use of the available storage is used which is optimised by having a low usage.

There are times when users may need to load extra packages into the file system and scripts are available to extend the filesystems to fill the partitions and make them writeable but I have resisted that so far.

For information I have big cards as they were a special black Friday price on Amazon and cheaper than the smaller cards I had intended to buy! The downside is the extra time to clone them as backups.

I have therefore taken a cloned copy (just in case) of my current Sandisk Extreme Pro 64 Gbyte microSd card and expanded the two root partitions and sized the data partition suitably for shrinking to fit a 32 Gbyte card. It could easily be reduced again for a 16 Gbyte card if required. The changes are almost trivial to make in gparted - it is the backup/cloning which takes the time. The Sandisk Extreme Pro card is supposed to be one of the best and fastest cards when tested in the Pi and were again at a favourable price for the 64 Gbyte sizes from Amazon but eventually I may need to use smaller cards.

Changes to partitioning to a more sensible usage of the space
which can also be shrunk or expanded easily for smaller/larger cards

If one wants to reduce the above image for cloning onto a 32 Gbyte card one just saves the first 32 Gbytes with this incantation (all on one line): The 7608 is the number of 4 mbyte blocks corresponding to a standard 32 Gbytes card

sudo dd bs=4M count=7608 if=/dev/mmcblk2 | gzip > /media/DATA/shrunk_image_count_7608`date +%d%m%y`.gz

Appendix C - Updating the Venus OS - Routine and Fresh Install

The updating of Venus OS has turned out to be easier than I expected and with the latest versions you can also install the Large version directly. You open the remote console (via VRM or preferably via the local network at local.venus) and go to menu -> Settings -> Firmware . Initially I always used the Install from SD/USB -> Check for firmware from SD/USB -> Select/Install . I now use the Online Updates option where you select the feed where " Release" is much safer than the "Release Candidate" which is a beta. I only use the "release candidates" on my development system". In my case the Image is always Large as I use Node-RED. I check and update manually.

That should be the update complete barring re-entering the root password which seems to be the only parameter that does not persist through the update (on security grounds??) . The password is set in Menu -> Settings -> General. You need to set your Access Level to become a superuser first by holding down right arrow for 5 secs then you can type the password into the set root password box. Check that SSH on LAN and Remote Support are still On. Everything including WiFi seems to be persistent.

Occasionally there is an upgrade which needs a new SD card image to be used and then I use a new MicroSD card and have bought several identical ones so I could make exact clones for backup. In theory the cards only need to be the same size when cloning but in practice it has been reported that anomalies can occur due to detailed differences in the internal implementation of the cards. One can also use a program such as gparted to adjust the partitioning to use a smaller micro SD card

The following is last (largely for my own benefit) of the changes I have made which may need to be replaced on a fresh install Many (in brackets) can be made in Remote Console once you are connected. The ones I know are not persistent are in bold. Includes Node-RED for Venus OS Large versions

1. Pair Bluetooth on phones with Victron Venus/Pi device. (Must be paired in SmartConnect not in system of the device you are pairing)
2. Provide Password for Wifi (Settings -> Wifi)
   1. Find and note ip address.
3. Set access level to Superuser (Settings -> General)
4. Create a root password. Needs to be re-created after an update to enable SSH and SFTP login.
5. Enable Remote Support to enable sshd and reverse tunnel on VRM and LAN in Remote Console
6. Turn Off auto-update (Settings -> Firmware)
7. Turn on Node-RED (Venus OS Large Feature)
8. Make changes to provide persistence in Node-RED in settings.js (Survived update but possible that settings.js is changed in future updates overwriting the change I have made)
9. Add extra nodes to Node-RED palette. (Manage Palette within Node-RED Menu)
   1. node-red-dashboard
   2. node-red-contrib-victron
   3. node-red-contrib-cpu
   4. node-red-contrib-stoptimer
   5. node-red-contrib-timeswitch
   6. node-red-contrib-os
   7. node-red-contrib-simpletime
   8. node-red-contrib-ui-artless-gauge

Node-RED is covered in much more detail in a separate page The Extension to Venus OS Large with Node-RED

NOTE: The Appendix covering Partitioning on the Venus OS has moved to the The Extension to Venus OS Large with Node-RED page.

Appendix D. Powering the Raspberry Pi and power usage estimates.

A disadvantage of using a Raspberry Pi (or any other Venus controller) is that it uses some of ones valuable power itself and also needs a Wifi connection to the internet if one is going to use the Victron VRM portal for remote monitoring and/or a mobile device round the boat.

We currently use a dedicated tethered Android mobile phone for our local network and internet connection (Samsung S3 Mini for 3G or Samsung M32 for 4G) which currently runs off the 12 volt supply via a standard cigar lighter converter to 5V whilst cruising or a 240 converter whilst on the shore hookup whilst stationary. Tethering is a major power drain on a mobile phone and the S3 Mini runs noticeably warm.

The Raspberry Pi 3B+ was run over the winter from an official mains supply providing up to 2.5 Amps at 5V, the Pi 4 official supply is 3 Amps. Whilst the peak demands could be this high when there are many peripherals also connected the average which concerns us more should be much lower but is still sufficient for the Pi to run slightly warm and with the processor ~ 25 to 30 degrees above ambient with a typical CPU load of 4%.

A quick check using a current measurement using a digital meter in a 12.5 V supply line to a 12 to 5 volt power supply indicted a lower than expected minimum reading of ~ 180 ma but with many upward variations (~ 2.5 watts). The resolution on the load output of the SmartSolar MPPT 100|20 is only .1 Amp but the reading of .1 A at 26.5 V is consistent with that. The Orion Tr has a no load current of of <40ma (~ 1 watt) and efficiency of ~80% so overall this gives an a best of estimate of ~60 Wh per day for the Pi 3B+ system.

There are currently 2 relays on the Raspberry Pi system which take an additional 70ma at 5V when activated ~ 350mwatts which is ~0.5 watts when active and two levels of inefficient conversion are taken into account. With current usage of 5 hours a day this adds 2.5 Wh/day

The Samsung S3 Mini is even more of a difficulty to measure and estimate but say 40 Whr day based on ~6 hrs tethering from a 7.2 Whr battery of 3.6 V with three levels of losses from 27V.

So monitoring has an overall penalty of up to 102 Whr/day or 4 Ah/day if solely taken from solar through the 24 V Lithium battery load output, the Orion TR 24|12 to the 12 V Lead Acid Domestic battery and then further 12 to 5 volt convertion. This figure can be compaired to the cumulative load figures on my dashboard which also have loses of various forms in the domestic battery and they are in the same ballpark.

An alternative is to power via a converter from the 24V system and I have a simple converter board from Amazon but am not currently using it.

Appendix E Use of Keith Windrem's SetupHelper, GuiMods and RpiGpioSetup packages.

I have just started to explore these packages which provide significant extensions to the basic system without the use of Node-RED. They are easy to install and seem to be compatible with my usage of Node-RED. Once the SetupHelper package has been installed all the remaining installation and updating of packages is done from within the Remote Console. Kevin recoomends an "Unattended Install" from a USB stick but I find the alternative from to access via SSH as below (if you copy and paste note the long line wget -qO - https://github.com/kwindrem/SetupHelper/archive/latest.tar.gz | tar -xzf - -C /data ):

peter@gemini:~$ ssh root@venus.local
root@venus.local's password: ****
Last login: Mon Oct 31 11:38:04 2022 from 192.168.1.86
root@raspberrypi4:~# wget -qO - https://github.com/kwindrem/SetupHelper/archive/latest.tar.gz | tar -xzf - -C /data
root@raspberrypi4:~# mv /data/SetupHelper-latest /data/SetupHelper
root@raspberrypi4:~# /data/SetupHelper/setup
--- starting setup script v4.29
creating root setup options directory /data/setupOptions
creating package options directory /data/setupOptions/SetupHelper

This package provides support functions and utilities for Venus modification packages
Packages are automatically reinstalled following a Venus OS update
All actions are controlled via /Settings/PackageMonitor in the GUI
Packages may be automatically updated from GitHub or a USB stick
Previously uninstalled packages can also be downloaded an installed

Available actions:
Install and activate (i)
Reinstall (r) based on options provided at last install
Uninstall (u) and restores all files to stock
Quit (q) without further action
Display setup log (s) outputs the last 100 lines of the log

Choose an action from the list above: i
installing PackageManager service - please wait
adding SetupHelper/setup to reinstallScriptsList
creating /data/rcS.local
Restart the GUI now (y) or issue a do it manually later (n): y
restarting GUI ...
root@raspberrypi4:~# shutdown -r now

Broadcast message from root@raspberrypi4 (pts/0) (Fri Oct 28 09:40:14 2022):
The system is going down for reboot NOW!
root@raspberrypi4:~#

Once SetupHelper is installed, updates to it and other packages can be performed through the GUI using the PackageManager menus and can be made automatic. The interface through the remote console is unusual but easy to use after you get used to it.

Overall this is a very elegant piece of work which significantly enhances the basic Venus OS and is a desirable install especially if you do not want to initially use Node-RED and I will come back to use of both together after some more exploration.