Using SCP/RSYNC Through Balena Tunnel into a container

Project help
,
1

Hi everyone,

I’m currently working on automating a task that involves remotely copying files from a field data-logger device and uploading them to a local machine. Unfortunately, the Balena releases for these devices are managed by another group, so I have limited control over how the system is set up. Here’s what I’ve been able to achieve so far through my process research:

  1. Use Balena Tunnel to access the host OS.
  2. Copy files from the desired container to the host OS.
  3. Run SCP on my local machine to retrieve the files from the host OS.

The challenge is that some field files (e.g., video files) can be larger than 100GB, but the host OS only has about 30GB of available storage. This creates a bottleneck when transferring large files because I cannot temporarily store them on the eMMC. I have struggled to SCP from the container to my local machine directly, but I’ve had no problems through the tunnel in the hostOS.

To work around this, I am exploring two potential solutions:

  1. Use Balena Tunnel to directly access the container I need, allowing me to run SCP directly from the container to my local machine.
  2. Use Balena Tunnel to access the host OS and configure SCP to copy files through the tunnel directly from the container, without temporarily storing them on the eMMC.

Does anyone have experience with these approaches or suggestions on how to implement either solution? Any advice on setting up a direct file transfer pipeline through Balena Tunnel would be greatly appreciated.

Thanks in advance!

2

One suggestion (assuming the video files live in a named volume):

Take a peek (from the host OS) at /var/lib/docker/volumes/. In there you’ll find e.g 12345_volume-name/_data/... (with the contents of the volume)

I believe that should remove the need to make a copy, and instead you can access the files directly.