I’m currently working on automating a task that involves remotely copying files from a field data-logger device and uploading them to a local machine. Unfortunately, the Balena releases for these devices are managed by another group, so I have limited control over how the system is set up. Here’s what I’ve been able to achieve so far through my process research:
Use Balena Tunnel to access the host OS.
Copy files from the desired container to the host OS.
Run SCP on my local machine to retrieve the files from the host OS.
The challenge is that some field files (e.g., video files) can be larger than 100GB, but the host OS only has about 30GB of available storage. This creates a bottleneck when transferring large files because I cannot temporarily store them on the eMMC. I have struggled to SCP from the container to my local machine directly, but I’ve had no problems through the tunnel in the hostOS.
To work around this, I am exploring two potential solutions:
Use Balena Tunnel to directly access the container I need, allowing me to run SCP directly from the container to my local machine.
Use Balena Tunnel to access the host OS and configure SCP to copy files through the tunnel directly from the container, without temporarily storing them on the eMMC.
Does anyone have experience with these approaches or suggestions on how to implement either solution? Any advice on setting up a direct file transfer pipeline through Balena Tunnel would be greatly appreciated.
Unfortunately It looks like that directory only has data on mounted devices. I thought this was on the right track with one of my units, but it turns out that one has been writing to the eMMC since it doesn’t have an NVME installed.
The NVME is mounted in another container. Is there any other way you know where I might be able to access it similarly to this method?
@atwoodc first of all, we strongly discourage you to use the balena tunnel. The balena tunnel is designed to maintain stability and reliability to all the customers that need to interact to their remote devices. We do not recommend to send huge amounts of data through it. Actually we are thinking on putting in place limits.
We have been thinking about alternatives. Does the remote device run balena? I was thinking if you can add an extra container that uploads the files to S3, or use a kind of p2p system in case there is an issue with connectivity.
A colleague mentioned thet there is a relay based system that may be worth to try.
hi, @mpous. Thanks for all those suggestions! I’ll have to chat with my team, but I’m sure at least one of them will work well for what we need. I’ll come back and mark resolved if anything we try works.
Extra context. We are using the embedded version of the Pi 4 with Balena OS 4.0.23.
Use case wise, we have a remote telematics fleet that also records video. There’s infrastructure in place for sensor data and short (4 second) video clips. Currently, video is taken of our remote units by using removable hard drives that a technician has to remove and upload from satellite offices, but we’d like to be able to recover video more often to reduce that demand on technicians and have more of an “on-demand/as needed” data extraction service available.
We’re not sure how willing our hardware vendor will be to committing big changes since our development contract is coming to a close, so a lightweight solution would be ideal.