How to get rid of Online (VPN only) mode in a beaglebone-black device

agaurav · November 20, 2020, 2:47am

Hi, I have a couple of questions/doubts.
1.) After getting “No Space Left On Device” message in the log window, my device has gone into Online (VPN only) mode and I am not able to restart/reboot it any more.
It gives the below error.

How can I go back to normal without re-flashing the device?

2.) I also see that the storage if full but I am not sure what caused this and I am somewhat certain it was not my applications. I am also not able to see the logs anymore because I am not able to ssh inside the container.

What could have caused the memory to be full?

Is the docker diffs in /var/lib/docker/?

If Yes, Can I clean some of these?

Can someone please give me insight of what is going on.

Thanks

ab77 · November 20, 2020, 2:55am

Hi there, if you are able to connect to this device still with a terminal, running df -h would be the first thing I would run to see where the disk space has been consumed. If you grant support access to the device and share the uuid, we could also take a look for you.

agaurav · November 20, 2020, 3:16am

@ab77
Hi, The device came back to normal itself after sometime. I am keeping an eye on it. I will let you know if it happens again.
Thanks

anujdeshpande · November 20, 2020, 7:02am

Hi

It would be nice if you can also enable persistent logging on the device. This will help debug this issue faster if it happens again.

Clint · January 12, 2021, 2:59pm

Hi,

I encounter the same issue but on Jetson Nano devices. Persistent logging is enabled. I see following messages in journaling:

Jan 12 14:40:51 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 12 14:40:51 37b512f systemd[1]: balena.service: Found left-over process 7985 (balena) in control group while starting unit. Ignoring.

pipex · January 12, 2021, 6:30pm

Hi.

Did you also get the No space left on your device on these devices? What is the output of the df -h command? Could you also provide the output of running the following command on your devices?

journalctl -u resin-supervisor --no-pager

If you can run ‘Device Diagnostics’ and share the output with us, that would also be very helpful. Thank you!

Clint · January 12, 2021, 6:49pm

I didn’t saw the “No space left …” message but I know the devices are full. When I check using df and du, I see almost all data is consumed by /mnt/data/docker/overlay2.

The output of the command journalctl -u resin-supervisor --no-pager (most important part):

Jan 11 04:41:19 37b512f resin-supervisor[5193]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Jan 11 04:41:19 37b512f resin-supervisor[5193]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Jan 11 04:41:35 37b512f resin-supervisor[5193]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Jan 11 04:41:35 37b512f resin-supervisor[5193]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Jan 11 04:41:54 37b512f resin-supervisor[5193]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Jan 11 04:41:54 37b512f resin-supervisor[5193]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Jan 11 04:42:09 37b512f resin-supervisor[5193]: [event]   Event: Device state report failure {"error":{"message":""}}
Jan 11 04:42:18 37b512f resin-supervisor[5193]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Jan 11 04:42:18 37b512f resin-supervisor[5193]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
-- Reboot --
Jan 12 18:33:30 37b512f systemd[1]: resin-supervisor.service: Start-pre operation timed out. Terminating.
Jan 12 18:33:30 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=killed, status=15/TERM
Jan 12 18:33:30 37b512f resin-supervisor[27702]: inactive
Jan 12 18:33:30 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 12 18:33:30 37b512f systemd[1]: resin-supervisor.service: Failed with result 'timeout'.
Jan 12 18:33:30 37b512f systemd[1]: Failed to start Balena supervisor.
Jan 12 18:33:42 37b512f resin-supervisor[27747]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Jan 12 18:33:42 37b512f resin-supervisor[27755]: activating
Jan 12 18:33:42 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 12 18:33:42 37b512f systemd[1]: resin-supervisor.service: Failed with result 'exit-code'.
Jan 12 18:33:42 37b512f systemd[1]: Failed to start Balena supervisor.
Jan 12 18:33:54 37b512f resin-supervisor[27952]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Jan 12 18:33:54 37b512f resin-supervisor[27961]: activating
Jan 12 18:33:54 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 12 18:33:54 37b512f systemd[1]: resin-supervisor.service: Failed with result 'exit-code'.
Jan 12 18:33:54 37b512f systemd[1]: Failed to start Balena supervisor.
Jan 12 18:35:36 37b512f systemd[1]: resin-supervisor.service: Start-pre operation timed out. Terminating.
Jan 12 18:35:36 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=killed, status=15/TERM
Jan 12 18:35:36 37b512f resin-supervisor[28329]: inactive
Jan 12 18:35:36 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 12 18:35:36 37b512f systemd[1]: resin-supervisor.service: Failed with result 'timeout'.
Jan 12 18:35:36 37b512f systemd[1]: Failed to start Balena supervisor.
Jan 12 18:37:18 37b512f systemd[1]: resin-supervisor.service: Start-pre operation timed out. Terminating.
Jan 12 18:37:18 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=killed, status=15/TERM
Jan 12 18:37:18 37b512f resin-supervisor[28556]: inactive
Jan 12 18:37:18 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 12 18:37:18 37b512f systemd[1]: resin-supervisor.service: Failed with result 'timeout'.
Jan 12 18:37:18 37b512f systemd[1]: Failed to start Balena supervisor.
Jan 12 18:39:00 37b512f systemd[1]: resin-supervisor.service: Start-pre operation timed out. Terminating.
Jan 12 18:39:00 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=killed, status=15/TERM
Jan 12 18:39:00 37b512f resin-supervisor[28775]: inactive
Jan 12 18:39:00 37b512f systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 12 18:39:00 37b512f systemd[1]: resin-supervisor.service: Failed with result 'timeout'.
Jan 12 18:39:00 37b512f systemd[1]: Failed to start Balena supervisor

pipex · January 12, 2021, 7:36pm

Hi again, if the drives are full, that would explain the Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. You could try getting the engine log using journalctl -u balena --no-pager -n 500 to see if there is any relevant information there.

Otherwise to clean up the data dir, you could first see if you can get the engine up with systemctl restart balena and check for status systemctl status balena. If that works, you can try doing a balena system prune as well as deleting any unused images you see listed with balena images.

If restarting the engine fails, you can try cleaning up the /var/lib/docker directory, but this will delete all your application images and data. If you are ok with that, you can do the following to perform a cleanup

systemctl stop resin-supervisor
systemctl stop balena
rm -rf /var/lib/docker/{aufs,overlay,containers,image,tmp}
systemctl start balena
systemctl start resin-supervisor

Do you have any clues about why your data partition might be full, what is the size of your drive? What is the size of the application images?

Clint · January 12, 2021, 10:30pm

Balena journaling:

Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 31936 (exe) in control group while starting unit. Ignoring.                                                 
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                                 
Jan 12 22:12:18 37b512f balenad[15249]: time="2021-01-12T22:12:18.952836917Z" level=info msg="Starting up"                                                                                
Jan 12 22:12:18 37b512f balenad[15249]: time="2021-01-12T22:12:18.952922492Z" level=warning msg="Running experimental build"                                                                  
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 31938 (balena) in control group while starting unit. Ignoring.                                                   
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                        
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 31937 (balena-healthch) in control group while starting unit. Ignoring.                             
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                          
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32334 (exe) in control group while starting unit. Ignoring.                                                  
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                            
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32335 (balena-healthch) in control group while starting unit. Ignoring.                             
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                    
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32338 (balena) in control group while starting unit. Ignoring.                                
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                   
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32474 (balena-healthch) in control group while starting unit. Ignoring.                              
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                   
Jan 12 22:12:18 37b512f balenad[15249]: time="2021-01-12T22:12:18.958820500Z" level=info msg="libcontainerd: started new balena-engine-containerd process" pid=15267        
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32473 (exe) in control group while starting unit. Ignoring.                                         
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                        
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32479 (balena) in control group while starting unit. Ignoring.                                                                
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                      
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32232 (balena-healthch) in control group while starting unit. Ignoring.                        
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                    
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32231 (exe) in control group while starting unit. Ignoring.                                            
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                            
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32238 (balena) in control group while starting unit. Ignoring.                                           
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                      
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32522 (balena) in control group while starting unit. Ignoring.
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                           
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32518 (balena-healthch) in control group while starting unit. Ignoring.                
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32517 (exe) in control group while starting unit. Ignoring.                                                                    
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                                                  
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32590 (exe) in control group while starting unit. Ignoring.                                                                    
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32591 (balena-healthch) in control group while starting unit. Ignoring.                                                        
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                                                  
Jan 12 22:12:18 37b512f systemd[1]: balena.service: Found left-over process 32595 (balena) in control group while starting unit. Ignoring.                                                                 
Jan 12 22:12:18 37b512f systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.                                                                  
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.028157130Z" level=info msg="starting containerd" revision= version=1.2.0+unknown                                                         
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.029566747Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1                        
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.029672010Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.029672010Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1                   
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.033258920Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="modprobe aufs failed: \"modprobe: FATAL:
 Module aufs not found in directory /lib/modules/4.9.140-l4t-r32.4.2\\n\": exit status 1: skip plugin" type=io.containerd.snapshotter.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.033376319Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.033484863Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.033745651Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.033815757Z" level=info msg="metadata content store policy set" policy=shared
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034074461Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034134202Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034239882Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034300196Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034356499Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034417698Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034466293Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034514576Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034562233Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034610672Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.034853178Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.035034484Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037132504Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037221308Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037356415Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037413969Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037465376Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037511419Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037555535Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037602776Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037655173Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037700538Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037748404Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037831583Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037876793Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037929763Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.037990337Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.038037005Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.038482693Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.038670302Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock.ttrpc
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.038818222Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.038859108Z" level=info msg="containerd successfully booted in 0.012270s"
Jan 12 22:12:19 37b512f balenad[15249]: time="2021-01-12T22:12:19.055090492Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
Jan 12 22:12:20 37b512f balenad[15249]: failed to start daemon: Unable to get the TempDir under /var/lib/docker: mkdir /var/lib/docker/tmp: no space left on device
Jan 12 22:12:20 37b512f systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE
Jan 12 22:12:20 37b512f systemd[1]: balena.service: Failed with result 'exit-code'.
Jan 12 22:12:20 37b512f systemd[1]: Failed to start Balena Application Container Engine.
Jan 12 22:12:20 37b512f systemd[1]: Dependency failed for Balena Application Container Engine.
Jan 12 22:12:20 37b512f systemd[1]: balena.service: Job balena.service/start failed with result 'dependency'.
Jan 12 22:13:30 37b512f balenad[14838]: Parent process exited
Jan 12 22:13:31 37b512f balenad[14899]: Parent process exited
Jan 12 22:13:34 37b512f balenad[14937]: Parent process exited
Jan 12 22:13:36 37b512f balenad[14977]: Parent process exited
Jan 12 22:13:38 37b512f balenad[15025]: Parent process exited

I resolved the issue once removing the biggest folder in /mnt/data/docker/overlay2. And then do balena system prune which removes the dangling data.

I have no idea why the data partition went full, I can say the size of images/containers is approximately 60% of the full storage size.

anujdeshpande · January 13, 2021, 7:21am

Great to hear that you were able to resolve the issue after removing some stuff.

Can you share the size of your images?
Did you happen to see if there were a lot of errors (failed timeouts, etc) when updating your images?

Clint · January 13, 2021, 3:49pm

I solved the issue on a device once, but now I still have 2 devices running with the issue.

Size images:

registry2.balena-cloud.com/v2/3703309fe3d4fb268b11845c84441f79   delta-7a090c524bdb5ed8   85797bac958c        8 days ago          148MB
registry2.balena-cloud.com/v2/097553702371732f0c167e18a33bf44d   delta-ab3f91746ca969c0   e192667a6c6f        8 days ago          5.82GB
registry2.balena-cloud.com/v2/ef72aa350440259e21223968208ea21f   delta-8748c8d0f6f1362a   066e37187deb        3 weeks ago         450MB
registry2.balena-cloud.com/v2/d8652a62274a84a89010308bd0e4edc3   delta-d6ea46713927483c   e17cc4b23e92        4 weeks ago         609MB
registry2.balena-cloud.com/v2/711b6508db27833d5222f6de9d3aa88a   delta-02c21354053e8689   4d8a1870f023        7 weeks ago         146MB
registry2.balena-cloud.com/v2/c232396d9610ffc6a591b504da52bc59   delta-aa1f23d98eef4037   09852de1ac8e        2 months ago        150MB
registry2.balena-cloud.com/v2/9f11c39b2159a338fe39c35bdbd1bc8c   delta-4320b30b9f372db5   ab9585082ed2        3 months ago        153MB
balena/aarch64-supervisor                                        v11.14.0                 25d6abae14f0        4 months ago        72.3MB
registry2.balena-cloud.com/v2/d4b9dad4b74b6bd8636c8618eecb8d39   latest                   25d6abae14f0        4 months ago        72.3MB
balena-healthcheck-image                                         latest                   a29f45ccde2a        12 months ago       9.14kB

Maybe a good idea to give you guys access to the devices?

ab77 · January 13, 2021, 9:33pm

Hi there, well you do have one 5GB+ image there, so this one is most likely going to take some time to download/calculate deltas between releases and/or eat up disk space if there isn’t a lot to begin with. How much disk space do you have available on this device (/mnt/data)?

ab77 · January 14, 2021, 12:29am

You may also benefit from reviewing your build as a ~ 6GB docker image in IoT space is unusual. Perhaps this build can be optimised to use multi-stage builds, not to persist all of the build dependencies in the final runtime?

Clint · January 14, 2021, 10:48am

The mount /mnt/data is 14GB.
We have different layers and Dockerfiles in the images/containers, but it looks like when a Balena service is defined one final runtime is created.
The image is so big because of all the NVIDIA stuff. It’s a NVIDIA Jetson Nano device

rahul-thakoor · January 15, 2021, 10:59am

Hey there! I’m not sure I follow what you mean by

but it looks like when a Balena service is defined one final runtime is created.

Can you please clarify? or better, can you please share your Dockerfile?