/mnt/data full; what to do?

Product support
,
1

Hello,
one of my RPI3 won’t start because /mnt/data is full.
and the biggest directory is /mnt/data/docker/aufs/diff :
image
image

Is there any workaround for this?
Thanks for your help. :wink:

4

Hey @EugeneGwon,

I’m wondering what exactly is causing this issue, could you send me your dashboard URL and enable support access please?

My thoughts are that some intermediate layers are not getting cleaned up, but i’ll need to investigate to confirm this.

Thanks,
Cameron

5

Thanks. I enabled support access. here’s my dashboard URL :
https://dashboard.resin.io/devices/b14372edc03230676c755e77525beed1/summary

9

So to start with, docker was unable to start on this device because of out of space issues, this made diagnostics fairly difficult.

To combat that I removed everything in /var/lib/docker/aufs/diff, which allowed docker to startup, but running the standard tools like docker images -aq to find intermediate layers left around etc did not help - perhaps because of removing the diff directory, I’m not 100% sure of the mechanism docker uses to find these layers.

But this also created another problem, in that docker’s internal state did not match the filesystem. I tried a docker system prune -a which also did not seem to help. At this point I did

systemctl stop docker # Ensure docker has been stopped
rm -rf /var/lib/docker/* # Remove all docker state
systemctl start docker # Start docker again, this will recreate the docker directory contents
update-resin-supervisor # Re-download the supervisor
systemctl start resin-supervisor # start the supervisor

Now depending on your version of the supervisor, you may need to re-download the app image, as the supervisor expects it to be present. You can tell if this is the case by looking at the supervisor logs on the dashboard, and you will see something similar to:

Failed to start application 'registry2.resin.io/<appname>/<commit> due to '(HTTP code 404) no such image - no such image: registry2.resin.io/<appname>/<commit>: No such image: registry2.resin.io/<appname>/<commit>:latest '

With the <appname> and <commit> values being the correct ones for your application. In your case @EugeneGwon I’ve pulled that image again, and you should be good to go.

I’m sorry I did not find out the root cause of this issue, I’m fairly sure that this shouldn’t happen with a standard resin workflow, so if this does happen again, please do let us know!

Thanks,
Cameron

1 Like
10

@CameronDiver thank you very much! It works without problem :wink: