Hey
I run balena-engine on Debian distribution. I pull an image which have around 900 Mo. I remark on layer repository (balena-engine/vfs/dir) an excessive size of layer, around 5Go. There is only this image download on the board.
When I realize the pull operation with docker-engine, layers have around 900 Mo like the images. Why are there this difference ? And how reduce layer size ? 5Go is too much for our board.
Thanks,
Ben
based on this post https://docs.docker.com/v17.09/engine/userguide/storagedriver/imagesandcontainers/#images-and-layers layer is a instruction on Dockerfile. For Balena-engine, default repository of layer is /var/lib/balena-engine/vfs/dir/.
I determine their size with du -hs command.
Hey, a few questions to help us advise better:
This should help us replicate what you’ve observed.
Also, in general for issues with balena-engine, it’s best to file a Github issue at https://github.com/balena-os/balena-engine/issues to get to the maintainers the quickest possible way! Just a future reference.
How are you installing balena-engine in your distribution?
i downloaded package with cmd : curl -sfL https://balena.io/engine/install.sh | sh and i followed this issue 55 to setting and start the service.
https://github.com/balena-os/balena-engine/issues/55
What version of Debian are you running?
It is Debian 9 stretch
Can you tell us what flags you are starting the daemon with?
More over, I modified balena-engine.service to move balena-engine directory to a directory mounted on an usb key. My internal memory is limited to 8Go.
ExecStart=/usr/local/bin/balena-engine-daemon –data-root /mnt/usb/balena-engine-dir -H fd://
What image are you pulling?
It’s a private image base on this dockerfile :
FROM python:2.7-alpine3.9
RUN echo "http://mirror.aarnet.edu.au/pub/alpine/v3.8/main" > /etc/apk/repositories && \
echo "http://mirror.aarnet.edu.au/pub/alpine/v3.8/community" >> /etc/apk/repositories && \
echo "@testing http://nl.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories && \
apk add --no-cache libuuid su-exec curl git unzip make gcc g++ supervisor libxml2-dev libxslt-dev \
sqlite sqlite-libs sqlite-dev libxslt-dev glib cairo cairo-dev cairo-gobject py2-cairo \
py3-cairo openssl-dev linux-headers gobject-introspection-dev bluez-deprecated \
py-cairocffi@testing py2-gobject3 dbus && pip install pycairo && pip install pygobject
COPY requirements.txt /
RUN pip install --upgrade pip && \
pip install pycairo -r requirements.txt
COPY . /app
EXPOSE 5504
ENTRYPOINT "/entrypoint.sh"
requirement.txt :
appdirs == 1.4.0
enum34 == 1.1.6
pyserial == 3.4
smbus2 == 0.1.4
rpyc == 3.3.0
paho-mqtt == 1.4.0
pysmbus == 0.1.*
boto == 2.39.0
pymongo == 3.2.1
cryptography == 2.6.1
cffi == 1.8.3
smbus-cffi == 0.5.1
requests == 2.4.3
python_gsmmodem == 0.9
lxml == 3.4.0
pbkdf2 == 1.3
pycrypto == 2.6.1
Flask == 0.12.2
Flask_Cors == 3.0.4
Flask_HTTPAuth == 3.2.3
Flask_RESTful == 0.3.6
Flask_SocketIO == 3.0.1
pony == 0.7.1
expiringdict == 1.1.4
mock == 2.0.0
jsonmerge == 1.5.1
coverage == 4.5.1
pygobject == 3.32.0
Werkzeug == 0.14.1
apispec == 0.35.0
eventlet == 0.23.0
itsdangerous == 0.24
psutil == 5.4.5
pydbus == 0.6.0
bluepy==1.3.0
simple-crypt==4.1.7
bson == 0.5.8
dictdiffer == 0.7.1
pymodbus == 2.1.0
pyzmq == 17.1.2
ifaddr == 0.1.6
six == 1.11.0Okay, checking this out, and will come back with the results. Thanks for the details!
Hi, so I was trying to reproduce this:
balenalib/amd64-node:12-build
When I’ve checked, both in the balenaEngine storage and the Docker storage it was the same, and that size that is suggested by the images output
Docker:
greg@debianengine:/var/lib/docker$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
balenalib/amd64-node 12-build eaa6b216aaa0 6 days ago 756MB
greg@debianengine:/var/lib/docker$ sudo du -h -d 1 |sort -h -r
789M .
786M ./overlay2
2.0M ./image
72K ./buildkit
52K ./network
28K ./volumes
20K ./plugins
20K ./builder
4.0K ./trust
4.0K ./tmp
4.0K ./swarm
4.0K ./runtimes
4.0K ./containers
balena-engine:
greg@debianengine:/mnt/balena$ sudo balena-engine images
REPOSITORY TAG IMAGE ID CREATED SIZE
balenalib/amd64-node 12-build eaa6b216aaa0 6 days ago 756MB
greg@debianengine:/mnt/balena$ sudo du -h -d 1 /mnt/balena |sort -h -r
789M /mnt/balena
786M /mnt/balena/overlay2
1.9M /mnt/balena/image
60K /mnt/balena/containerd
40K /mnt/balena/network
28K /mnt/balena/volumes
20K /mnt/balena/plugins
20K /mnt/balena/builder
4.0K /mnt/balena/trust
4.0K /mnt/balena/tmp
4.0K /mnt/balena/runtimes
4.0K /mnt/balena/containers
So at least with the overlay2 storage engine it doesn’t make a difference, and things work as they should.
Then I tried it out also with vfs storage driver (looked up the vfs docs for Docker as it seemed the actual suspect), pulling the same image, after setting the storage engine in both:
Docker with vfs:
greg@debianengine:/var/lib/docker$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
balenalib/amd64-node 12-build eaa6b216aaa0 6 days ago 756MB
greg@debianengine:/var/lib/docker$ sudo du -h -d 1 |sort -h -r
5.1G ./vfs
5.1G .
1.9M ./image
72K ./buildkit
40K ./network
28K ./volumes
20K ./plugins
20K ./builder
4.0K ./trust
4.0K ./tmp
4.0K ./swarm
4.0K ./runtimes
4.0K ./containers
Balena-engine with vfs:
greg@debianengine:/mnt/balena$ sudo balena-engine images
REPOSITORY TAG IMAGE ID CREATED SIZE
balenalib/amd64-node 12-build eaa6b216aaa0 6 days ago 756MB
greg@debianengine:/mnt/balena$ sudo du -h -d 1 /mnt/balena |sort -h -r
5.1G /mnt/balena/vfs
5.1G /mnt/balena
1.9M /mnt/balena/image
60K /mnt/balena/containerd
40K /mnt/balena/network
28K /mnt/balena/volumes
20K /mnt/balena/plugins
20K /mnt/balena/builder
4.0K /mnt/balena/trust
4.0K /mnt/balena/tmp
4.0K /mnt/balena/runtimes
4.0K /mnt/balena/containers
Thus the result is the same here, using docker or balena-engine is not the cause of the size difference but the ability of the storage engines.
vfs) when you are comparing the size taken up?vfs is forced when used on that? I would think if you format the external drive as ext4, then the overlay2 engine should be able to use space efficiently for you.Thks for you feedback. Indeed docker and balena-engine use different storage driver (vfs for balena-engine and devicemapper for docker). External USB is formatted on ext4. I am going to search a way to change storage driver to fix my issue.
Unfortunately, I can’t change default storage driver with balena-engine. OverlayFS and aufs is not included on my distribution. Docker-engine use devicemapper like default storage driver. I would like to use the same driver for balena-engine but I have this error :
level=error msg=“Failed to GetDriver graph” driver=devicemapper error=“graphdriver plugins are only supported with experimental mode”
Docker root directory is stored on same place then balena-engine root directory. Based on docker article on devicemapper I changestorage driver for balena-engine by writing { “storage-driver”: “devicemapper” } on /etc/balena-engine/daemon.json.
Docker version :
docker info
Client:
Debug Mode: false
Server:
Server Version: 19.03.1
Storage Driver: devicemapper
Balena-engine version :
balena-engine info
Server Version: v17.12.0
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Hi there! The error graphdriver plugins are only supported with experimental mode. indicates that you may put the daemon on experimental mode by setting "experimental": true on daemon.json docker configuration file for that, but it would be recommended to change default storage driver.
You can also add the --experimental flag to the engine startup line in your service (and the storage driver settings as well with --storage-driver, can see all the daemon flags with balena-engine-daemon --help).
What distribution/version are you using?
And out of curiosity, what makes you choose balenaEngine over docker for your use case?
Even with experimental flag to true, I can’t change storage driver. ( level=warning msg=“Unable to locate plugin: devicemapper”).
I use Debian 9 with custom kernel and OverlayFS module is not set. I am going to recompile kernel to have the recommend storage driver.
I work on custom board based on IMX6-DL and we wants to support balena tools on it. I try to setting balenaOS module on our DebianOS to communicate with BalenaCloud.
Apart from using a better overlay2 storage driver,
another thing I can suggest is to perhaps investigate is to find what is eating the storage space.
e.g. Check the following three commands. The sizes are quite consistent. You should be able to run the same for docker. Then compare outputs. Something strange is happening here and 4gb isn’t a small amount to eat up…
root@balena:~# du -hs /var/lib/balena/
208M /var/lib/balena/
root@balena:~# balena images
REPOSITORY TAG IMAGE ID CREATED SIZE
balenalib/intel-nuc-debian latest d825379dc3e1 7 days ago 162MB
balena/amd64-supervisor v10.0.3 49183b33301b 5 weeks ago 49.5MB
hello-world latest fce289e99eb9 7 months ago 1.84kB
root@balena:~# balena system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 3 1 211.9MB 162.4MB (76%)
Containers 1 0 84B 84B (100%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
root@balena:~#