Unsure if anyone has tried to use disk quota on the named volume resin-data for a balena image. Currently I got a solution where resin-data is mounted to /persistent and would like to perform quota on this mount point.
From my understanding the following is needed in order to perform disk quota:
The balena container contain “UNCONFIGURED FSTAB FOR BASE SYSTEM” in /etc/fstab. Unsure of the implication of this statement.
Question:
Is it possible to modify the /etc/fstab in order for the quota properties to be added to the disk mount of the balena container?
Unsure if balena device host changes are required for this to work, or if this could be done from the container only + named volume setup from the docker-compose file?
Can you please elaborate your applications use-case a bit and share how you intend to use disk quotas?
As far as I know, in the host OS, we don’t use disk quotas.
And for the containers volumes, docker doesn’t really have a nice way to limit storage size of a volume. Some filesystem configurations allow it in docker but we use aufs/overlay backed by ext4 (which is quite a common approach) doesn’t support restricting the size of the volume…
Allow for disk usage to be available per user/group in a folder in a balena container. As of now the device host name volume ‘resin-data’ is mounted into the container as /persistent. Various folders exists within the /persistent folder which needs to be limited in terms of disk usage per user/group.
Ordinary this can be achieved by use of kernel config CONFIG_QUOTA=y and CONFIG_QUOTACTL=y. On balenaOS 2.38.3+rev4 “zcat /proc/config.gz | grep QUOTA” returns “# CONFIG_QUOTA is not set”. This may be fine as CONFIG_QUOTA=m and CONFIG_QUOTACTL=m should be possible and load modules dynamically at container startup. If QUOTA can be loaded as module dynamically or if it is needed to be compiled into the kernel is still unknown.
Secondly, the /etc/fstab needs to be changed which may be the blocking issue. Ordinary a fstab would look like the below (Disk quota - ArchWiki):
Unsure if such changes on a balena container will take affect?
Assuming the kernel module loaded dynamically and fstab can be changed in the container and given docker/balena supports this, then disk use limits can be set per user/group in sub folders of /persistent
Thank-you for elaborating your use-case. I’m not sure whether enabling CONFIG_QUOTA will enable you to limit storage used by a user inside the docker container. It might work inside a privileged container.
Ideally docker should provide a way to restrict a volume’s storage size. Then each user/group can be using a different storage volume and the storage size will be limited.
However looking at dockers docs, the option only seems to exist for some storage drivers backed by xfs. We use ext4 in our OS for the data partition.
A workaround would be to make a small daemon that periodically checks disk space used and cleans up if it goes over a limit. This isn’t really a storage limit per user/group. But a workaround / possible way forward.
I’ve asked internally if there is a better way forward for your use-case.
Another idea to enforce something like a quota within a volume, that can be part of your container/service start script
add a volume to the service
during startup, if doesn’t exists, create a file with a specific size that you want your quota to be, on that volume; format it to a file system of your choice (likely suggesting ext4)
mount that file as a loopback device inside your container
write to that loopback mounted folder instead of directly to the volume
This is a bit manual, but should create an effective quota. What do you think?
We are also looking at internally how to make a better (less manual) quota system part of the product (the OS, the container engine, or some other way).
I guess that could be a solution, but requires some setup.
Just thinking that if you got a kernel supporting CONFIG_QUOTA and CONFIG_QUOTACTL and if a ext4 partition is mounted into a balena container with the mount option usrquota and grpquota then this mount point will be under QUOTA control. The kernel understands QUOTA and the mounted partition is set to use the QUOTA feature.
The user/group quota piggybacks on file ownership to calculate disk usage on a per user/group basis. You can use that from inside a container because you can create files as any user/group inside the namespace.
The problem is of course that we don’t have user namespaces enabled and so two containers that use say user id 2 to store all their files will step on each other toes when working with limits. On top of that if you want to have multiple limits for different volumes you have to store the files as different user ids etc, which is very cumbersome.
XFS supports a feature called project quotas. A project is yet another piece of metadata that each file has and it’s just an id, similar to user id or group id. The advantage is that it’s not associated with anything so you don’t have to jump through hoops to tag your files. Docker integrated that to support limits per volume.
Now, etx4 also supports project quotas since linux 4.5 so the way forward for balenaOS would be to enable that feature in our filesystem and then add support for that in docker and as an extension to balenaEngine. The code shouldn’t be very different.
There is one unknown at the moment. All this work is for containers and not volumes (aren’t sure yet if it applies to volumes as well). But I guess if we get storage limits in containers, there should be a way forward for applications to utilise that.
As you can imagine, this is a rather larger piece of work and I can’t provide any timelines or even if we will fix this upstream or not. If upstream does add the ext4 project quota feature in docker, we will surely pull it in balenaEngine.