I am using balena os on Raspberry Pi.
Now I want to store the database and logs. I have multi containerized Application.
I read the documentation But I am unable to understand How can I add this in my docker-compose.yml?
Can anybody here provide a basic example for adding persistent storage for multiple application?
Hi there @sharvin26,
Permanent storage for a multicontainer application is described in our documentation here: https://www.balena.io/docs/learn/develop/multicontainer/#named-volumes
As a brief guide, you can define a named volume in your docker-compose.yml and then bind that volume to any services you define, for example:
...
volumes:
shared-data: {}
service-one-data: {}
service-two-data: {}
services:
serviceOne:
...
volumes:
- 'shared-data:/shared'
- 'service-one-data:/private'
serviceTwo:
...
volumes:
- 'shared-data:/shared'
- 'service-two-data:/private'
In the example above, the āshared-dataā volume is permanent storage shared by both services serviceOne and serviceTwo bound to the /shared path for both. service-one-data is only available to serviceOne, bound to the /private path, and service-two-data is only available to serviceTwo, bound to the /private path. In effect, there are three permanent storage volumes, one shared by both, and two private volumes.
Hope this helps, if you still have questions, please let us know!
Best regards, Heds
I read the documentation that you shared and a discussion on the balena forum #22
It cleared some concepts but I still have some doubts. Now I have a database file that I want to persists through all container update and changes.
First Question =>
Where should I add the database to persist it? and How can I add it?
Second Question =>
What I understood from the above 2 links is that using balena volume ls I can confirm if the volume is loaded. But I am confused about How docker compose file should be constructed for that purpose?
volumes:
Test-Database: {}
service-1:
volumes:
- 'Test-Database': #what should be added here?
Question 3 =>
what should I pass the database path in my application code?
Hi @Sharvin26,
The docker-compose volume binding works in exactly the same was as regular docker-compose named volumes, and the link I provided (https://www.balena.io/docs/learn/develop/multicontainer/#named-volumes) should explain how to use it.
As stated in the docs āNamed volumes can be given arbitrary names and can be linked to a directory in one or more containersā. So you are linking the volume youāve created to a named directory in your container:
volumes:
- 'Test-Database:/path/to/persistent/database/in/service/container'
Best regards, Heds
I did that I constructed docker-compose.yml in this way =>
volumes:
Test-Database: {}
service-1:
- 'Test-Database': /usr/src/app/app1/Database/test.db
Note
My Directory Tree Structure =>
app1
- application
- database
- test.db
app2
docker-compose.yml
Continuously itās only giving me installing service.
[Logs] [6/19/2019, 3:04:27 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs] [6/19/2019, 3:04:28 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs] [6/19/2019, 3:04:32 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs] [6/19/2019, 3:04:37 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs] [6/19/2019, 3:04:46 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs] [6/19/2019, 3:05:04 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs] [6/19/2019, 3:05:37 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
What am I doing wrong here?
- 'Test-Database': /usr/src/app/app1/Database/test.db
is not correct and does not conform to the docker-compose specification or the examples above. As noted in both Docker and balena documentation, everything is in quotes and is the path of a directory, not a file:
- 'Test-Database:/usr/src/app/app1/database'
(note also that in your file tree the database directory appears to be in lower case)
Iād highly recommend you look at some of the projects in balena-io-projects, perhaps https://github.com/balena-io-projects/balena-sense/blob/master/docker-compose.yml which also uses a persistent storage volume.
Thanks for the Solution. Itās working now.
But suppose I have a file which is dynamically created when running the Application and I want to persists that How can I add that to the persistent storage in that case?
I would just move that file in the persistent /path/to/persistent/ directory.
How can I do that? Do I need to create a file using a command in dockerfile and add that file path in docker compose like this =>
- 'Test-Database:/usr/src/app/app1/database'
Hi @Sharvin26, in the dockerfile you are linking a directory used by your service (in the container) to a persistent volume in the host OS (outside of the container). Then in your application code, what you write under this linked directory persists over application and host OS updates. You donāt need to specify a file name in the docker compose file. Itād work fine as long as you write your files under that linked directory.
In your case, when you setup the database in one of your services, you could specify that this database is written on persistent storage. Heds linked an example how this is done with influxdb.
Hope these make sense.
Thanks for the Response. I got that.
I have successfully added the database as a persistent storage but when it comes to logs I am facing issue.
I have a logging mechanism in my application but when it runs in the docker container it doesnāt write anything in the Log File.
I have confirmed this by doing ssh into the container and checking if anything is written that logfile.
Where is this file supposed to be located @Sharvin26?
Perhaps you could post some code and we could take a look. In general, being a container should not affect something so mundane as writing to a logfile.
Hi all.
Iāva just one more question to add to this topic (but escuse me if my question is a dumb question 
).
It is possible to directly create a volume on an unmounted drive (like a USB stick on the host),ā¦something like :
ā volumes:ā
ā - āshared-data:/dev/sda1ā
ā¦ ord do we have to mount it before on the host, or maybe create a container that mount the USB on its filesystemā¦ and expose it to others containers ā¦
regards
What are you trying to do? Do you want to extract data from you device (container)?
Hi.
I am trying to find the best way to share an USB stick of my host BalenaOS ?
-can i mount it directly on my balenaOS (using FASTAB, or maybe a startup srcipt)
-do i have to mount it on a āData COntainerā and share to to others container via Volumes ?
Iāve maybe just find a good way to do this using the udevrules in the json.config ?
Is it a good idea ?
Thank for you help.
Interesting questions.
Regarding the mounting.
I think you have to mount it by means of startup script in one of your containers.
Here 2 examples:
Note that you canāt mount the same drive partition in multiple containers (https://github.com/janvda/balena-node-red-mqtt-nginx-TIG-stack/issues/3).
ā¦ but what I didnāt try out (like you suggested) is to mount it in a data container and share it to other containers via Volumes. I think that is certainly worth a try. So for that I would create a subfolder in that volume and mount the drive to that subfolder.
If that doesnāt work, you can also mount it in one container and share it via nfs to the other containers.
kr
Jan.
Iāve just seen that we can also interact with the host config using the supervisor API
https://www.balena.io/docs/reference/supervisor/supervisor-api/
Waou, in fact the BalenaOS ecosystem is so rich that i do not know how to do a simpel thing :
regards
Whaouuuā¦ thank you for you answerā¦and yours projects are a plenty of usefull information .
I agree with you, i think il will mount the usb stick on a mount point on a containe, and share it with others through NFS.
I can alos answer you question, sharing this mountpoint through a Volume doesnt work ! Because ONLY the first container wich have the mount point is seeing data from the USB stickā¦others just see an empty Volume.
Thank for you help
PS : i will test it very soon because it is a working configuration, nut in a second time il will continue my searc in order to mount it on the Host OSā¦ in fact i am fearing maybe about performances issues) ā¦ did you face some performances issues on your setup ?
1/ Have you tried by first creating a subfolder in the volume and then mounting it to this subfolder ? I would at least expect that the other containers see this subfolder.
2/ Regarding performance issues. It all depends on the amount of data you want to read/write to the USB stick and the type of USB stick.