How to add persistent storage for Balena OS on Raspberry PI?

I am using balena os on Raspberry Pi.

Now I want to store the database and logs. I have multi containerized Application.

I read the documentation But I am unable to understand How can I add this in my docker-compose.yml?

Can anybody here provide a basic example for adding persistent storage for multiple application?

Hi there @sharvin26,

Permanent storage for a multicontainer application is described in our documentation here: https://www.balena.io/docs/learn/develop/multicontainer/#named-volumes

As a brief guide, you can define a named volume in your docker-compose.yml and then bind that volume to any services you define, for example:

...
volumes:
  shared-data: {}
  service-one-data: {}
  service-two-data: {}
services:
  serviceOne:
    ...
    volumes:
      - 'shared-data:/shared'
      - 'service-one-data:/private'
  serviceTwo:
    ...
    volumes:
      - 'shared-data:/shared'
      - 'service-two-data:/private'

In the example above, the ‘shared-data’ volume is permanent storage shared by both services serviceOne and serviceTwo bound to the /shared path for both. service-one-data is only available to serviceOne, bound to the /private path, and service-two-data is only available to serviceTwo, bound to the /private path. In effect, there are three permanent storage volumes, one shared by both, and two private volumes.

Hope this helps, if you still have questions, please let us know!

Best regards, Heds

I read the documentation that you shared and a discussion on the balena forum #22

It cleared some concepts but I still have some doubts. Now I have a database file that I want to persists through all container update and changes.

First Question =>

Where should I add the database to persist it? and How can I add it?

Second Question =>
What I understood from the above 2 links is that using balena volume ls I can confirm if the volume is loaded. But I am confused about How docker compose file should be constructed for that purpose?

volumes:
  Test-Database: {}
service-1:
  volumes:
    - 'Test-Database': #what should be added here?

Question 3 =>

what should I pass the database path in my application code?

Hi @Sharvin26,

The docker-compose volume binding works in exactly the same was as regular docker-compose named volumes, and the link I provided (https://www.balena.io/docs/learn/develop/multicontainer/#named-volumes) should explain how to use it.

As stated in the docs “Named volumes can be given arbitrary names and can be linked to a directory in one or more containers”. So you are linking the volume you’ve created to a named directory in your container:

volumes:
  - 'Test-Database:/path/to/persistent/database/in/service/container'

Best regards, Heds

I did that I constructed docker-compose.yml in this way =>

volumes:
  Test-Database: {}
service-1:
  - 'Test-Database': /usr/src/app/app1/Database/test.db

Note

  1. I am copying my application to /usr/src/app directory in the container.
  2. I haven’t other parameters here in the comment that is not related to this thread. But in my application, I have added all the parameters.

My Directory Tree Structure =>

app1
 - application
 - database
      - test.db
app2
docker-compose.yml

Continuously it’s only giving me installing service.

[Logs]    [6/19/2019, 3:04:27 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs]    [6/19/2019, 3:04:28 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs]    [6/19/2019, 3:04:32 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs]    [6/19/2019, 3:04:37 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs]    [6/19/2019, 3:04:46 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs]    [6/19/2019, 3:05:04 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'
[Logs]    [6/19/2019, 3:05:37 PM] Installing service 'service-1 sha256:a8ed7c00ddfad0a30d657dd07994af9a0fed08015b8843366cc93783e4e9541a'

What am I doing wrong here?

  - 'Test-Database': /usr/src/app/app1/Database/test.db

is not correct and does not conform to the docker-compose specification or the examples above. As noted in both Docker and balena documentation, everything is in quotes and is the path of a directory, not a file:

  - 'Test-Database:/usr/src/app/app1/database'

(note also that in your file tree the database directory appears to be in lower case)

I’d highly recommend you look at some of the projects in balena-io-projects, perhaps https://github.com/balena-io-projects/balena-sense/blob/master/docker-compose.yml which also uses a persistent storage volume.

Thanks for the Solution. It’s working now.

But suppose I have a file which is dynamically created when running the Application and I want to persists that How can I add that to the persistent storage in that case?

I would just move that file in the persistent /path/to/persistent/ directory.

How can I do that? Do I need to create a file using a command in dockerfile and add that file path in docker compose like this =>

  - 'Test-Database:/usr/src/app/app1/database'

Hi @Sharvin26, in the dockerfile you are linking a directory used by your service (in the container) to a persistent volume in the host OS (outside of the container). Then in your application code, what you write under this linked directory persists over application and host OS updates. You don’t need to specify a file name in the docker compose file. It’d work fine as long as you write your files under that linked directory.

In your case, when you setup the database in one of your services, you could specify that this database is written on persistent storage. Heds linked an example how this is done with influxdb.

Hope these make sense.

Thanks for the Response. I got that.

I have successfully added the database as a persistent storage but when it comes to logs I am facing issue.

I have a logging mechanism in my application but when it runs in the docker container it doesn’t write anything in the Log File.

I have confirmed this by doing ssh into the container and checking if anything is written that logfile.

Where is this file supposed to be located @Sharvin26?

Perhaps you could post some code and we could take a look. In general, being a container should not affect something so mundane as writing to a logfile.