Likely docker config issue: error creating volume with different config that already exists

owntheweb · May 4, 2020, 1:28pm

Hello Balena gurus,
I’m working up a scalable security camera solution using belanaCloud. It’s coming along well, however I still have a few quirks to work out with the camera software before proceeding much further.

While running, I’m seeing this being logged every few minutes:

Error creating volume ‘image-temp’ due to ‘Trying to create volume with name: image-temp, but a volume with that name and a different configuration already exists’

I thought I had resolved the issue, however I still see this message after posting a ‘fix’ grr. I’m thinking this has something to do with my docker compose file where I’ve created a tempfs/RAM volume (to prevent constant writes to SD card):

version: '2'
volumes:
  # This RAM drive is for raspicam to place a constant stream of images while not
  # burning out MicroSD card, pending motion detection and move to image-ready.
  image-temp:
    driver_opts:
      type: tmpfs
      device: tmpfs
      # 64MB (in bytes)
      size: 67108864
  # Images are stored here when motion is detected, ready for upload when possible.
  image-ready:
services:
  security-camera:
    build: ./security-camera
    volumes:
      - image-temp:/image-temp
      - image-ready:/image-ready
    devices:
      - /dev/vchiq:/dev/vchiq
    environment:
      - IMAGE_WIDTH
      - IMAGE_HEIGHT
      - IMAGE_QUALITY
      - IMAGE_ROTATION
      - THUMB_WIDTH
      - THUMB_HEIGHT
      - AWS_ENDPOINT
      - AWS_PRIVATE_CERT
      - AWS_ROOT_CERT
      - AWS_THING_CERT
      - AWS_REGION
      - AWS_IMAGE_BUCKET
      - MOTION_SENSITIVITY
      - MOTION_HOTSPOTS
      - DB_RECORD_TTL

Do you see any issues by chance (understanding this isn’t really a direct balena issue)? Even if I remove the - image-temp:/image-temp line, I still get this error, so maybe the issue is in the top volumes section. I don’t get an error for image-ready.

I still have a bit of learning to do when it comes to docker. Thanks for your posts that have helped boost that. Also, thanks for taking a look at this in advance!

EDIT: This could be a docker-compose versioning issue. Checking at the next break…

saintaardvark · May 4, 2020, 6:10pm

Hi @owntheweb – thanks for your post. Can you let me know what version of balenaOS you’re running? Also, I’m wondering if it’s possible that this behaviour will go away with a reboot – if you restart the device, does the problem come back?

If it does come back, are you able to connect to the Host OS of your device and run balena volumes ls? You can find info on how to do that here: https://www.balena.io/docs/learn/manage/ssh-access/.

All the best,
Hugh

owntheweb · May 4, 2020, 6:16pm

Hello @saintaardvark, thanks for the quick response.

Let’s see…

Reboot doesn’t help. It just keeps at it on reboot, restart, shutdown and start, etc. Here’s an example of what it looks like, brining up the message when everything seems to work fine (granted not sure if writing to tmpfs drive yet or not, was going to check that soon):

I’m using balenaOS 2.46.1+rev1.

I was thinking my versioning issue had to do with docker-compose.yml version listed there (currently ‘2’), may not be the case as I revisit docs. Now I’m wondering if I just have something malformed in general. If I beat you to the answer, I’ll post write away. :slight_smile:

UPDATE:
@saintaardvark Here’s the output:

root@dbbec78:~# balena volume ls
DRIVER              VOLUME NAME
local               1613664_image-ready
local               1613664_image-temp
local               1613664_resin-data

Thanks again,

owntheweb · May 4, 2020, 6:29pm

@saintaardvark Here’s a bit more info on the volume:

root@dbbec78:~# balena volume inspect 1613664_image-temp
[
    {
        "CreatedAt": "2020-05-04T18:27:46Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/1613664_image-temp/_data",
        "Name": "1613664_image-temp",
        "Options": null,
        "Scope": "local"
    }
]

saintaardvark · May 4, 2020, 6:30pm

Hi @owntheweb - I’m asking one of our balena Engine engineers to take a look at this. In the meantime, I wonder if it’s possible that the image-temp volume might have been created earlier with a different configuration (possibly as part of your development). Are you able to try removing the volume (balena volume rm 1613664_image-temp)? Keep in mind that this will lose any data that might be on the volume itself.

All the best,
Hugh

owntheweb · May 4, 2020, 6:35pm

Thanks @saintaardvark,
Ah, would that info stick/save as part of balena config on device? There was a point where I was futzing with tmpfs config on my device while working this out. The only place I would have created/altered this is in the docker-compose.yml file.

Attempting to remove is producing an error:

balena volume rm 1613664_image-temp
Error response from daemon: remove 1613664_image-temp: volume is in use - [24dde05600fcfa50d6403b6b3bf5685d2cc6b1cdda482170c240654e7ccbdd8c]

saintaardvark · May 4, 2020, 6:48pm

Hi @owntheweb – ah, if you created a volume with that name earlier, that would make sense. If you go into the dashboard for your device and stop the security-camera service, are you able to remove the volume then?

All the best,
Hugh

owntheweb · May 4, 2020, 7:10pm

Thanks again @saintaardvark,

No dice so far. It’s hanging on and restarting the app quickly when I stop it. Purging data didn’t help either. I could burn a new disk to see if that helps?

saintaardvark · May 4, 2020, 7:13pm

Hey @owntheworld – at this point, I suspect that re-flashing the device as you suggest will be the quickest way forward. If you have a second SD card, you can preserve any data that may be on the first one; alternately, you will need to manually back up any data you may want to retain before flashing it.

All the best,
Hugh

owntheweb · May 4, 2020, 10:28pm

Cool. No need to preserve anything here.

Alright, now that I have a fresh start with balenaOS 2.48.0+rev1, I’m wondering if all my volume changes are cached in balenaCloud somewhere? While the factory version was updating:

Supervisor starting
Applying configuration change {"SUPERVISOR_POLL_INTERVAL":"900000","SUPERVISOR_DELTA":"1","SUPERVISOR_DELTA_VERSION":"3"}
Applying boot config: {"avoid_warnings":"1","gpu_mem":"128","start_x":"1","disable_splash":"1","dtparam":["i2c_arm=on","spi=on","audio=on"]}
Applied boot config: {"avoid_warnings":"1","gpu_mem":"128","start_x":"1","disable_splash":"1","dtparam":["i2c_arm=on","spi=on","audio=on"]}
Applied configuration change {"SUPERVISOR_POLL_INTERVAL":"900000","SUPERVISOR_DELTA":"1","SUPERVISOR_DELTA_VERSION":"3"}
Rebooting
Supervisor starting
Creating volume 'image-ready'
Creating network 'default'
Creating volume 'image-temp'
Downloading image 'registry2.balena-cloud.com/v2/84834473719f2bdbd9e36f0cc13e9163@sha256:cccca5df6d233d59193ea42d316be132d11b78b29bec6a41df40f6a716d4c4f1'
Error creating network 'default' due to 'Trying to create network with name: default, but a network with that name and a different configuration already exists'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Creating volume 'image-temp'
Downloaded image 'registry2.balena-cloud.com/v2/84834473719f2bdbd9e36f0cc13e9163@sha256:cccca5df6d233d59193ea42d316be132d11b78b29bec6a41df40f6a716d4c4f1'
Creating volume 'image-temp'
Installing service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Installed service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Starting service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Started service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
 security-camera  
 security-camera  > security-camera@1.0.0 serve /usr/src/app
 security-camera  > node dist/server.js
 ... (good stuff)

After that, I didn’t run into any errors for hours UNTIL I click the “restart” button:

Service exited 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Killed service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Installing service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Installed service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Starting service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Started service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Error creating volume 'image-temp' due to 'Trying to create volume with name: image-temp, but a volume with that name and a different configuration already exists'
Killing service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Error creating volume 'image-temp' due to 'Trying to create volume with name: image-temp, but a volume with that name and a different configuration already exists'
 security-camera  
 security-camera  > security-camera@1.0.0 serve /usr/src/app
 security-camera  > node dist/server.js
 security-camera  
Service exited 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0'
Killed service 'security-camera sha256:3dc1553d321c94042b941e475175abbbdd1c6c67dfd26463a95e283ac9cd74f0

It sits there for about 10 minutes, eventually starting again and I now see the original error on occasion as I did before.

That’s good progress! Any ideas on how to proceed from here?

garethtdavies · May 5, 2020, 8:59pm

Hi, can you try defining the tmpfs volume using the following syntax in your docker-compose file?

volumes:
    tmp:
        driver_opts:
            type: tmpfs
            device: tmpfs
            o: "size=67108864"

That being said, we’re a little confused by the error messages you are seeing but as a first step can you try the above syntax and see if the errors persist?

owntheweb · May 5, 2020, 10:40pm

Hello @garethtdavies,

With your suggestion, I’m no longer getting errors (even with restarts and such).

Here’s the updated docker compose file for reference:

version: '2'
volumes:
  # This RAM drive is for raspicam to place a constant stream of images while not
  # burning out MicroSD card, pending motion detection and move to image-ready.
  tmp:
    driver_opts:
      type: tmpfs
      device: tmpfs
      o: "size=67108864"
  # Images are stored here when motion is detected, ready for upload when possible.
  image-ready:
services:
  security-camera:
    build: ./security-camera
    volumes:
      - 'tmp:/image-temp'
      - 'image-ready:/image-ready'
    devices:
      - /dev/vchiq:/dev/vchiq
    environment:
      - IMAGE_WIDTH
      - IMAGE_HEIGHT
      - IMAGE_QUALITY
      - IMAGE_ROTATION
      - THUMB_WIDTH
      - THUMB_HEIGHT
      - AWS_ENDPOINT
      - AWS_PRIVATE_CERT
      - AWS_ROOT_CERT
      - AWS_THING_CERT
      - AWS_REGION
      - AWS_IMAGE_BUCKET
      - MOTION_SENSITIVITY
      - MOTION_HOTSPOTS
      - DB_RECORD_TTL

Note: This also removed the 10 minute-or-so lag between restarts.

Thanks a ton for the assist! I appreciate it.

Topic		Replies	Views
Containers and tmpfs balenaOS	3	1644	November 20, 2019
Initially Populating a Volume Project help	3	386	January 7, 2021
Balena container not starting after modification in Volume balenaOS docker	13	806	July 13, 2022
Stuck devices - "error while creating mount source path: file exists" Product support	4	1314	January 29, 2019
Pi Zero Wireless: troubleshooting raspistill image capture errors in multi-container docker app balenaOS docker , docker-compose , dockerfile , raspberrypizero	2	895	April 1, 2020

Likely docker config issue: error creating volume with different config that already exists

Related topics