Problems running OpenBalena in LXC container under ProxMox

Hi,

I’ve been working with a colleague here for some time to run OpenBalena in an LXC container on ProxMox. There are some issues as Docker doesn’t like running in LXC but we have been making progress (Docker hello world runs and the majority of the OpenBalena containers seem to be starting)

I’m stuck on starting up the S3 container though. It’s coming back with a weird error about a tty not being present. This isn’t a problem for the other containers.

I’ve tried quite a few things including looking at user namespace remapping and setting “tty” in the docker compose file.

Would really appreciate some advice on what is going on here…

Thanks!

Alex

balena@openbalena:~/open-balena$ ./scripts/compose down
Stopping openbalena_redis_1 … done
Stopping openbalena_db_1 … done
Stopping openbalena_cert-provider_1 … done
Removing openbalena_redis_1 … done
Removing openbalena_db_1 … done
Removing openbalena_s3_1 … done
Removing openbalena_cert-provider_1 … done
Removing network openbalena_default
balena@openbalena:~/open-balena$ ./scripts/compose up -d
Creating network “openbalena_default” with the default driver
Creating openbalena_db_1 … done
Creating openbalena_cert-provider_1 … done
Creating openbalena_s3_1 … error
Creating openbalena_redis_1 … done

ERROR: for openbalena_s3_1 Cannot start service s3: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:70: creating device nodes caused \“open /var/lib/docker/vfs/dir/6db83cf82cc8e6067918ad011eb00290d4df821fc2c6a86b6005737ff738dc24/dev/tty: no such device or address\”””: unknown

ERROR: for s3 Cannot start service s3: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:70: creating device nodes caused \“open /var/lib/docker/vfs/dir/6db83cf82cc8e6067918ad011eb00290d4df821fc2c6a86b6005737ff738dc24/dev/tty: no such device or address\”””: unknown
ERROR: Encountered errors while bringing up the project.

Hello there,

As you say, Docker doesn’t like running on a non-KVM VPS, and we haven’t carried out any testing specifically for OpenBalena in these environments. There’s actually another user who’s also come across the same problem you’ve seen (https://jel.ly.fish/ee0a14da-878d-424d-a41e-9a626b2e28cb). I’ve suggested that updating to the latest version of the S3 image might make a difference.

However, it’s occurred to me that we don’t actually use S3 yet (it is intended to be used for the registry, which we’ve recently made changes for), so you should just be able to excise it from the open-balena/compose/services.yml file completely, and try with that.

The caveat here, of course, is that we are intending to move to using S3 in future, so this might mean an upgrade path in the future becomes difficult/unviable.

Best regards, Heds

That’s really helpful thanks @hedss. Will give it a go in the short term…

Same problem running up the registry container unfortunately

balena@openbalena:~/open-balena$ ./scripts/compose up -d
openbalena_cert-provider_1 is up-to-date
openbalena_db_1 is up-to-date
openbalena_redis_1 is up-to-date
Creating openbalena_registry_1 … error
Creating openbalena_api_1 …

ERROR: for openbalena_registry_1 Cannot start service registry: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "rootfs_linux.go:70: creating device nodes caused \"open /var/lib/docker/vfs/dir/eb3d17b500f9fc99b78c2f3c659025488597595960c865244Creating openbalena_api_1 … error

ERROR: for openbalena_api_1 Cannot start service api: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:70: creating device nodes caused \“open /var/lib/docker/vfs/dir/3c9344069705d3895fc07dd6d38eccca26a66543cdceeb33ef059b09a3bce586/dev/tty: no such device or address\”””: unknown

ERROR: for registry Cannot start service registry: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:70: creating device nodes caused \“open /var/lib/docker/vfs/dir/eb3d17b500f9fc99b78c2f3c659025488597595960c865244fd4ffb527735281/dev/tty: no such device or address\”””: unknown

ERROR: for api Cannot start service api: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:70: creating device nodes caused \“open /var/lib/docker/vfs/dir/3c9344069705d3895fc07dd6d38eccca26a66543cdceeb33ef059b09a3bce586/dev/tty: no such device or address\”””: unknown
ERROR: Encountered errors while bringing up the project.
balena@openbalena:~/open-balena$

Hi @ajlennon ,

Thank you very much for getting back to us!

Hmm, unfortunately I suspect this is a general issue with Docker and LXC shared kernels, then. I’ve raised an issue on OpenBalena here: https://github.com/balena-io/open-balena/issues/58 Unfortunately, I can’t give you any time frame when we might look into this, as it’s not one of our main targets and we’re trying to finalise a lot of it at the moment. However, I will discuss this further with the team.

Best regards, Heds

OK quick update - we moved over to a VM on ProxMox now and all is running well. Thanks :slight_smile:

Hi,

Great to hear that it works now!