Hey @jpk! There’s almost no overhead to speak of when running services in containers, and the pros nearly always outweigh the cons. Here’s a snippet from our docs that says it well:
As your applications grow more complex, you may find significant benefit in running some services in separate containers. Splitting your application into multiple containers allows you to better isolate and maintain key services, providing a more modular and secure approach to application management. Each service can be packaged with the operating environment and tools it specifically needs to run, and each service can be limited to the minimum system resources necessary to perform its task. The benefits of multicontainer applications compound as the complexity of the application grows. Because each service can be updated independently, larger applications can be developed and maintained by separate teams, each free to work in a way that best supports their service.
More specific to your question, keeping the individual services small and modular can significantly improve upgrade times as the diffs will be smaller or even non-existent for some services. Versus the alternative of always generating a monolithic image download for one big app container. With multiple services the startup order can also be controlled and permissions applied to only the services that require them.
Also note that our engine fork was also created with many considerations for embedded platforms:
BalenaEngine is balena’s modified Docker daemon fork that allows the management and running of application service images, containers, volumes, and networking. BalenaEngine supports container deltas for 10-70x more efficient bandwidth usage, has 3.5x smaller binaries, uses RAM and storage more conservatively, and focuses on atomicity and durability of container pulling.
Hopefully this addresses your concerns, but please reach out if you require more information!