Hi. I have a problem with using my docker-compose file on Balena OS. Whereas it works well on Raspbian, I have problems with getting in running on Balena OS.
After I do “balena push” command all containers are starting fine, but after device reboot containers are stuck in restart loop. I attach all configurations and logs I was able to gather.
Diagnostics log (attached):
8b0dce9004e0a2da90be1d6993a3c021_2020.10.12_10.34.00+0000.log (788.5 KB)
Hey @ artas182x, could you please share the docker-compose with us?
Thanks,
Georgia
Hey @georgiats, here it is:
version: "2.1"
services:
redis:
privileged: false
image: redis:alpine
networks:
example_net:
ipv4_address: 10.1.0.101
command: [sh, -c, "rm -f /data/* && redis-server --save '' --appendonly no"]
database:
privileged: false
build:
context: "."
dockerfile: "./configs/Dockerfile.database"
networks:
example_net:
ipv4_address: 10.1.0.102
volumes:
- db_data:/var/lib/databaseql/data
service_1:
privileged: true
network_mode: "host"
build:
context: "."
dockerfile: "./configs/Dockerfile.service_1"
depends_on:
- database
- redis
- mosquitto
environment:
- UDEV=1
cap_add:
- NET_ADMIN
labels:
io.balena.features.dbus: 1
service_2:
privileged: false
build:
context: "."
dockerfile: "./configs/Dockerfile.service_2"
depends_on:
- service_1
networks:
example_net:
ipv4_address: 10.1.0.104
mosquitto:
privileged: false
build:
context: "."
dockerfile: "./configs/Dockerfile.mosquitto"
expose:
- "1883"
- "9001"
ports:
- "1883:1883"
- "9001:9001"
networks:
example_net:
ipv4_address: 10.1.0.103
service_3:
privileged: false
build:
context: "."
dockerfile: "./configs/Dockerfile.service_3"
depends_on:
- service_2
ports:
- 80:80
- 443:443
- 8080:8080
networks:
example_net:
ipv4_address: 10.1.0.100
networks:
example_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 10.1.0.0/24
volumes:
redis: {}
db_data: {}Hello Arthur, I had a look at the logs you sent, I see the supervisor reporting the containers being killed and restarted continuously, but I could’t gather more information from those logs regarding what was causing it. Can you attach logs from the terminal, I would be curious to see the individual service logs to see if there is something there that can shed some light on the issue.
Could you maybe also try removing individual services from the file above and checking if the restart loop is still there, it is possible the issues is with one of the services and this could help narrow down where the problem is.
One other thing, does the restart loop eventually resolve itself? From the logs I see a lot of service restarts, but it looks like they also eventually stop and the supervisor just reports being idle
@nazrhom Sometimes yes, sometimes not, sometimes manually restarting resin-supervisor helps. But when I do “balena push myapp” command it always works until reboot.
@nazrhom Here are full logs attaches. Some services are crashing, because dependent services are being killed. I will try remove some services from compose file and see what container causes this issue.
I granted support access if you need to see something: 8b0dce9004e0a2da90be1d6993a3c021
balena_containers.log (112.0 KB)
@nazrhom @georgiats I found cause of the problem. When I define networks in docker-compose then containers start crashing after reboot.
example_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 10.1.0.0/24
Do you have any recommendation to link one container working in host mode and the rest not working in host mode?
Hello Artur, i’ve used the following tricks in the past when trying to do networking with a mix of host and non-host services.
Use the following hostname depending on your case:
localhost
localhost
docker-compose.yml. Example: webserver
ip route | awk '/default / { print $3 }' to figure out the host internal IP address (there might be a better way, if so please comment :))balenaSound uses most if not all of the cases listed if you are interested in taking a look.
Hello @tmigone. Probably this solution will work for me, but when connecting host to non-host I will need to expose non-host container ports outside container. It’s not secure for me.
Hi Artur – just to be clear, currently we only support specifying network names in docker-compose network stanzas.
As for your concern – can you share a bit more information about your use case, and give an example of how you’d be exposing non-host container ports outside the container? It would be good to understand the larger goal you’re trying to accomplish.
Thanks,
Hugh
@saintaardvark I wanted to connect containers running in net=host mode with containers running in isolated network. For this I created separate network and assigned containers static IP (everything you can see in docker-compose file I attached)
However, It does not work on balena, because containers are crashing after device reboot (what you can see in this discussion). Removing network helped for this. However, I still need somehow to connect these container or get fixed case with defining own network.
Hi Artur – apologies for missing something, but can you expand a bit on how Tomas’ suggestion up above won’t work for you? You said that you would need to expose non-host container ports outside the container, and that it’s not secure for you. Can you please give details here? I want to make sure I understand your case.
Thanks,
Hugh
Hey Artur, just checking in to see if you found a workaround for this problem. If not, care to give us a few more details as requested by Hugh above? Thanks!
Hi. Sorry for no response. I deleted all network definitions from compose file and used workaround mentioned in this thread by @tmigone to find needed IP addresses.
Thanks, Artur. Did that workaround solve your problem?
John
Partially yes, but availability of adding own networks would be nice feature.