Container restart loop problem

Product support balenaOS
#1

Hi. I have a problem with using my docker-compose file on Balena OS. Whereas it works well on Raspbian, I have problems with getting in running on Balena OS.

After I do “balena push” command all containers are starting fine, but after device reboot containers are stuck in restart loop. I attach all configurations and logs I was able to gather.

Diagnostics log (attached):

8b0dce9004e0a2da90be1d6993a3c021_2020.10.12_10.34.00+0000.log (788.5 KB)

#5

Hey @ artas182x, could you please share the docker-compose with us?
Thanks,
Georgia

#6

Hey @georgiats, here it is:

version: "2.1"

services:

  redis:
    privileged: false
    image: redis:alpine
    networks:
      example_net:
        ipv4_address: 10.1.0.101
    command: [sh, -c, "rm -f /data/* && redis-server --save '' --appendonly no"]

  database:
    privileged: false
    build:
      context: "."
      dockerfile: "./configs/Dockerfile.database"
    networks:
      example_net:
        ipv4_address: 10.1.0.102
    volumes:
      - db_data:/var/lib/databaseql/data

  service_1:
    privileged: true
    network_mode: "host"
    build:
      context: "."
      dockerfile: "./configs/Dockerfile.service_1"
    depends_on:
      - database
      - redis
      - mosquitto
    environment:
      - UDEV=1
    cap_add:
      - NET_ADMIN
    labels:
      io.balena.features.dbus: 1

  service_2:
    privileged: false
    build:
      context: "."
      dockerfile: "./configs/Dockerfile.service_2"
    depends_on:
      - service_1
    networks:
      example_net:
        ipv4_address: 10.1.0.104

  mosquitto:
    privileged: false
    build:
      context: "."
      dockerfile: "./configs/Dockerfile.mosquitto"
    expose:
      - "1883"
      - "9001"
    ports:
      - "1883:1883"
      - "9001:9001"
    networks:
      example_net:
        ipv4_address: 10.1.0.103

  service_3:
    privileged: false
    build:
      context: "."
      dockerfile: "./configs/Dockerfile.service_3"
    depends_on: 
      - service_2
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    networks:
      example_net:
        ipv4_address: 10.1.0.100

networks:
  example_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.1.0.0/24

volumes:
  redis: {}
  db_data: {}
#7

Hello Arthur, I had a look at the logs you sent, I see the supervisor reporting the containers being killed and restarted continuously, but I could’t gather more information from those logs regarding what was causing it. Can you attach logs from the terminal, I would be curious to see the individual service logs to see if there is something there that can shed some light on the issue.
Could you maybe also try removing individual services from the file above and checking if the restart loop is still there, it is possible the issues is with one of the services and this could help narrow down where the problem is.

#8

One other thing, does the restart loop eventually resolve itself? From the logs I see a lot of service restarts, but it looks like they also eventually stop and the supervisor just reports being idle

#10

@nazrhom Sometimes yes, sometimes not, sometimes manually restarting resin-supervisor helps. But when I do “balena push myapp” command it always works until reboot.

#11

@nazrhom Here are full logs attaches. Some services are crashing, because dependent services are being killed. I will try remove some services from compose file and see what container causes this issue.

I granted support access if you need to see something: 8b0dce9004e0a2da90be1d6993a3c021

balena_containers.log (112.0 KB)

#13

@nazrhom @georgiats I found cause of the problem. When I define networks in docker-compose then containers start crashing after reboot.

  example_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.1.0.0/24

Do you have any recommendation to link one container working in host mode and the rest not working in host mode?

#14

Hello Artur, i’ve used the following tricks in the past when trying to do networking with a mix of host and non-host services.

Use the following hostname depending on your case:

  • Host to host: use localhost
  • Host to non-host: use localhost
  • Non-host to non-host: use service name as defined in docker-compose.yml. Example: webserver
  • Non-host to host: run ip route | awk '/default / { print $3 }' to figure out the host internal IP address (there might be a better way, if so please comment :))

balenaSound uses most if not all of the cases listed if you are interested in taking a look.

#15

Hello @tmigone. Probably this solution will work for me, but when connecting host to non-host I will need to expose non-host container ports outside container. It’s not secure for me.

#16

Hi Artur – just to be clear, currently we only support specifying network names in docker-compose network stanzas.

As for your concern – can you share a bit more information about your use case, and give an example of how you’d be exposing non-host container ports outside the container? It would be good to understand the larger goal you’re trying to accomplish.

Thanks,
Hugh

#18

@saintaardvark I wanted to connect containers running in net=host mode with containers running in isolated network. For this I created separate network and assigned containers static IP (everything you can see in docker-compose file I attached)

However, It does not work on balena, because containers are crashing after device reboot (what you can see in this discussion). Removing network helped for this. However, I still need somehow to connect these container or get fixed case with defining own network.

#19

Hi Artur – apologies for missing something, but can you expand a bit on how Tomas’ suggestion up above won’t work for you? You said that you would need to expose non-host container ports outside the container, and that it’s not secure for you. Can you please give details here? I want to make sure I understand your case.

Thanks,
Hugh

#21

Hey Artur, just checking in to see if you found a workaround for this problem. If not, care to give us a few more details as requested by Hugh above? Thanks!

#23

Hi. Sorry for no response. I deleted all network definitions from compose file and used workaround mentioned in this thread by @tmigone to find needed IP addresses.

#24

Thanks, Artur. Did that workaround solve your problem?

John

#26

Partially yes, but availability of adding own networks would be nice feature.