Container in endless installing - starting - installing - stopping loop since upgrade from Resin Os 2.15.1+rev2 to balenaOS 2.27.0+rev1

Hi,

I have composite docker application : https://github.com/janvda/balena-node-red-mqtt-nginx-TIG-stack
This has worked fine but since the upgrade to the latest balena OS version I have issues with the influxdb container. It is stuck in an endless Installing - Starting - Installing - Stopping loop of the service influxdb sha256:739ccc7e7ca4d47646209f454c24254a641ad438ff74cc5f2c2ff680421dbfbc
. I have no clue why.

FYI here below you see my 2 devices (actually it are 2 SD cards that I am using on the same raspberry 3 B+ device).

Any help would be appreciated, as I am stuck at this point.
Jan

Maybe the following forum topic reports an issue with same root cause: Multicontainer project that works on RPI3 fails on Fin board (avahi & supervisor interaction)

I also get the same problem for the influxdb container with following specification in the docker compose file:

services:
  influxdb:
    #build: influxdb
    image: arm32v7/influxdb
    container_name: influxdb
    # privileged added so usb drive can be mounted.
    #privileged: true
    #volumes:
        # take care that the mount location "/mnt/influxdb" becomes overwritten
        # by the mount location specified in the influxdb Dockerfile.
    #   - 'influxdb-data:/mnt/influxdb'   
    ports:
      # The API for InfluxDB is served on port 8086
      - "8086:8086"
      - "8082:8082"
      # UDP Port
      - "8089:8089"
    #restart: always
  telegraf:
...

FYI this is what I see in the log file:

22.12.18 13:05:23 (+0100) Installing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:23 (+0100) Installed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:23 (+0100) Starting service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:26 (+0100) Started service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:27 (+0100) Killing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:29 (+0100) Service exited 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:29 (+0100) Killed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:30 (+0100) Installing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:31 (+0100) Installed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:31 (+0100) Starting service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:34 (+0100) Started service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:35 (+0100) Killing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:37 (+0100) Service exited 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:37 (+0100) Killed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:38 (+0100) Installing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:39 (+0100) Installed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'

What do you get in the logs when this happens? Have you configured both ethernet and wifi on your image?

  1. The logs you can find above. It is not clear to me what else you would need as log file.
  2. both ethernet and wifi are setup for my device.

I have tried an older version (= version 1.6) of the influxdb container:

so my content of the docker-compose.yml looks like:

...
services:
  influxdb:
    #build: influxdb
    image: arm32v7/influxdb:1.6
    container_name: influxdb
    # privileged added so usb drive can be mounted.
    #privileged: true
    #volumes:
        # take care that the mount location "/mnt/influxdb" becomes overwritten
        # by the mount location specified in the influxdb Dockerfile.
    #   - 'influxdb-data:/mnt/influxdb'   
    ports:
      # The API for InfluxDB is served on port 8086
      - "8086:8086"
      - "8082:8082"
      # UDP Port
      - "8089:8089"
    #restart: always
  telegraf:
...

this influxdb version is logging more info but has same problem (it looks like a signal is send to this container to shut it down).

24.12.18 11:08:30 (+0100)  influxdb  2018-12-24T10:08:30.657757Z        info    Listening for signals   {"log_id": "0CZvqRGl000"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.208433Z        info    Signal received, initializing clean shutdown... {"log_id": "0CZvqRGl000"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.208682Z        info    Waiting for clean shutdown...   {"log_id": "0CZvqRGl000"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209121Z        info    Listener closed {"log_id": "0CZvqRGl000", "service": "snapshot"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209603Z        info    Shutting down monitor service   {"log_id": "0CZvqRGl000", "service": "monitor"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209782Z        info    Terminating precreation service {"log_id": "0CZvqRGl000", "service": "shard-precreation"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209883Z        info    Terminating continuous query service    {"log_id": "0CZvqRGl000", "service": "continuous_querier"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.210221Z        info    Closing retention policy enforcement service    {"log_id": "0CZvqRGl000", "service": "retention"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.210494Z        info    Closed service  {"log_id": "0CZvqRGl000", "service": "subscriber"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.210864Z        info    Server shutdown completed       {"log_id": "0CZvqRGl000"}
24.12.18 11:08:38 (+0100)  influxdb
24.12.18 11:08:38 (+0100)  influxdb   8888888           .d888 888                   8888888b.  888888b.
24.12.18 11:08:38 (+0100)  influxdb     888            d88P"  888                   888  "Y88b 888  "88b
24.12.18 11:08:38 (+0100)  influxdb     888            888    888                   888    888 888  .88P
24.12.18 11:08:38 (+0100)  influxdb     888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
24.12.18 11:08:38 (+0100)  influxdb     888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
24.12.18 11:08:38 (+0100)  influxdb     888   888  888 888    888 888  888   X88K   888    888 888    888
24.12.18 11:08:38 (+0100)  influxdb     888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
24.12.18 11:08:38 (+0100)  influxdb   8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"
24.12.18 11:08:38 (+0100)  influxdb
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.318273Z        info    InfluxDB starting     {"log_id": "0CZvqugG000", "version": "1.6.4", "branch": "1.6", "commit": "a2ba6e7654fb078f8a3f5add1f8d935df38161bd"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.318515Z        info    Go runtime      {"log_id": "0CZvqugG000", "version": "go1.10.3", "maxprocs": 4}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.432304Z        info    Using data dir  {"log_id": "0CZvqugG000", "service": "store", "path": "/var/lib/influxdb/data"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.433873Z        info    Compaction settings   {"log_id": "0CZvqugG000", "service": "store", "max_concurrent_compactions": 2, "throughput_bytes_per_second": 50331648, "throughput_burst_bytes": 50331648}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.434139Z        info    Open store (start)    {"log_id": "0CZvqugG000", "service": "store", "trace_id": "0CZvqv8W000", "op_name": "tsdb_open", "op_event": "start"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.434554Z        info    Open store (end)      {"log_id": "0CZvqugG000", "service": "store", "trace_id": "0CZvqv8W000", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "0.427ms"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435561Z        info    Opened service  {"log_id": "0CZvqugG000", "service": "subscriber"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435768Z        info    Starting monitor service       {"log_id": "0CZvqugG000", "service": "monitor"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435888Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "build"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435998Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "runtime"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436112Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "network"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436251Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "system"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436645Z        info    Starting precreation service   {"log_id": "0CZvqugG000", "service": "shard-precreation", "check_interval": "10m", "advance_period": "30m"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436808Z        info    Starting snapshot service      {"log_id": "0CZvqugG000", "service": "snapshot"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436913Z        info    Starting continuous query service      {"log_id": "0CZvqugG000", "service": "continuous_querier"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437142Z        info    Starting HTTP service {"log_id": "0CZvqugG000", "service": "httpd", "authentication": false}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437259Z        info    opened HTTP access log{"log_id": "0CZvqugG000", "service": "httpd", "path": "stderr"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437686Z        info    Listening on HTTP     {"log_id": "0CZvqugG000", "service": "httpd", "addr": "[::]:8086", "https": false}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437960Z        info    Starting retention policy enforcement service  {"log_id": "0CZvqugG000", "service": "retention", "check_interval": "30m"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.439571Z        info    Listening for signals {"log_id": "0CZvqugG000"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.439159Z        info    Sending usage statistics to usage.influxdata.com       {"log_id": "0CZvqugG000"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.439560Z        info    Storing statistics    {"log_id": "0CZvqugG000", "service": "monitor", "db_instance": "_internal", "db_rp": "monitor", "interval": "10s"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.148281Z        info    Signal received, initializing clean shutdown...        {"log_id": "0CZvqugG000"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.148623Z        info    Waiting for clean shutdown...  {"log_id": "0CZvqugG000"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.148957Z        info    Listener closed {"log_id": "0CZvqugG000", "service": "snapshot"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149099Z        info    Shutting down monitor service  {"log_id": "0CZvqugG000", "service": "monitor"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149195Z        info    Terminating precreation service        {"log_id": "0CZvqugG000", "service": "shard-precreation"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149301Z        info    Terminating continuousquery service   {"log_id": "0CZvqugG000", "service": "continuous_querier"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149542Z        info    Closing retention policy enforcement service   {"log_id": "0CZvqugG000", "service": "retention"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149830Z        info    Closed service  {"log_id": "0CZvqugG000", "service": "subscriber"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149995Z        info    Server shutdown completed      {"log_id": "0CZvqugG000"}
24.12.18 11:08:46 (+0100)  influxdb
24.12.18 11:08:46 (+0100)  influxdb   8888888           .d888 888                   8888888b.  888888b.
24.12.18 11:08:46 (+0100)  influxdb     888            d88P"  888                   888  "Y88b 888  "88b
24.12.18 11:08:46 (+0100)  influxdb     888            888    888                   888    888 888  .88P
24.12.18 11:08:46 (+0100)  influxdb     888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
24.12.18 11:08:46 (+0100)  influxdb     888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
24.12.18 11:08:46 (+0100)  influxdb     888   888  888 888    888 888  888   X88K   888    888 888    888
24.12.18 11:08:46 (+0100)  influxdb     888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
24.12.18 11:08:46 (+0100)  influxdb   8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"
24.12.18 11:08:46 (+0100)  influxdb
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.604850Z        info    InfluxDB starting     {"log_id": "0CZvrQ30000", "version": "1.6.4", "branch": "1.6", "commit": "a2ba6e7654fb078f8a3f5add1f8d935df38161bd"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.605065Z        info    Go runtime      {"log_id": "0CZvrQ30000", "version": "go1.10.3", "maxprocs": 4}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.718268Z        info    Using data dir  {"log_id": "0CZvrQ30000", "service": "store", "path": "/var/lib/influxdb/data"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.719295Z        info    Compaction settings   {"log_id": "0CZvrQ30000", "service": "store", "max_concurrent_compactions": 2, "throughput_bytes_per_second": 50331648, "throughput_burst_bytes": 50331648}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.719500Z        info    Open store (start)    {"log_id": "0CZvrQ30000", "service": "store", "trace_id": "0CZvrQVl000", "op_name": "tsdb_open", "op_event": "start"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.720466Z        info    Open store (end)      {"log_id": "0CZvrQ30000", "service": "store", "trace_id": "0CZvrQVl000", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "0.982ms"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.721676Z        info    Opened service  {"log_id": "0CZvrQ30000", "service": "subscriber"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.721782Z        info    Starting monitor service       {"log_id": "0CZvrQ30000", "service": "monitor"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.722558Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "build"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.722659Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "runtime"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.723676Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "network"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.723800Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "system"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725241Z        info    Starting precreation service   {"log_id": "0CZvrQ30000", "service": "shard-precreation", "check_interval": "10m", "advance_period": "30m"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725503Z        info    Starting snapshot service      {"log_id": "0CZvrQ30000", "service": "snapshot"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725597Z        info    Starting continuous query service      {"log_id": "0CZvrQ30000", "service": "continuous_querier"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725836Z        info    Starting HTTP service {"log_id": "0CZvrQ30000", "service": "httpd", "authentication": false}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725919Z        info    opened HTTP access log{"log_id": "0CZvrQ30000", "service": "httpd", "path": "stderr"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.727122Z        info    Storing statistics    {"log_id": "0CZvrQ30000", "service": "monitor", "db_instance": "_internal", "db_rp": "monitor", "interval": "10s"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.726458Z        info    Listening on HTTP     {"log_id": "0CZvrQ30000", "service": "httpd", "addr": "[::]:8086", "https": false}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.727764Z        info    Starting retention policy enforcement service  {"log_id": "0CZvrQ30000", "service": "retention", "check_interval": "30m"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.728677Z        info    Listening for signals {"log_id": "0CZvrQ30000"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.729431Z        info    Sending usage statistics to usage.influxdata.com       {"log_id": "0CZvrQ30000"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.349773Z        info    Signal received, initializing clean shutdown...        {"log_id": "0CZvrQ30000"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350008Z        info    Waiting for clean shutdown...  {"log_id": "0CZvrQ30000"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350300Z        info    Shutting down monitor service  {"log_id": "0CZvrQ30000", "service": "monitor"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350362Z        info    Listener closed {"log_id": "0CZvrQ30000", "service": "snapshot"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350476Z        info    Terminating precreation service        {"log_id": "0CZvrQ30000", "service": "shard-precreation"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350574Z        info    Terminating continuousquery service   {"log_id": "0CZvrQ30000", "service": "continuous_querier"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350784Z        info    Closing retention policy enforcement service   {"log_id": "0CZvrQ30000", "service": "retention"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.351037Z        info    Closed service  {"log_id": "0CZvrQ30000", "service": "subscriber"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.351184Z        info    Server shutdown completed      {"log_id": "0CZvrQ30000"}
24.12.18 11:08:54 (+0100)  influxdb
24.12.18 11:08:54 (+0100)  influxdb   8888888           .d888 888                   8888888b.  888888b.
24.12.18 11:08:54 (+0100)  influxdb     888            d88P"  888                   888  "Y88b 888  "88b
24.12.18 11:08:54 (+0100)  influxdb     888            888    888                   888    888 888  .88P
24.12.18 11:08:54 (+0100)  influxdb     888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
24.12.18 11:08:54 (+0100)  influxdb     888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
24.12.18 11:08:54 (+0100)  influxdb     888   888  888 888    888 888  888   X88K   888    888 888    888
24.12.18 11:08:54 (+0100)  influxdb     888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
24.12.18 11:08:54 (+0100)  influxdb   8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"
24.12.18 11:08:54 (+0100)  influxdb

Hello,

can you try ordering your ports in docker-compose in ascending order? This seems to make my container start and is related to this.

Thanks,

Alex

Thanks for the tip.
Reordering the ports in docker-compose for the influxdb service fixed the issue (= work-around )

Note that the balena team will also fix this issue: see https://github.com/balena-io/balena-supervisor/issues/824

This has been fixed and merged here: https://github.com/balena-io/balena-supervisor/pull/862

It still needs to be merged into the OS and to undergo testing as part of that release process, but it should be part of the next balenaOS release.

1 Like