Container in endless installing - starting - installing - stopping loop since upgrade from Resin Os 2.15.1+rev2 to balenaOS 2.27.0+rev1

raspberrypi3

#1

Hi,

I have composite docker application : https://github.com/janvda/balena-node-red-mqtt-nginx-TIG-stack
This has worked fine but since the upgrade to the latest balena OS version I have issues with the influxdb container. It is stuck in an endless Installing - Starting - Installing - Stopping loop of the service influxdb sha256:739ccc7e7ca4d47646209f454c24254a641ad438ff74cc5f2c2ff680421dbfbc
. I have no clue why.

FYI here below you see my 2 devices (actually it are 2 SD cards that I am using on the same raspberry 3 B+ device).

Any help would be appreciated, as I am stuck at this point.
Jan


Multicontainer project that works on RPI3 fails on Fin board (avahi & supervisor interaction)
ODROID bigLittle only uses 1/3 processing power
#2

Maybe the following forum topic reports an issue with same root cause: Multicontainer project that works on RPI3 fails on Fin board (avahi & supervisor interaction)


#3

I also get the same problem for the influxdb container with following specification in the docker compose file:

services:
  influxdb:
    #build: influxdb
    image: arm32v7/influxdb
    container_name: influxdb
    # privileged added so usb drive can be mounted.
    #privileged: true
    #volumes:
        # take care that the mount location "/mnt/influxdb" becomes overwritten
        # by the mount location specified in the influxdb Dockerfile.
    #   - 'influxdb-data:/mnt/influxdb'   
    ports:
      # The API for InfluxDB is served on port 8086
      - "8086:8086"
      - "8082:8082"
      # UDP Port
      - "8089:8089"
    #restart: always
  telegraf:
...

FYI this is what I see in the log file:

22.12.18 13:05:23 (+0100) Installing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:23 (+0100) Installed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:23 (+0100) Starting service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:26 (+0100) Started service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:27 (+0100) Killing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:29 (+0100) Service exited 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:29 (+0100) Killed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:30 (+0100) Installing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:31 (+0100) Installed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:31 (+0100) Starting service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:34 (+0100) Started service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:35 (+0100) Killing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:37 (+0100) Service exited 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:37 (+0100) Killed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:38 (+0100) Installing service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'
22.12.18 13:05:39 (+0100) Installed service 'influxdb sha256:7fe7c85ef99dc0c5711fa6f87259ea0b66c34e9702eedf00c05b680b4ff4b099'

#4

What do you get in the logs when this happens? Have you configured both ethernet and wifi on your image?


#5
  1. The logs you can find above. It is not clear to me what else you would need as log file.
  2. both ethernet and wifi are setup for my device.

#6

I have tried an older version (= version 1.6) of the influxdb container:

so my content of the docker-compose.yml looks like:

...
services:
  influxdb:
    #build: influxdb
    image: arm32v7/influxdb:1.6
    container_name: influxdb
    # privileged added so usb drive can be mounted.
    #privileged: true
    #volumes:
        # take care that the mount location "/mnt/influxdb" becomes overwritten
        # by the mount location specified in the influxdb Dockerfile.
    #   - 'influxdb-data:/mnt/influxdb'   
    ports:
      # The API for InfluxDB is served on port 8086
      - "8086:8086"
      - "8082:8082"
      # UDP Port
      - "8089:8089"
    #restart: always
  telegraf:
...

this influxdb version is logging more info but has same problem (it looks like a signal is send to this container to shut it down).

24.12.18 11:08:30 (+0100)  influxdb  2018-12-24T10:08:30.657757Z        info    Listening for signals   {"log_id": "0CZvqRGl000"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.208433Z        info    Signal received, initializing clean shutdown... {"log_id": "0CZvqRGl000"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.208682Z        info    Waiting for clean shutdown...   {"log_id": "0CZvqRGl000"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209121Z        info    Listener closed {"log_id": "0CZvqRGl000", "service": "snapshot"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209603Z        info    Shutting down monitor service   {"log_id": "0CZvqRGl000", "service": "monitor"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209782Z        info    Terminating precreation service {"log_id": "0CZvqRGl000", "service": "shard-precreation"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.209883Z        info    Terminating continuous query service    {"log_id": "0CZvqRGl000", "service": "continuous_querier"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.210221Z        info    Closing retention policy enforcement service    {"log_id": "0CZvqRGl000", "service": "retention"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.210494Z        info    Closed service  {"log_id": "0CZvqRGl000", "service": "subscriber"}
24.12.18 11:08:31 (+0100)  influxdb  2018-12-24T10:08:31.210864Z        info    Server shutdown completed       {"log_id": "0CZvqRGl000"}
24.12.18 11:08:38 (+0100)  influxdb
24.12.18 11:08:38 (+0100)  influxdb   8888888           .d888 888                   8888888b.  888888b.
24.12.18 11:08:38 (+0100)  influxdb     888            d88P"  888                   888  "Y88b 888  "88b
24.12.18 11:08:38 (+0100)  influxdb     888            888    888                   888    888 888  .88P
24.12.18 11:08:38 (+0100)  influxdb     888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
24.12.18 11:08:38 (+0100)  influxdb     888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
24.12.18 11:08:38 (+0100)  influxdb     888   888  888 888    888 888  888   X88K   888    888 888    888
24.12.18 11:08:38 (+0100)  influxdb     888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
24.12.18 11:08:38 (+0100)  influxdb   8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"
24.12.18 11:08:38 (+0100)  influxdb
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.318273Z        info    InfluxDB starting     {"log_id": "0CZvqugG000", "version": "1.6.4", "branch": "1.6", "commit": "a2ba6e7654fb078f8a3f5add1f8d935df38161bd"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.318515Z        info    Go runtime      {"log_id": "0CZvqugG000", "version": "go1.10.3", "maxprocs": 4}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.432304Z        info    Using data dir  {"log_id": "0CZvqugG000", "service": "store", "path": "/var/lib/influxdb/data"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.433873Z        info    Compaction settings   {"log_id": "0CZvqugG000", "service": "store", "max_concurrent_compactions": 2, "throughput_bytes_per_second": 50331648, "throughput_burst_bytes": 50331648}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.434139Z        info    Open store (start)    {"log_id": "0CZvqugG000", "service": "store", "trace_id": "0CZvqv8W000", "op_name": "tsdb_open", "op_event": "start"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.434554Z        info    Open store (end)      {"log_id": "0CZvqugG000", "service": "store", "trace_id": "0CZvqv8W000", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "0.427ms"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435561Z        info    Opened service  {"log_id": "0CZvqugG000", "service": "subscriber"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435768Z        info    Starting monitor service       {"log_id": "0CZvqugG000", "service": "monitor"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435888Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "build"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.435998Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "runtime"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436112Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "network"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436251Z        info    Registered diagnosticsclient  {"log_id": "0CZvqugG000", "service": "monitor", "name": "system"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436645Z        info    Starting precreation service   {"log_id": "0CZvqugG000", "service": "shard-precreation", "check_interval": "10m", "advance_period": "30m"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436808Z        info    Starting snapshot service      {"log_id": "0CZvqugG000", "service": "snapshot"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.436913Z        info    Starting continuous query service      {"log_id": "0CZvqugG000", "service": "continuous_querier"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437142Z        info    Starting HTTP service {"log_id": "0CZvqugG000", "service": "httpd", "authentication": false}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437259Z        info    opened HTTP access log{"log_id": "0CZvqugG000", "service": "httpd", "path": "stderr"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437686Z        info    Listening on HTTP     {"log_id": "0CZvqugG000", "service": "httpd", "addr": "[::]:8086", "https": false}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.437960Z        info    Starting retention policy enforcement service  {"log_id": "0CZvqugG000", "service": "retention", "check_interval": "30m"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.439571Z        info    Listening for signals {"log_id": "0CZvqugG000"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.439159Z        info    Sending usage statistics to usage.influxdata.com       {"log_id": "0CZvqugG000"}
24.12.18 11:08:38 (+0100)  influxdb  2018-12-24T10:08:38.439560Z        info    Storing statistics    {"log_id": "0CZvqugG000", "service": "monitor", "db_instance": "_internal", "db_rp": "monitor", "interval": "10s"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.148281Z        info    Signal received, initializing clean shutdown...        {"log_id": "0CZvqugG000"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.148623Z        info    Waiting for clean shutdown...  {"log_id": "0CZvqugG000"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.148957Z        info    Listener closed {"log_id": "0CZvqugG000", "service": "snapshot"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149099Z        info    Shutting down monitor service  {"log_id": "0CZvqugG000", "service": "monitor"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149195Z        info    Terminating precreation service        {"log_id": "0CZvqugG000", "service": "shard-precreation"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149301Z        info    Terminating continuousquery service   {"log_id": "0CZvqugG000", "service": "continuous_querier"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149542Z        info    Closing retention policy enforcement service   {"log_id": "0CZvqugG000", "service": "retention"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149830Z        info    Closed service  {"log_id": "0CZvqugG000", "service": "subscriber"}
24.12.18 11:08:39 (+0100)  influxdb  2018-12-24T10:08:39.149995Z        info    Server shutdown completed      {"log_id": "0CZvqugG000"}
24.12.18 11:08:46 (+0100)  influxdb
24.12.18 11:08:46 (+0100)  influxdb   8888888           .d888 888                   8888888b.  888888b.
24.12.18 11:08:46 (+0100)  influxdb     888            d88P"  888                   888  "Y88b 888  "88b
24.12.18 11:08:46 (+0100)  influxdb     888            888    888                   888    888 888  .88P
24.12.18 11:08:46 (+0100)  influxdb     888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
24.12.18 11:08:46 (+0100)  influxdb     888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
24.12.18 11:08:46 (+0100)  influxdb     888   888  888 888    888 888  888   X88K   888    888 888    888
24.12.18 11:08:46 (+0100)  influxdb     888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
24.12.18 11:08:46 (+0100)  influxdb   8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"
24.12.18 11:08:46 (+0100)  influxdb
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.604850Z        info    InfluxDB starting     {"log_id": "0CZvrQ30000", "version": "1.6.4", "branch": "1.6", "commit": "a2ba6e7654fb078f8a3f5add1f8d935df38161bd"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.605065Z        info    Go runtime      {"log_id": "0CZvrQ30000", "version": "go1.10.3", "maxprocs": 4}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.718268Z        info    Using data dir  {"log_id": "0CZvrQ30000", "service": "store", "path": "/var/lib/influxdb/data"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.719295Z        info    Compaction settings   {"log_id": "0CZvrQ30000", "service": "store", "max_concurrent_compactions": 2, "throughput_bytes_per_second": 50331648, "throughput_burst_bytes": 50331648}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.719500Z        info    Open store (start)    {"log_id": "0CZvrQ30000", "service": "store", "trace_id": "0CZvrQVl000", "op_name": "tsdb_open", "op_event": "start"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.720466Z        info    Open store (end)      {"log_id": "0CZvrQ30000", "service": "store", "trace_id": "0CZvrQVl000", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "0.982ms"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.721676Z        info    Opened service  {"log_id": "0CZvrQ30000", "service": "subscriber"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.721782Z        info    Starting monitor service       {"log_id": "0CZvrQ30000", "service": "monitor"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.722558Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "build"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.722659Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "runtime"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.723676Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "network"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.723800Z        info    Registered diagnosticsclient  {"log_id": "0CZvrQ30000", "service": "monitor", "name": "system"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725241Z        info    Starting precreation service   {"log_id": "0CZvrQ30000", "service": "shard-precreation", "check_interval": "10m", "advance_period": "30m"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725503Z        info    Starting snapshot service      {"log_id": "0CZvrQ30000", "service": "snapshot"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725597Z        info    Starting continuous query service      {"log_id": "0CZvrQ30000", "service": "continuous_querier"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725836Z        info    Starting HTTP service {"log_id": "0CZvrQ30000", "service": "httpd", "authentication": false}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.725919Z        info    opened HTTP access log{"log_id": "0CZvrQ30000", "service": "httpd", "path": "stderr"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.727122Z        info    Storing statistics    {"log_id": "0CZvrQ30000", "service": "monitor", "db_instance": "_internal", "db_rp": "monitor", "interval": "10s"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.726458Z        info    Listening on HTTP     {"log_id": "0CZvrQ30000", "service": "httpd", "addr": "[::]:8086", "https": false}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.727764Z        info    Starting retention policy enforcement service  {"log_id": "0CZvrQ30000", "service": "retention", "check_interval": "30m"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.728677Z        info    Listening for signals {"log_id": "0CZvrQ30000"}
24.12.18 11:08:46 (+0100)  influxdb  2018-12-24T10:08:46.729431Z        info    Sending usage statistics to usage.influxdata.com       {"log_id": "0CZvrQ30000"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.349773Z        info    Signal received, initializing clean shutdown...        {"log_id": "0CZvrQ30000"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350008Z        info    Waiting for clean shutdown...  {"log_id": "0CZvrQ30000"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350300Z        info    Shutting down monitor service  {"log_id": "0CZvrQ30000", "service": "monitor"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350362Z        info    Listener closed {"log_id": "0CZvrQ30000", "service": "snapshot"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350476Z        info    Terminating precreation service        {"log_id": "0CZvrQ30000", "service": "shard-precreation"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350574Z        info    Terminating continuousquery service   {"log_id": "0CZvrQ30000", "service": "continuous_querier"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.350784Z        info    Closing retention policy enforcement service   {"log_id": "0CZvrQ30000", "service": "retention"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.351037Z        info    Closed service  {"log_id": "0CZvrQ30000", "service": "subscriber"}
24.12.18 11:08:47 (+0100)  influxdb  2018-12-24T10:08:47.351184Z        info    Server shutdown completed      {"log_id": "0CZvrQ30000"}
24.12.18 11:08:54 (+0100)  influxdb
24.12.18 11:08:54 (+0100)  influxdb   8888888           .d888 888                   8888888b.  888888b.
24.12.18 11:08:54 (+0100)  influxdb     888            d88P"  888                   888  "Y88b 888  "88b
24.12.18 11:08:54 (+0100)  influxdb     888            888    888                   888    888 888  .88P
24.12.18 11:08:54 (+0100)  influxdb     888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
24.12.18 11:08:54 (+0100)  influxdb     888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
24.12.18 11:08:54 (+0100)  influxdb     888   888  888 888    888 888  888   X88K   888    888 888    888
24.12.18 11:08:54 (+0100)  influxdb     888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
24.12.18 11:08:54 (+0100)  influxdb   8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"
24.12.18 11:08:54 (+0100)  influxdb

#7

Hello,

can you try ordering your ports in docker-compose in ascending order? This seems to make my container start and is related to this.

Thanks,

Alex


#8

Thanks for the tip.
Reordering the ports in docker-compose for the influxdb service fixed the issue (= work-around )

Note that the balena team will also fix this issue: see https://github.com/balena-io/balena-supervisor/issues/824


#9

This has been fixed and merged here: https://github.com/balena-io/balena-supervisor/pull/862

It still needs to be merged into the OS and to undergo testing as part of that release process, but it should be part of the next balenaOS release.