Container crashed?

[thomasw]

Hello,
I run the application https://github.com/jpmeijers/ttn-resin-gateway-rpi in a container on raspberry 2.
The application seems to have stopped and no more log output was written. (for more than 3 hours) But also no error was loggeds, seems just frozen.
Also in TTN console it was displayed as last seen 3hours ago, so I guess it was really stopped/blocked.
In Resin the container was displayed as running.
After restart the application was running fine again.
Is there any change to find out why the application stopps from time to time?
It does not happen in regular way, sometimes after days of runtime.
regards,
Thomas

[floion]

Hi, when this happens, can you try and fetch the supervisor logs and post them here please? This will help us debug the problem

journalctl --no-pager -u resin-supervisor

ran from the host OS webterminal would fetch you these logs.

[thomasw]

Happened again:
Application log stopped at:
29.09.18 10:45:45 (+0200) main 08:45:45 INFO: [up] TTN lora packet send to server “ttn.thingsconnected.net”

root@e349225:~# journalctl --no-pager -u resin-supervisor
– Logs begin at Wed 2018-07-18 15:08:12 UTC, end at Sat 2018-09-29 14:28:31 UTC.–
Sep 29 08:36:08 e349225 systemd[1]: Starting Resin supervisor…
Sep 29 08:36:08 e349225 balena[1146]: resin_supervisor
Sep 29 08:36:08 e349225 systemd[1]: Started Resin supervisor.
Sep 29 08:36:17 e349225 healthdog[1160]: [2018-09-29T08:36:17.824Z] Event: Supervisor start {}
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.162Z] Event: Service already running {“service”:{“appId”:1055162,“serviceId”:84093,“serviceName”:“main”,“releaseId”:592760,“image”:“sha256:c4253d89ef4f66e3c73349de010f4d77e104669df8af6b4241fcca97c4a3a325”,“labels”:{“io.resin.app-id”:“1055162”,“io.resin.architecture”:“armv7hf”,“io.resin.device-type”:“raspberry-pi2”,“io.resin.features.dbus”:“1”,“io.resin.features.firmware”:“1”,“io.resin.features.kernel-modules”:“1”,“io.resin.features.resin-api”:“1”,“io.resin.features.supervisor-api”:“1”,“io.resin.qemu.version”:“3.0.0+resin-arm”,“io.resin.service-id”:“84093”,“io.resin.service-name”:“main”,“io.resin.supervised”:“true”}}}
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.208Z] Connectivity check enabled: true
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.215Z] Starting periodic check for IP addresses
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.228Z] Reporting initial state, supervisor version and API info
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.232Z] Starting API server
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.270Z] Ensuring device is provisioned
Sep 29 08:36:18 e349225 healthdog[1160]: [2018-09-29T08:36:18.287Z] Applying target state
Sep 29 08:36:28 e349225 healthdog[1160]: [2018-09-29T08:36:28.229Z] Waiting for connectivity…
Sep 29 08:36:37 e349225 healthdog[1160]: [2018-09-29T08:36:37.902Z] Event: Devicebootstrap {}
Sep 29 08:36:38 e349225 healthdog[1160]: [2018-09-29T08:36:38.164Z] LogBackend: unexpected error: { Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:36:38 e349225 healthdog[1160]: at Object.exports._errnoException (util.js:1020:11)
Sep 29 08:36:38 e349225 healthdog[1160]: at errnoException (dns.js:33:15)
Sep 29 08:36:38 e349225 healthdog[1160]: at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
Sep 29 08:36:38 e349225 healthdog[1160]: code: ‘EAI_AGAIN’,
Sep 29 08:36:38 e349225 healthdog[1160]: errno: ‘EAI_AGAIN’,
Sep 29 08:36:38 e349225 healthdog[1160]: syscall: ‘getaddrinfo’,
Sep 29 08:36:38 e349225 healthdog[1160]: hostname: ‘api.resin.io’,
Sep 29 08:36:38 e349225 healthdog[1160]: host: ‘api.resin.io’,
Sep 29 08:36:38 e349225 healthdog[1160]: port: 443 }
Sep 29 08:36:38 e349225 healthdog[1160]: [2018-09-29T08:36:38.319Z] Starting current state report
Sep 29 08:36:38 e349225 healthdog[1160]: [2018-09-29T08:36:38.331Z] Starting target state poll
Sep 29 08:36:58 e349225 healthdog[1160]: [2018-09-29T08:36:58.021Z] Finished applying target state
Sep 29 08:36:58 e349225 healthdog[1160]: [2018-09-29T08:36:58.023Z] Apply success!
Sep 29 08:36:58 e349225 healthdog[1160]: [2018-09-29T08:36:58.403Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:36:58 e349225 healthdog[1160]: [2018-09-29T08:36:58.558Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:37:19 e349225 healthdog[1160]: [2018-09-29T08:37:19.663Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:37:33 e349225 healthdog[1160]: [2018-09-29T08:37:33.463Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:37:41 e349225 healthdog[1160]: [2018-09-29T08:37:41.789Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:38:05 e349225 healthdog[1160]: [2018-09-29T08:38:05.918Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:38:23 e349225 healthdog[1160]: [2018-09-29T08:38:23.518Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:38:33 e349225 healthdog[1160]: [2018-09-29T08:38:33.995Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:39:10 e349225 healthdog[1160]: [2018-09-29T08:39:10.105Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:39:43 e349225 healthdog[1160]: [2018-09-29T08:39:43.553Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:40:02 e349225 healthdog[1160]: [2018-09-29T08:40:02.237Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:41:22 e349225 healthdog[1160]: [2018-09-29T08:41:22.321Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:42:03 e349225 healthdog[1160]: [2018-09-29T08:42:03.587Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:42:42 e349225 healthdog[1160]: [2018-09-29T08:42:42.408Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:44:02 e349225 healthdog[1160]: [2018-09-29T08:44:02.480Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:45:22 e349225 healthdog[1160]: [2018-09-29T08:45:22.575Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:46:23 e349225 healthdog[1160]: [2018-09-29T08:46:23.620Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:46:42 e349225 healthdog[1160]: [2018-09-29T08:46:42.648Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:48:02 e349225 healthdog[1160]: [2018-09-29T08:48:02.744Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:49:22 e349225 healthdog[1160]: [2018-09-29T08:49:22.829Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:50:42 e349225 healthdog[1160]: [2018-09-29T08:50:42.921Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:52:03 e349225 healthdog[1160]: [2018-09-29T08:52:03.003Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:53:23 e349225 healthdog[1160]: [2018-09-29T08:53:23.090Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:54:43 e349225 healthdog[1160]: [2018-09-29T08:54:43.179Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:54:43 e349225 healthdog[1160]: [2018-09-29T08:54:43.656Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 08:56:03 e349225 healthdog[1160]: [2018-09-29T08:56:03.262Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:57:23 e349225 healthdog[1160]: [2018-09-29T08:57:23.377Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 08:58:43 e349225 healthdog[1160]: [2018-09-29T08:58:43.451Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:00:03 e349225 healthdog[1160]: [2018-09-29T09:00:03.527Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:01:23 e349225 healthdog[1160]: [2018-09-29T09:01:23.601Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:02:43 e349225 healthdog[1160]: [2018-09-29T09:02:43.685Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:04:03 e349225 healthdog[1160]: [2018-09-29T09:04:03.783Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:05:03 e349225 healthdog[1160]: [2018-09-29T09:05:03.687Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 09:05:23 e349225 healthdog[1160]: [2018-09-29T09:05:23.885Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:06:44 e349225 healthdog[1160]: [2018-09-29T09:06:44.010Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:08:04 e349225 healthdog[1160]: [2018-09-29T09:08:04.078Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:09:24 e349225 healthdog[1160]: [2018-09-29T09:09:24.154Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:10:44 e349225 healthdog[1160]: [2018-09-29T09:10:44.230Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:12:04 e349225 healthdog[1160]: [2018-09-29T09:12:04.310Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:13:24 e349225 healthdog[1160]: [2018-09-29T09:13:24.377Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:14:44 e349225 healthdog[1160]: [2018-09-29T09:14:44.453Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:15:23 e349225 healthdog[1160]: [2018-09-29T09:15:23.720Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 09:16:04 e349225 healthdog[1160]: [2018-09-29T09:16:04.530Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:17:24 e349225 healthdog[1160]: [2018-09-29T09:17:24.608Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:18:44 e349225 healthdog[1160]: [2018-09-29T09:18:44.698Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:20:04 e349225 healthdog[1160]: [2018-09-29T09:20:04.778Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:21:24 e349225 healthdog[1160]: [2018-09-29T09:21:24.868Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:22:44 e349225 healthdog[1160]: [2018-09-29T09:22:44.956Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:24:05 e349225 healthdog[1160]: [2018-09-29T09:24:05.045Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:25:25 e349225 healthdog[1160]: [2018-09-29T09:25:25.121Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:25:43 e349225 healthdog[1160]: [2018-09-29T09:25:43.753Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 09:26:45 e349225 healthdog[1160]: [2018-09-29T09:26:45.203Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:28:05 e349225 healthdog[1160]: [2018-09-29T09:28:05.303Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:29:25 e349225 healthdog[1160]: [2018-09-29T09:29:25.371Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:30:45 e349225 healthdog[1160]: [2018-09-29T09:30:45.440Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:32:05 e349225 healthdog[1160]: [2018-09-29T09:32:05.501Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:33:25 e349225 healthdog[1160]: [2018-09-29T09:33:25.571Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:34:45 e349225 healthdog[1160]: [2018-09-29T09:34:45.648Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:36:03 e349225 healthdog[1160]: [2018-09-29T09:36:03.784Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 09:36:05 e349225 healthdog[1160]: [2018-09-29T09:36:05.712Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:37:25 e349225 healthdog[1160]: [2018-09-29T09:37:25.778Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:38:45 e349225 healthdog[1160]: [2018-09-29T09:38:45.844Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:40:05 e349225 healthdog[1160]: [2018-09-29T09:40:05.964Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:41:26 e349225 healthdog[1160]: [2018-09-29T09:41:26.081Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:42:46 e349225 healthdog[1160]: [2018-09-29T09:42:46.177Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:44:06 e349225 healthdog[1160]: [2018-09-29T09:44:06.254Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:45:26 e349225 healthdog[1160]: [2018-09-29T09:45:26.330Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:46:23 e349225 healthdog[1160]: [2018-09-29T09:46:23.821Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 09:46:46 e349225 healthdog[1160]: [2018-09-29T09:46:46.398Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:48:06 e349225 healthdog[1160]: [2018-09-29T09:48:06.472Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:49:26 e349225 healthdog[1160]: [2018-09-29T09:49:26.561Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:50:46 e349225 healthdog[1160]: [2018-09-29T09:50:46.657Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:52:06 e349225 healthdog[1160]: [2018-09-29T09:52:06.754Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:53:26 e349225 healthdog[1160]: [2018-09-29T09:53:26.841Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:54:46 e349225 healthdog[1160]: [2018-09-29T09:54:46.910Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:56:07 e349225 healthdog[1160]: [2018-09-29T09:56:07.009Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:56:43 e349225 healthdog[1160]: [2018-09-29T09:56:43.856Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 09:57:27 e349225 healthdog[1160]: [2018-09-29T09:57:27.092Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 09:58:47 e349225 healthdog[1160]: [2018-09-29T09:58:47.174Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:00:07 e349225 healthdog[1160]: [2018-09-29T10:00:07.252Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:01:27 e349225 healthdog[1160]: [2018-09-29T10:01:27.351Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:02:47 e349225 healthdog[1160]: [2018-09-29T10:02:47.441Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:04:07 e349225 healthdog[1160]: [2018-09-29T10:04:07.516Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:05:27 e349225 healthdog[1160]: [2018-09-29T10:05:27.587Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:06:47 e349225 healthdog[1160]: [2018-09-29T10:06:47.658Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:07:03 e349225 healthdog[1160]: [2018-09-29T10:07:03.891Z] Failed to gettarget state for device: Error: getaddrinfo EAI_AGAIN api.resin.io:443
Sep 29 10:08:07 e349225 healthdog[1160]: [2018-09-29T10:08:07.734Z] Event: Devicestate report failure {“error”:{“message”:""}}
Sep 29 10:08:39 e349225 healthdog[1160]: [2018-09-29T10:08:39.338Z] Internet Connectivity: OK
Sep 29 10:17:05 e349225 healthdog[1160]: [2018-09-29T10:17:05.012Z] Applying target state
Sep 29 10:17:05 e349225 healthdog[1160]: [2018-09-29T10:17:05.587Z] Finished applying target state
Sep 29 10:17:05 e349225 healthdog[1160]: [2018-09-29T10:17:05.589Z] Apply success!

[thomasw]

Any update regarding to my log?

[xginn8]

Hi @thomasw,

From the looks of it, this issue seems to be related to DNS lookup failures (EAI_AGAIN is a generic DNS failure response, see this manpage). Is your device fully inaccessible from the web UI during these periods? Would you mind detailing a bit about the network topology (connected via WiFi or ethernet)? How are you able to restart the container?

Thank you!