I have tried to purge the data on my Up2 but it does not seem to clear /data.
I am using baleanOS 2.39.0+rev1 (supervisor 10.0.3)
When I run purge data on a device it stops the services, logs that it has purged the data but when the services restart the data is still in /data. Is this a known issue?
I have two containers running, each with their own volume and neither is purged
That sounds weird, what does journalctl -afn 200 -u resin-supervisor show after you issued a Purge if you SSH into the host from the Dashboard terminal?
Here is the output
root@7409930:~# journalctl -afn 200 -u resin-supervisor
-- Logs begin at Mon 2019-09-09 16:35:55 UTC. --
Sep 09 16:36:19 7409930 systemd[1]: Starting Resin supervisor...
Sep 09 16:36:19 7409930 resin-supervisor[2092]: resin_supervisor
Sep 09 16:36:19 7409930 resin-supervisor[2099]: active
Sep 09 16:36:19 7409930 systemd[1]: Started Resin supervisor.
Sep 09 16:36:20 7409930 resin-supervisor[2100]: Container config has not changed
Sep 09 16:36:20 7409930 resin-supervisor[2100]: Starting system message bus: dbus.
Sep 09 16:36:20 7409930 resin-supervisor[2100]: * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon
Sep 09 16:36:21 7409930 resin-supervisor[2100]: ...done.
Sep 09 16:36:23 7409930 resin-supervisor[2100]: [info] Supervisor v10.0.3 starting up...
Sep 09 16:36:23 7409930 resin-supervisor[2100]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Starting event tracker
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Starting api binder
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Starting logging infrastructure
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [event] Event: Supervisor start {}
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Performing database cleanup for container log timestamps
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [info] Previous engine snapshot was not stored. Skipping cleanup.
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Handling of local mode switch is completed
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Connectivity check enabled: true
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Starting periodic check for IP addresses
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [info] Reporting initial state, supervisor version and API info
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] VPN status path exists.
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [info] Waiting for connectivity...
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Skipping preloading
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [info] Starting API server
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [info] Applying target state
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Ensuring device is provisioned
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Starting current state report
Sep 09 16:36:24 7409930 resin-supervisor[2100]: [debug] Starting target state poll
Sep 09 16:36:25 7409930 resin-supervisor[2100]: [debug] Supervisor API listening on allowed interfaces only
Sep 09 16:36:25 7409930 resin-supervisor[2100]: [debug] Finished applying target state
Sep 09 16:36:25 7409930 resin-supervisor[2100]: [success] Device state apply success
Sep 09 16:36:25 7409930 resin-supervisor[2100]: [info] Applying target state
Sep 09 16:36:25 7409930 resin-supervisor[2100]: [debug] Finished applying target state
Sep 09 16:36:25 7409930 resin-supervisor[2100]: [success] Device state apply success
Sep 09 16:36:34 7409930 resin-supervisor[2100]: [info] Internet Connectivity: OK
Sep 09 16:38:55 7409930 resin-supervisor[2100]: [event] Event: Update notification {}
Sep 09 16:38:55 7409930 resin-supervisor[2100]: [api] POST /v1/update 204 - 24.400 ms
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [info] Applying target state
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/7db9612528a88b6a996fee702c775428@sha256:1b0910b415f7670f2c1490333de9b72a872a66f621af55946243ea7cfcb23bb7","workingDir":""}}
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/7db9612528a88b6a996fee702c775428@sha256:1b0910b415f7670f2c1490333de9b72a872a66f621af55946243ea7cfcb23bb7","workingDir":""}}
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:38:56 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/7db9612528a88b6a996fee702c775428@sha256:1b0910b415f7670f2c1490333de9b72a872a66f621af55946243ea7cfcb23bb7","workingDir":""}}
Sep 09 16:38:57 7409930 resin-supervisor[2100]: [event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/7db9612528a88b6a996fee702c775428@sha256:1b0910b415f7670f2c1490333de9b72a872a66f621af55946243ea7cfcb23bb7","appId":1507282,"serviceId":318092,"serviceName":"mobius","imageId":1483271,"releaseId":1059344,"dependent":0,"dockerImageId":null}}
Sep 09 16:39:35 7409930 resin-supervisor[2100]: [event] Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/7db9612528a88b6a996fee702c775428@sha256:1b0910b415f7670f2c1490333de9b72a872a66f621af55946243ea7cfcb23bb7","appId":1507282,"serviceId":318092,"serviceName":"mobius","imageId":1483271,"releaseId":1059344,"dependent":0,"dockerImageId":null}}
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [event] Event: Service kill {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059214}}
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:36 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:37 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:37 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:37 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:37 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:38 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:38 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:38 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:38 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:39 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:39 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:39 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:39 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:40 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:40 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:40 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:40 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:41 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:41 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:41 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:42 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:43 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:43 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:43 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:43 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:44 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:44 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:44 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:44 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:45 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:45 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:45 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:45 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:47 7409930 resin-supervisor[2100]: [event] Event: Service exit {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059214}}
Sep 09 16:39:48 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:48 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:48 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:39:48 7409930 resin-supervisor[2100]: [debug] Non-array fields: {"added":{},"deleted":{},"updated":{"image":"sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff"}}
Sep 09 16:39:48 7409930 resin-supervisor[2100]: [event] Event: Service stop {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059214}}
Sep 09 16:39:49 7409930 resin-supervisor[2100]: [event] Event: Image removal {"image":{"name":"registry2.balena-cloud.com/v2/f44ec5961557eeeac6fdcf1d17885ffc@sha256:15b6ebb805e051aede17e4fd7235ec7e00fa16cf8f8180aa105ec94d826f0953","appId":1507282,"serviceId":318092,"serviceName":"mobius","imageId":1483008,"releaseId":1059214,"dependent":0}}
Sep 09 16:39:49 7409930 resin-supervisor[2100]: [event] Event: Image removed {"image":{"name":"registry2.balena-cloud.com/v2/f44ec5961557eeeac6fdcf1d17885ffc@sha256:15b6ebb805e051aede17e4fd7235ec7e00fa16cf8f8180aa105ec94d826f0953","appId":1507282,"serviceId":318092,"serviceName":"mobius","imageId":1483008,"releaseId":1059214,"dependent":0}}
Sep 09 16:39:53 7409930 resin-supervisor[2100]: [event] Event: Service install {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:39:54 7409930 resin-supervisor[2100]: [event] Event: Service installed {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:39:54 7409930 resin-supervisor[2100]: [event] Event: Service start {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:39:56 7409930 resin-supervisor[2100]: [event] Event: Service started {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:39:58 7409930 resin-supervisor[2100]: [debug] Finished applying target state
Sep 09 16:39:58 7409930 resin-supervisor[2100]: [success] Device state apply success
Sep 09 16:41:06 7409930 resin-supervisor[2100]: [event] Event: Purge data {"appId":1507282}
Sep 09 16:41:06 7409930 resin-supervisor[2100]: [event] Event: Service kill {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:06 7409930 resin-supervisor[2100]: [event] Event: Service kill {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:17 7409930 resin-supervisor[2100]: [event] Event: Service exit {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:17 7409930 resin-supervisor[2100]: [event] Event: Service stop {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:18 7409930 resin-supervisor[2100]: [event] Event: Service exit {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:18 7409930 resin-supervisor[2100]: [event] Event: Service stop {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:19 7409930 resin-supervisor[2100]: [success] Device state apply success
Sep 09 16:41:20 7409930 resin-supervisor[2100]: [event] Event: Service install {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:20 7409930 resin-supervisor[2100]: [event] Event: Service install {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:20 7409930 resin-supervisor[2100]: [event] Event: Service installed {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:20 7409930 resin-supervisor[2100]: [event] Event: Service start {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:20 7409930 resin-supervisor[2100]: [event] Event: Service installed {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:20 7409930 resin-supervisor[2100]: [event] Event: Service start {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:21 7409930 resin-supervisor[2100]: [api] GET /v1/healthy 200 - 13.953 ms
Sep 09 16:41:21 7409930 resin-supervisor[2100]: [event] Event: Service started {"service":{"appId":1507282,"serviceId":318093,"serviceName":"manage","releaseId":1059344}}
Sep 09 16:41:24 7409930 resin-supervisor[2100]: [event] Event: Service started {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:24 7409930 resin-supervisor[2100]: [success] Device state apply success
Sep 09 16:41:24 7409930 resin-supervisor[2100]: [info] Applying target state
Sep 09 16:41:24 7409930 resin-supervisor[2100]: [event] Event: Purge data success {"appId":1507282}
Sep 09 16:41:24 7409930 resin-supervisor[2100]: [api] POST /v1/purge 200 - 18573.728 ms
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [event] Event: Service kill {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:25 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:26 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:26 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:26 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:26 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:27 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:27 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:27 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:27 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:28 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:28 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:28 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:28 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:29 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:29 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:29 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:29 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:31 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:31 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:31 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:31 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:32 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:32 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:32 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:32 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:33 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:33 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:33 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:33 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:34 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:34 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:34 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:34 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:35 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:35 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:35 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:35 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:37 7409930 resin-supervisor[2100]: [event] Event: Service exit {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:37 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:37 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:37 7409930 resin-supervisor[2100]: [debug] Replacing container for service mobius because of config changes:
Sep 09 16:41:37 7409930 resin-supervisor[2100]: [debug] Network changes detected
Sep 09 16:41:37 7409930 resin-supervisor[2100]: [event] Event: Service stop {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:38 7409930 resin-supervisor[2100]: [event] Event: Service install {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:39 7409930 resin-supervisor[2100]: [event] Event: Service installed {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:39 7409930 resin-supervisor[2100]: [event] Event: Service start {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:41 7409930 resin-supervisor[2100]: [event] Event: Service started {"service":{"appId":1507282,"serviceId":318092,"serviceName":"mobius","releaseId":1059344}}
Sep 09 16:41:42 7409930 resin-supervisor[2100]: [debug] Finished applying target state
Sep 09 16:41:42 7409930 resin-supervisor[2100]: [success] Device state apply success
The other thing I noticed is that when I look at the logs during a purge the containers restart after the purge, then I get a log to say the purge has completed and then the containers restart again. I have attached the relevant logs below:
Purging data for app 1507282
Killing service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Killing service 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
Service exited 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
Killed service 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
Service exited 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Killed service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Installing service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Installing service 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
Installed service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Starting service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Installed service 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
Starting service 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
Started service 'manage sha256:605670793216c5515a7d0779b6e4f8847bbd15723be76c71b02363004577b490’
+ export ‘DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket’
+ sleep 1
+ ‘[’ -f /data/.mobius/manage/networks.json ]
+ echo ‘Network config exists - default config not copied’
Network config exists - default config not copied
+ mkdir -p /mnt/mobius/config
+ mount /mnt/mobius/config
mount: mounting LABEL=MOBIUSCONF on /mnt/mobius/config failed: No such file or directory
+ python setup-networks.py
Setting up networks…
Setting up ethernet connection with settings: {u’connection’: {u’type’: u’ethernet’, u’enabled’: True, u’id’: u’Wired connection 1’, u’interface-name’: u’enp2s0’}, u’ipv4’: {u’prefix’: 24, u’gateway’: u’’, u’method’: u’auto’, u’dns’: u’’, u’address’: u’’}}
Activated: enp2s0
Setting up ethernet connection complete
Setting up ethernet connection with settings: {u’connection’: {u’type’: u’ethernet’, u’enabled’: True, u’id’: u’Wired connection 2’, u’interface-name’: u’enp3s0’}, u’ipv4’: {u’prefix’: 24, u’gateway’: u’’, u’method’: u’manual’, u’dns’: u’’, u’address’: u’192.168.1.100’}}
Gateway was invalid - not set
DNS was invalid - not set
Failed to activate: enp3s0
Setting up ethernet connection complete
WiFi connection is disabled: WiFi connection 1
WiFi hotspot is disabled: WiFi hotspot 1
Setting up GSM connection with settings: ({u’connection’: {u’type’: u’gsm’, u’enabled’: True, u’id’: u’GSM connection 1’, u’interface-name’: u’cdc-wdm0’}, u’gsm’: {u’username’: u’arkessa’, u’password’: u’arkessa’, u’apn’: u’arkessa.com’}})
Failed to activate: cdc-wdm0
Setting up GSM connection complete
+ node network-check.js
Network check disabled
Started service ‘mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
+ echo ‘Executing docker-entry.sh’
Executing docker-entry.sh
+ ./docker-pre-entry.sh
./docker-entry.sh: line 6: ./docker-pre-entry.sh: Permission denied
+ ‘[’ -f /data/.mobius/hub/node-config.json ]
+ rm -f ./.mobius/hub/node-config.json
+ rm -rf /data/.mobius/templates/
+ rsync -rtvu .mobius/ /data/.mobius/
sending incremental file list
hub/
templates/
templates/objects/
templates/objects/000B-obj-template.json
templates/objects/000C-obj-template.json
templates/objects/000D-obj-template.json
templates/objects/000E-obj-template.json
templates/objects/000F-obj-template.json
templates/objects/0010-obj-template.json
templates/objects/0011-obj-template.json
templates/objects/0012-obj-template.json
templates/objects/0013-obj-template.json
templates/objects/0014-obj-template.json
templates/objects/0015-obj-template.json
templates/objects/0016-obj-template.json
templates/objects/0017-obj-template.json
templates/objects/0018-obj-template.json
templates/objects/0019-obj-template.json
templates/objects/001A-obj-template.json
templates/objects/001B-obj-template.json
templates/objects/001C-obj-template.json
templates/objects/001D-obj-template.json
templates/objects/001E-obj-template.json
templates/objects/001F-obj-template.json
templates/objects/0020-obj-template.json
templates/objects/0021-obj-template.json
templates/objects/0022-obj-template.json
templates/objects/0023-obj-template.json
templates/objects/0024-obj-template.json
templates/objects/0025-obj-template.json
templates/objects/0026-obj-template.json
templates/objects/0027-obj-template.json
templates/objects/0028-obj-template.json
templates/objects/0029-obj-template.json
templates/objects/002A-obj-template.json
templates/objects/002B-obj-template.json
templates/objects/002C-obj-template.json
templates/objects/002D-obj-template.json
templates/objects/002E-obj-template.json
templates/objects/002F-obj-template.json
templates/objects/0030-obj-template.json
templates/objects/0031-obj-template.json
templates/objects/0032-obj-template.json
Purged data
templates/objects/0033-obj-template.json
templates/objects/0034-obj-template.json
templates/objects/0035-obj-template.json
templates/objects/0036-obj-template.json
templates/objects/0037-obj-template.json
templates/objects/0038-obj-template.json
templates/objects/0039-obj-template.json
templates/objects/004F-obj-template.json
templates/objects/0050-obj-template.json
templates/objects/0051-obj-template.json
templates/objects/012A-obj-template.json
templates/objects/012B-obj-template.json
templates/objects/012D-obj-template.json
templates/objects/012E-obj-template.json
templates/objects/0143-obj-template.json
templates/objects/0144-obj-template.json
templates/objects/0145-obj-template.json
templates/objects/0146-obj-template.json
templates/objects/0148-obj-template.json
templates/objects/0149-obj-template.json
templates/objects/014A-obj-template.json
templates/objects/014B-obj-template.json
templates/objects/01AE-obj-template.json
templates/objects/01AF-obj-template.json
templates/objects/01B0-obj-template.json
templates/services/
templates/services/0001-srvc-template.json
templates/services/0003-srvc-template.json
templates/services/000E-srvc-template.json
templates/services/000F-srvc-template.json
templates/services/0020-srvc-template.json
templates/services/0021-srvc-template.json
templates/services/0023-srvc-template.json
templates/services/0024-srvc-template.json
templates/services/0025-srvc-template.json
templates/services/0026-srvc-template.json
templates/services/0027-srvc-template.json
templates/services/0028-srvc-template.json
templates/services/0029-srvc-template.json
templates/services/002A-srvc-template.json
templates/services/002B-srvc-template.json
templates/services/002C-srvc-template.json
templates/services/002D-srvc-template.json
templates/services/002E-srvc-template.json
templates/services/002F-srvc-template.json
templates/services/0030-srvc-template.json
templates/services/0031-srvc-template.json
sent 78,520 bytes received 1,671 bytes 160,382.00 bytes/sec
total size is 218,788 speedup is 2.73
+ setcap ‘cap_net_bind_service=+ep’ /usr/local/bin/node
Killing service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
+ chown -R mobius:mobius /data
+ mv ./.bashrc /root/
+ exec su-exec mobius ./docker-run.sh
+ echo ‘Executing docker-run.sh’
Executing docker-run.sh
+ sed -n ‘s/^."([0-9][^"])".*$/\1/p’
+ grep version node_modules/package.json
+ pver=4.0.0
+ export ‘MOBIUS_NODE_BASE_VERSION=4.0.0-gateway’
+ echo MOBIUS_NODE_BASE_VERSION 4.0.0-gateway
MOBIUS_NODE_BASE_VERSION 4.0.0-gateway
+ echo ‘Stopping mobius’
Stopping mobius
+ pm2 delete all
[PM2] Spawning PM2 daemon with pm2_home=/data/.pm2
[PM2] PM2 Successfully daemonized
[PM2][WARN] No process found
┌──────────┬────┬─────────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴─────────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
Use pm2 show <id|name> to get more details about an app
+ true
+ pm2 flush
[PM2] Flushing /data/.pm2/pm2.log
[PM2] Logs flushed
+ pm2 start --name mobius-hub node_modules/@ia/mobius-hub/bin/hub.js
[PM2] Starting /usr/src/app/node_modules/@ia/mobius-hub/bin/hub.js in fork_mode (1 instance)
[PM2] Done.
┌────────────┬────┬─────────┬──────┬─────┬────────┬─────────┬────────┬─────┬───────────┬────────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
├────────────┼────┼─────────┼──────┼─────┼────────┼─────────┼────────┼─────┼───────────┼────────┼──────────┤
│ mobius-hub │ 0 │ 3.0.16 │ fork │ 86 │ online │ 0 │ 0s │ 0% │ 24.9 MB │ mobius │ disabled │
└────────────┴────┴─────────┴──────┴─────┴────────┴─────────┴────────┴─────┴───────────┴────────┴──────────┘
Use pm2 show <id|name> to get more details about an app
+ pm2 log
[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
/data/.pm2/pm2.log last 15 lines:
PM2 | 2019-09-09T16:41:28: PM2 log: App [mobius-hub:0] starting in -fork mode-
PM2 | 2019-09-09T16:41:28: PM2 log: App [mobius-hub:0] online
/data/.pm2/logs/mobius-hub-out.log last 15 lines:
/data/.pm2/logs/mobius-hub-error.log last 15 lines:
0|mobius-hub | [2019-09-09T16:41:31.029] [WARN] mobius-hub - Loaded new HUB Id from environment variables: DE0001
0|mobius-hub | [2019-09-09T16:41:31.120] [WARN] mobius-hub - Mobius Config UI is enabled: true
0|mobius-hub | MobiusHub DE0001 with name Hub 000001 running and listening on port 2016 …
PM2 | App [0012/012:1] starting in -fork mode-
PM2 | App [0012/012:1] online
Service exited 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Killed service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Installing service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Installed service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Starting service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
Started service 'mobius sha256:69030101a1b93a836d45b88993e884ba2a93450adf97d703706b4a2707a381ff’
+ echo ‘Executing docker-entry.sh’
Executing docker-entry.sh
+ ./docker-pre-entry.sh
./docker-entry.sh: line 6: ./docker-pre-entry.sh: Permission denied
+ ‘[’ -f /data/.mobius/hub/node-config.json ]
+ rm -f ./.mobius/hub/node-config.json
+ rm -rf /data/.mobius/templates/
+ rsync -rtvu .mobius/ /data/.mobius/
sending incremental file list
hub/
templates/
templates/objects/
templates/objects/000B-obj-template.json
templates/objects/000C-obj-template.json
templates/objects/000D-obj-template.json
But only one of the services is killed and restarted a second time!
All above was with an UP2 board on latest balenaOS. You will notice the after killing the services and before restarting them it does not state that it has removed and re-created the volumes.
I tested on a raspberry PI3 with latest balenaOS it does purge properly and does say it has deleted and re-created volumes. Seems to be a problem with UP version of the OS. See below for PI logs:
Purging data for app 1376356
Killing service ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Killing service ‘mobius sha256:e00ddc0e4de3566f3cc48777b5d81fd32851cc13fd7bd0e36867f7402f7615a5’
Service exited ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Killed service ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Service exited ‘mobius sha256:e00ddc0e4de3566f3cc48777b5d81fd32851cc13fd7bd0e36867f7402f7615a5’
Killed service ‘mobius sha256:e00ddc0e4de3566f3cc48777b5d81fd32851cc13fd7bd0e36867f7402f7615a5’
Removing volume 'conf-data’
Removing volume 'resin-data’
Creating volume 'conf-data’
Creating volume 'resin-data’
Installing service ‘mobius sha256:e00ddc0e4de3566f3cc48777b5d81fd32851cc13fd7bd0e36867f7402f7615a5’
Installing service ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Installed service ‘mobius sha256:e00ddc0e4de3566f3cc48777b5d81fd32851cc13fd7bd0e36867f7402f7615a5’
Starting service ‘mobius sha256:e00ddc0e4de3566f3cc48777b5d81fd32851cc13fd7bd0e36867f7402f7615a5’
Installed service ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Starting service ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Started service ‘manage sha256:c42d90d80b80540b39eead8cb04945b33d2cadfab9e6e5e9a6381c4f2c04fd25’
Hey @chrismoorhouse thanks for reporting this. I have forwarded the ticket to the supervisor and OS team so they can have a look. Will get back to you as soon as we have an idea of what may cause this behaviour
Hey @ChrisMoorhouse, tested this out, and indeed there seems to be a supervisor issue with the “Purge” action on supervisors above v10.0.0 at least. The reason why it worked on a Pi3 because it’s running an earlier supervisor, I would think (looking at the currently available versions for the Pi3 and the Up-board)
I believe (but we will be able to confirm tomorrow with the maintainer) that the issue is because of the change to volume management in the supervisor v10.x.y series. To avoid user data loss in certain situations, volumes that have no references are no longer cleared up (so if you accidentally remove a volume definition from your docker-compose.yml, you won’t lose the data, can get it back). This required a different behaviour and different supervisor functionality to clear up those unreferenced volumes. (see the docs here). I suspect that this inadvertently broke the purge functionality compared to before.
We’ve opened a supervisor issue here, you can subscribe to it to follow the changes as well:
Will keep you posted (and hopefully we’ll get a confirmation of the validity of this issue as well soon). Thanks a lot for reporting this!
Hi, just wanted to let you know that purging issue is now resolved on the supervisor side since version 10.3.6.
balena team will soon be releasing OS versions including the supervisor fix.