Unable to Start openBalena instance

openBalena
1

Hi All,

I am trying to start the balena instance and getting pplyLayer exit status 1 stdout: stderr: permission denied

Could you please help?

root@Balena:~/open-balena# ./scripts/compose up -d
Pulling s3 (balena/open-balena-s3:v2.6.2)…
v2.6.2: Pulling from balena/open-balena-s3
22dbe790f715: Extracting [==================================================>] 45.34MB/45.34MB
417fb282227e: Download complete
0b592aa7ebd4: Download complete
b92ebe9e7cc4: Download complete
aa08e70f73ba: Download complete
577c6a896d0e: Download complete
dc3c3434bf41: Download complete
c5f438b6290f: Download complete
86840dc98271: Download complete
5a82baa7b904: Download complete
7bc36acb5127: Download complete
9ff553a54788: Download complete
3ae20e6a3dcb: Download complete
e1e775bc6b19: Download complete
2822433bcfd3: Download complete
e3269537733a: Download complete
bcce7b5c2b5b: Download complete
0317148ae6f3: Download complete
67ec0dd97efb: Download complete
a2562b82dc10: Download complete
c01b68bd8038: Download complete
08d34353a24f: Download complete
f30cbdac80aa: Download complete
e9b28f006ab2: Download complete
ERROR: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied
root@Balena:~/open-balena# ./scripts/compose up -d
Pulling db (balena/open-balena-db:v2.0.3)…
v2.0.3: Pulling from balena/open-balena-db
f7e2b70d04ae: Extracting [==================================================>] 22.5MB/22.5MB
027ad848ac9c: Download complete
7c040ef66643: Download complete
b891079ad2eb: Download complete
cb64a97e42d9: Download complete
1b88625f7d89: Download complete
a6ac0b663e77: Download complete
594497f0a694: Download complete
dad6671cf9e4: Download complete
031e395c48c5: Download complete
94bb003366d6: Download complete
9f5ee56c6727: Download complete
9f7973e52e7c: Download complete
d2a24cf07a9a: Download complete
3ce003e0ceba: Download complete
ERROR: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied
root@Balena:~/open-balena# ./scripts/compose up -d
Pulling s3 (balena/open-balena-s3:v2.6.2)…
v2.6.2: Pulling from balena/open-balena-s3
22dbe790f715: Extracting [==================================================>] 45.34MB/45.34MB
417fb282227e: Download complete
0b592aa7ebd4: Download complete
b92ebe9e7cc4: Download complete
aa08e70f73ba: Download complete
577c6a896d0e: Download complete
dc3c3434bf41: Download complete
c5f438b6290f: Download complete
86840dc98271: Download complete
5a82baa7b904: Download complete
7bc36acb5127: Download complete
9ff553a54788: Download complete
3ae20e6a3dcb: Download complete
e1e775bc6b19: Download complete
2822433bcfd3: Download complete
e3269537733a: Download complete
bcce7b5c2b5b: Download complete
0317148ae6f3: Download complete
67ec0dd97efb: Download complete
a2562b82dc10: Download complete
c01b68bd8038: Download complete
08d34353a24f: Download complete
f30cbdac80aa: Download complete
e9b28f006ab2: Download complete
ERROR: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied

Log:

root@Balena:/var/log# tail -f syslog
Jun 12 13:39:00 Balena kernel: [19548108.338235] audit: type=1400 audit(1560346740.605:70156): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=44443 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 13:41:38 Balena dockerd[338]: time=“2019-06-12T13:41:38.574392300Z” level=warning msg=“failed to retrieve runc version: unknown output format: runc version spec: 1.0.1-dev\n”
Jun 12 13:41:48 Balena dockerd[338]: time=“2019-06-12T13:41:48.622527463Z” level=info msg=“Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied”
Jun 12 13:43:30 Balena systemd[1]: systemd-tmpfiles-clean.service: Failed to reset devices.list: Operation not permitted
Jun 12 13:43:30 Balena systemd[1]: Starting Cleanup of Temporary Directories…
Jun 12 13:43:30 Balena systemd[1]: Started Cleanup of Temporary Directories.
Jun 12 13:43:45 Balena dockerd[338]: time=“2019-06-12T13:43:45.274275882Z” level=warning msg=“failed to retrieve runc version: unknown output format: runc version spec: 1.0.1-dev\n”
Jun 12 13:43:48 Balena kernel: [19548395.981861] audit: type=1400 audit(1560347028.249:70166): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name="/" pid=52445 comm=“exe” flags=“rw, rslave”
Jun 12 13:43:55 Balena dockerd[338]: time=“2019-06-12T13:43:55.217933703Z” level=info msg=“Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied”
Jun 12 13:45:09 Balena systemd[1]: Started Session 57829 of user root.
Jun 12 13:45:44 Balena dockerd[338]: time=“2019-06-12T13:45:44.390466061Z” level=warning msg=“failed to retrieve runc version: unknown output format: runc version spec: 1.0.1-dev\n”
Jun 12 13:45:46 Balena kernel: [19548514.707481] audit: type=1400 audit(1560347146.973:70167): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name="/" pid=55527 comm=“exe” flags=“rw, rslave”
Jun 12 13:45:47 Balena dockerd[338]: time=“2019-06-12T13:45:47.603069111Z” level=info msg=“Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied”
Jun 12 13:45:56 Balena dockerd[338]: time=“2019-06-12T13:45:56.838216416Z” level=warning msg=“failed to retrieve runc version: unknown output format: runc version spec: 1.0.1-dev\n”
Jun 12 13:45:59 Balena kernel: [19548527.336739] audit: type=1400 audit(1560347159.601:70168): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name="/" pid=56339 comm=“exe” flags=“rw, rslave”
Jun 12 13:46:06 Balena dockerd[338]: time=“2019-06-12T13:46:06.686234977Z” level=info msg=“Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied”
Jun 12 13:48:15 Balena dockerd[338]: time=“2019-06-12T13:48:15.938350569Z” level=warning msg=“failed to retrieve runc version: unknown output format: runc version spec: 1.0.1-dev\n”
Jun 12 13:48:18 Balena kernel: [19548666.101952] audit: type=1400 audit(1560347298.369:70169): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name="/" pid=60363 comm=“exe” flags=“rw, rslave”
Jun 12 13:48:22 Balena dockerd[338]: time=“2019-06-12T13:48:22.225111591Z” level=info msg=“Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: permission denied”
Jun 12 14:05:01 Balena CRON[719]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jun 12 14:16:33 Balena systemd[1]: motd-news.service: Failed to reset devices.list: Operation not permitted
Jun 12 14:16:33 Balena systemd[1]: Starting Message of the Day…
Jun 12 14:16:33 Balena 50-motd-news[722]: * Ubuntu’s Kubernetes 1.14 distributions can bypass Docker and use containerd
Jun 12 14:16:33 Balena 50-motd-news[722]: directly, see https://bit.ly/ubuntu-containerd or try it now with
Jun 12 14:16:33 Balena 50-motd-news[722]: snap install microk8s --classic
Jun 12 14:16:33 Balena systemd[1]: Started Message of the Day.
Jun 12 14:39:00 Balena kernel: [19551708.408611] kauditd_printk_skb: 2 callbacks suppressed
Jun 12 14:39:00 Balena kernel: [19551708.408618] audit: type=1400 audit(1560350340.673:70183): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12671 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 14:39:00 Balena kernel: [19551708.408625] audit: type=1400 audit(1560350340.673:70185): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12671 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 14:39:01 Balena kernel: [19551708.911193] audit: type=1400 audit(1560350341.177:70186): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12700 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 14:39:01 Balena kernel: [19551708.911198] audit: type=1400 audit(1560350341.177:70187): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12700 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 14:39:01 Balena kernel: [19551708.911201] audit: type=1400 audit(1560350341.177:70188): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12700 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 14:39:01 Balena kernel: [19551708.911204] audit: type=1400 audit(1560350341.177:70189): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12700 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none
Jun 12 14:39:04 Balena kernel: [19551712.505132] audit: type=1400 audit(1560350344.773:70191): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12981 comm="(ionclean)" family=“unix” sock_type=“dgram” protocol=0 addr=none

4

Hi,

Based on some of the log entries you shared, this doesn’t look like an issue in openBalena but rather an issue in your host OS…

Ubuntu’s Kubernetes 1.14 distributions can bypass Docker and use containerd

Is this some kind of VPS you are using, and if so maybe you could share some specifics?

8

Just adding a thought, that the issue could actually be related to your AppArmor configuration:

audit: type=1400 audit(1560350340.673:70183): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=12671 comm=“(ionclean)” family=“unix” sock_type=“dgram” protocol=0 addr=none

Perhaps try (temporarily, selectively) disabling AppArmor, and see if it helps.

10

The server is hosted in proxmox container. It is the test platform and running Ubuntu 18.04 x64 . I will check with VM instead of Container and let you know how it goes

Thank you.

11

Hello [pdcastro], It is not getting better after disabling AppArmor.

12

Are you able to pull and run other docker images on this OS? What happens if you run docker run node?

14

It looks like the issue was with LC. When i have hosted ubuntu on KVM, i am able to start balena instance without any problem.

Thank you all for your support.

15

Glad to hear the issue is resolved!

16

For anyone else encountering a similar problem, do you know what the exact cause of the issue was?

17

It looks like a docker didnt like linux container ( May be there is a way to make it work ). But when i have hosted in KVM, everything went well.