balena preload does not work on aws codebuild

openBalena
1

Hi,
Preload is working well when run it on my local ubuntu pc.
But if I run preload on aws codebuild, it fails.

balena preload balena.img --app $BALENA_APP_NAME --commit current --add-certificate ca.crt
Here are logs.

Traceback (most recent call last):
File “/usr/src/app/preload.py”, line 843, in
result = method(**data.get(“parameters”, {}))
File “/usr/src/app/preload.py”, line 802, in get_image_info
images, supervisor_version = get_images_and_supervisor_version()
File “/usr/src/app/preload.py”, line 685, in get_images_and_supervisor_version
return _get_images_and_supervisor_version(inner_image_path)
File “/usr/src/app/preload.py”, line 657, in _get_images_and_supervisor_version
with docker_context_manager(driver, mountpoint):
File “/usr/lib/python3.6/contextlib.py”, line 81, in enter
return next(self.gen)
File “/usr/src/app/preload.py”, line 524, in docker_context_manager
running_dockerd = start_docker_daemon(storage_driver, docker_dir)
File “/usr/src/app/preload.py”, line 493, in start_docker_daemon
running_dockerd.wait()
File “/usr/lib/python3.6/site-packages/sh.py”, line 792, in wait
self.handle_command_exit_code(exit_code)
File “/usr/lib/python3.6/site-packages/sh.py”, line 815, in handle_command_exit_code
raise exc
sh.ErrorReturnCode_1:

RAN: /usr/local/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmp7x8f4m2a/docker --host=tcp://0.0.0.0:44997

STDOUT:

STDERR:
time=“2020-04-06T07:17:42.684486970Z” level=warning msg="[!] DON’T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON’T KNOW WHAT YOU’RE DOING [!]"
time=“2020-04-06T07:17:42.685041612Z” level=info msg=“libcontainerd: started new docker-containerd process” pid=238
time=“2020-04-06T07:17:42Z” level=info msg=“starting containerd” module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
time=“2020-04-06T07:17:42Z” level=info msg=“setting subreaper…” module=containerd
time=“2020-04-06T07:17:42Z” level=info msg=“changing OOM score to -500” module=containerd
time=“2020-04-06T07:17:42Z” level=info msg=“loading plugin “io.containerd.content.v1.content”…” module=containerd type=io.containerd.content.v1
tim… (3787 more, please see e.stderr)
at /usr/local/lib/node_modules/balena-cli/node_modules/balena-preload/lib/preload.js:317:19
at /usr/local/lib/node_modules/balena-cli/build/actions/preload.js:289:14
at processImmediate (internal/timers.js:456:21)
at Command.action (/usr/local/lib/node_modules/balena-cli/build/actions/preload.js:270:43)
at /usr/local/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:98:37
at runNextTicks (internal/process/task_queues.js:62:5)
at processImmediate (internal/timers.js:429:9)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at runCommand (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:173:14)
at Object.exports.run (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:185:39)
at routeCliFramework (/usr/local/lib/node_modules/balena-cli/build/preparser.js:39:79)
at async Object.run (/usr/local/lib/node_modules/balena-cli/build/app.js:14:5)

If you need help, don’t hesitate in contacting our support forums at
https://forums.balena.io

For CLI bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/

3

Hi there. Good to see that is working on your local machine - do any other operations work on the aws deployment, or is it just preload that fails?

5

Hi, @rcooke-warwick
Hope you are doing well.

Yes. Other operations works well. Only preload fails.

7

Interesting. The team think that the AWS linux kernel may be missing modules required for AUFS. Could you try running the same command, prefixed with DEBUG=1 please? (DEBUG=1 balena preload balena.img --app $BALENA_APP_NAME --commit current --add-certificate ca.crt)

This may give us some more info, thanks!

9

I used export DEBUG=1.

[Container] 2020/04/06 10:26:55 Running command balena preload balena.img --app $BALENA_APP_NAME --commit current --add-certificate ca.crt
[debug] original argv0=“node” argv=[/usr/local/bin/node,/usr/local/bin/balena,preload,balena.img,–app,liu-test,–commit,current,–add-certificate,ca.crt] length=12

·[1ABuilding Docker preloader image. [ ] 0%
·[1A
·[1ABuilding Docker preloader image. [=== ] 12%
Step 1/7 : FROM docker:17.12.0-ce-dind
—> 08c245f2381e
·[1A
·[1ABuilding Docker preloader image. [====== ] 25%
Step 2/7 : RUN apk update && apk add --no-cache python3 parted btrfs-progs util-linux sfdisk file coreutils sgdisk
—> Running in 3a48140547b5
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
v3.7.3-180-g5372bc29f3 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.3-165-g0f7ecd696d [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
OK: 9067 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/27) Installing libattr (2.4.47-r6)
(2/27) Installing libacl (2.2.52-r3)
(3/27) Installing coreutils (8.28-r0)
(4/27) Installing libmagic (5.32-r2)
(5/27) Installing file (5.32-r2)
(6/27) Installing device-mapper-libs (2.02.175-r0)
(7/27) Installing ncurses-terminfo-base (6.0_p20171125-r1)
(8/27) Installing ncurses-terminfo (6.0_p20171125-r1)
(9/27) Installing ncurses-libs (6.0_p20171125-r1)
(10/27) Installing readline (7.0.003-r0)
(11/27) Installing parted (3.2-r6)
(12/27) Installing libbz2 (1.0.6-r7)
(13/27) Installing expat (2.2.8-r0)
(14/27) Installing libffi (3.2.1-r4)
(15/27) Installing gdbm (1.13-r1)
(16/27) Installing sqlite-libs (3.25.3-r2)
(17/27) Installing python3 (3.6.9-r1)
(18/27) Installing libfdisk (2.31.1-r0)
(19/27) Installing libsmartcols (2.31.1-r0)
(20/27) Installing sfdisk (2.31.1-r0)
(21/27) Installing libgcc (6.4.0-r5)
(22/27) Installing popt (1.16-r7)
(23/27) Installing libstdc++ (6.4.0-r5)
(24/27) Installing sgdisk (1.0.3-r0)
(25/27) Installing libmount (2.31.1-r0)
(26/27) Installing findmnt (2.31.1-r0)
(27/27) Installing util-linux (2.31.1-r0)
Executing busybox-1.27.2-r7.trigger
OK: 95 MiB in 62 packages
Removing intermediate container 3a48140547b5
—> 04b1f6b5c2ae
·[1A
·[1ABuilding Docker preloader image. [========= ] 37%
Step 3/7 : COPY ./requirements.txt /tmp/
—> afc4b8e6c5f0
·[1A
·[1ABuilding Docker preloader image. [============ ] 50%
Step 4/7 : RUN pip3 install -r /tmp/requirements.txt
—> Running in ae617e44963f
Collecting sh==1.12.14 (from -r /tmp/requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/4a/22/17b22ef5b049f12080f5815c41bf94de3c229217609e469001a8f80c1b3d/sh-1.12.14-py2.py3-none-any.whl
Collecting retry==0.9.2 (from -r /tmp/requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/4b/0d/53aea75710af4528a25ed6837d71d117602b01946b307a3912cb3cfcbcba/retry-0.9.2-py2.py3-none-any.whl
Collecting py<2.0.0,>=1.4.26 (from retry==0.9.2->-r /tmp/requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/99/8d/21e1767c009211a62a8e3067280bfce76e89c9f876180308515942304d2d/py-1.8.1-py2.py3-none-any.whl (83kB)
Collecting decorator>=3.4.2 (from retry==0.9.2->-r /tmp/requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl
Installing collected packages: sh, py, decorator, retry
Successfully installed decorator-4.4.2 py-1.8.1 retry-0.9.2 sh-1.12.14
·[91mYou are using pip version 18.1, however version 20.0.2 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
·[0mRemoving intermediate container ae617e44963f
—> 563b2b207a2d
·[1A
·[1ABuilding Docker preloader image. [=============== ] 62%
Step 5/7 : COPY ./src /usr/src/app
—> 29115138ee36
·[1A
·[1ABuilding Docker preloader image. [================== ] 75%
Step 6/7 : WORKDIR /usr/src/app
—> Running in becbd470c330
Removing intermediate container becbd470c330
—> 47f0b27794fa
·[1A
·[1ABuilding Docker preloader image. [===================== ] 87%
Step 7/7 : CMD [“python3”, “/usr/src/app/preload.py”]
—> Running in a2e808558955
Removing intermediate container a2e808558955
—> d635a02cdc78
Successfully built d635a02cdc78
Successfully tagged balena/balena-preload:latest
·[1A
·[1ABuilding Docker preloader image. [========================] 100%

·[2K·[1G
·[2K·[1G
·[2K·[1G
·[2K·[1G
·[2K·[1G
·[2K·[1GWaiting for Docker to start…
Exception in thread background thread for pid 213:
Traceback (most recent call last):
File “/usr/lib/python3.6/threading.py”, line 916, in _bootstrap_inner
self.run()
File “/usr/lib/python3.6/threading.py”, line 864, in run
self._target(*self._args, **self._kwargs)
File “/usr/lib/python3.6/site-packages/sh.py”, line 1540, in wrap
fn(*args, **kwargs)
File “/usr/lib/python3.6/site-packages/sh.py”, line 2459, in background_thread
handle_exit_code(exit_code)
File “/usr/lib/python3.6/site-packages/sh.py”, line 2157, in fn
return self.command.handle_command_exit_code(exit_code)
File “/usr/lib/python3.6/site-packages/sh.py”, line 815, in handle_command_exit_code
raise exc
sh.ErrorReturnCode_1:

RAN: /usr/local/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmpzbxm_9n6/docker --host=tcp://0.0.0.0:44315

STDOUT:

STDERR:
time=“2020-04-06T10:27:33.681805502Z” level=warning msg="[!] DON’T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON’T KNOW WHAT YOU’RE DOING [!]"
time=“2020-04-06T10:27:33.682639246Z” level=info msg=“libcontainerd: started new docker-containerd process” pid=232
time=“2020-04-06T10:27:33Z” level=info msg=“starting containerd” module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
time=“2020-04-06T10:27:33Z” level=info msg=“setting subreaper…” module=containerd
time=“2020-04-06T10:27:33Z” level=info msg=“changing OOM score to -500” module=containerd
time=“2020-04-06T10:27:33Z” level=info msg=“loading plugin “io.containerd.content.v1.content”…” module=containerd type=io.containerd.content.v1
tim… (3787 more, please see e.stderr)

Traceback (most recent call last):
File “/usr/src/app/preload.py”, line 843, in
result = method(**data.get(“parameters”, {}))
File “/usr/src/app/preload.py”, line 802, in get_image_info
images, supervisor_version = get_images_and_supervisor_version()
File “/usr/src/app/preload.py”, line 685, in get_images_and_supervisor_version
return _get_images_and_supervisor_version(inner_image_path)
File “/usr/src/app/preload.py”, line 657, in _get_images_and_supervisor_version
with docker_context_manager(driver, mountpoint):
File “/usr/lib/python3.6/contextlib.py”, line 81, in enter
return next(self.gen)
File “/usr/src/app/preload.py”, line 524, in docker_context_manager
running_dockerd = start_docker_daemon(storage_driver, docker_dir)
File “/usr/src/app/preload.py”, line 493, in start_docker_daemon
running_dockerd.wait()
File “/usr/lib/python3.6/site-packages/sh.py”, line 792, in wait
self.handle_command_exit_code(exit_code)
File “/usr/lib/python3.6/site-packages/sh.py”, line 815, in handle_command_exit_code
raise exc
sh.ErrorReturnCode_1:

RAN: /usr/local/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmpzbxm_9n6/docker --host=tcp://0.0.0.0:44315

STDOUT:

STDERR:
time=“2020-04-06T10:27:33.681805502Z” level=warning msg="[!] DON’T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON’T KNOW WHAT YOU’RE DOING [!]"
time=“2020-04-06T10:27:33.682639246Z” level=info msg=“libcontainerd: started new docker-containerd process” pid=232
time=“2020-04-06T10:27:33Z” level=info msg=“starting containerd” module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
time=“2020-04-06T10:27:33Z” level=info msg=“setting subreaper…” module=containerd
time=“2020-04-06T10:27:33Z” level=info msg=“changing OOM score to -500” module=containerd
time=“2020-04-06T10:27:33Z” level=info msg=“loading plugin “io.containerd.content.v1.content”…” module=containerd type=io.containerd.content.v1
tim… (3787 more, please see e.stderr)
at /usr/local/lib/node_modules/balena-cli/node_modules/balena-preload/lib/preload.js:317:19
at /usr/local/lib/node_modules/balena-cli/node_modules/balena-preload/lib/preload.js:317:19
at /usr/local/lib/node_modules/balena-cli/build/actions/preload.js:289:14
at processImmediate (internal/timers.js:456:21)
at Command.action (/usr/local/lib/node_modules/balena-cli/build/actions/preload.js:270:43)
at /usr/local/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:98:37
at runNextTicks (internal/process/task_queues.js:62:5)
at processImmediate (internal/timers.js:429:9)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at runCommand (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:173:14)
at Object.exports.run (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:185:39)
at routeCliFramework (/usr/local/lib/node_modules/balena-cli/build/preparser.js:39:79)
at async Object.run (/usr/local/lib/node_modules/balena-cli/build/app.js:14:5)

If you need help, don’t hesitate in contacting our support forums at
https://forums.balena.io

For CLI bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/

10

Hello, balena preload requires a Linux kernel supporting aufs. Can you try running the following as root on aws:

mkdir dir1
mkdir root
mount -t aufs -o br=dir1 none root
touch dir1/some-file
ls root
12

Hi, @zvin
It seems aws code build does not support aufs file system

[Container] 2020/04/07 06:58:19 Running command mkdir dir1
[Container] 2020/04/07 06:58:19 Running command mount -t aufs -o br=dir1 none root
mount: /root: unknown filesystem type 'aufs'.
13

Hi, the balena preload command really needs aufs to run. Searching around a bit it seems like someone got aufs working on AWS: https://forums.aws.amazon.com/thread.jspa?threadID=196979
That might help you get it running.