Custom BalenaOS NUC build 'find' error

G’day

Trying to get a quick balenaos build going - ran into the find error below, find works fine out of the Python process call.

My goal is to enable a few existing modules and flags in the base BalenaOS to get the NUC spin working for my device.

My main dev machine is Arch, not a directly supported Yocto host platform but that really should be fine. Before I re-do the whole thing in a Ubuntu container - is there a quick fix for the below ? Barring that, seems likely a balena build container already exists - are there any current docs on that ?

❯ ./balena-yocto-scripts/build/barys -m genericx86-64
Building JSON manifest...
audited 3 packages in 1.224s
found 0 vulnerabilities
test_append_and_override_1 (bb.tests.data.TestOverrides) ... ok
...Done
test_append_and_override_3 (bb.tests.data.TestOverrides) ... ok
  _           _                   ___  ____
 | |__   __ _| | ___ _ __   __ _ / _ \/ ___|
 | '_ \ / _` | |/ _ \ '_ \ / _` | | | \___ \
 | |_) | (_| | |  __/ | | | (_| | |_| |___) |
 |_.__/ \__,_|_|\___|_| |_|\__,_|\___/|____/
test_one_override (bb.tests.data.TestOverrides) ... ok
 -------------------------------------------- 
test_remove_with_override (bb.tests.data.TestOverrides) ... ok
Resin specific images available:
	resin-image-flasher
test_append (bb.tests.data.TestRemoteConcat) ... ok
Intel NUC                                : $ MACHINE=genericx86-64 bitbake resin-image-flasher
Microsoft Surface Go (NEW)               : $ MACHINE=surface-go bitbake resin-image-flasher
Microsoft Surface 6 (NEW)                : $ MACHINE=surface-pro-6 bitbake resin-image-flasher

[000000002][LOG]BalenaOS build initialized in directory: build.
[000000002][LOG]Run build for genericx86-64: MACHINE=genericx86-64 bitbake resin-image-flasher 
[000000002][LOG]This might take a while ...
WARNING: Host distribution "arch" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |###################################################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 3341 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.42.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "arch"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "genericx86-64"
DISTRO               = "balena-os"
DISTRO_VERSION       = "2.48.0"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "HEAD:b021992106c6b5ba02b825afa7dcc422b135b59b"
meta-oe              
meta-filesystems     
meta-networking      
meta-python          = "HEAD:6fa72d587aab8e9d56d67a2552eb9cfbe25c86d2"
meta-balena-common   
meta-balena-warrior  = "HEAD:19c838a3322f6b419a6c2a08fa33677016a2bac9"
meta-balena-genericx86 = "master:af1de5a1b7acf31cbf66476bda754a2ee951d4b1"
meta-rust            = "HEAD:c5ff05d1e99c1e80844c1673456e2368499918c7"

WARNING: Your host glibc verson (2.31) is newer than that in uninative (2.30). Disabling uninative so that sstate is not corrupted.
Initialising tasks: 100% |##############################################################################################################################################################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 1407 Found 0 Missed 1407 Current 72 (0% match, 4% complete)
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: base-files-3.0.14-r89 do_package: Fatal errors occurred in subprocesses:
Command '['file', '-b', '/home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.profile']' died with <Signals.SIGSYS: 31>.: Traceback (most recent call last):
  File "/home/thys/balena/balena-intel/build/../layers/poky/meta/lib/oe/utils.py", line 276, in run
    ret = self._target(*self._args, **self._kwargs)
  File "/home/thys/balena/balena-intel/build/../layers/poky/meta/lib/oe/package.py", line 74, in is_elf
    result = subprocess.check_output(["file", "-b", path], stderr=subprocess.STDOUT).decode("utf-8")
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['file', '-b', '/home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.profile']' died with <Signals.SIGSYS: 31>.

Command '['file', '-b', '/home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.bashrc']' died with <Signals.SIGSYS: 31>.: Traceback (most recent call last):
  File "/home/thys/balena/balena-intel/build/../layers/poky/meta/lib/oe/utils.py", line 276, in run
    ret = self._target(*self._args, **self._kwargs)
  File "/home/thys/balena/balena-intel/build/../layers/poky/meta/lib/oe/package.py", line 74, in is_elf
    result = subprocess.check_output(["file", "-b", path], stderr=subprocess.STDOUT).decode("utf-8")
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['file', '-b', '/home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.bashrc']' died with <Signals.SIGSYS: 31>.
Test LogExecTTY event class ... ok
test_LogHandler (bb.tests.event.EventClassesTest)
ERROR: base-files-3.0.14-r89 do_package: 
ERROR: base-files-3.0.14-r89 do_package: Function failed: split_and_strip_files
ERROR: Logfile of failure stored in: /home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/temp/log.do_package.2601353
ERROR: Task (/home/thys/balena/balena-intel/build/../layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package) failed with exit code '1'
NOTE: Tasks Summary: Attempted 761 tasks of which 751 didn't need to be rerun and 1 failed.
test_MultipleProviders (bb.tests.event.EventClassesTest)
Summary: 1 task failed:
  /home/thys/balena/balena-intel/build/../layers/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_package
Summary: There were 2 WARNING messages shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
[000000020][LOG]Build for genericx86-64 failed. Check failed log in build/tmp/log/cooker/genericx86-64 .
[000000020][LOG]If build for genericx86-64 succeeded, final image should have been generated here:
[000000020][LOG]   build/tmp/deploy/images/genericx86-64/resin-image-flasher-genericx86-64.resinos-img
[000000020][LOG]Done.
❯ file -b /home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.profile
ASCII text
❯ file -b /home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.profile
ASCII text
❯ less build/tmp/log/cooker/genericx86-64/console-latest.log
❯ file -b /home/thys/balena/balena-intel/build/tmp/work/genericx86_64-poky-linux/base-files/3.0.14-r89/package/etc/skel/.bashrc
ASCII text

Hi, given that Arch is not a Yocto Project supported distribution there might be other issues waiting along the road. My suggestion would be to use a container as you suggest. There is no official BalenaOS builder, but you can try to use https://hub.docker.com/r/aggurio/docker-balena.
Let me know how it goes,

I came much further with the container builder than before thanks, however when I try and build the NUC image like so:

docker run --rm -it --privileged -v $(pwd)/build/balena-intel:/home/build balena-builder:$RELEASE ./balena-yocto-scripts/build/barys --rm-work -d --machine genericx86-64

I get the following failure, and no *img file

| Docker started.
| Pulling balena/amd64-supervisor:v10.8.0...
| Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
| Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
| WARNING: /home/build/build/tmp/work/genericx86_64-poky-linux/docker-disk/balena_amd64-supervisor-v10.8.0-r0/temp/run.do_compile.1212:1 exit 1 from '$DOCKER run --privileged --rm -e BALENA_STORAGE=aufs -e USER_ID=$(id -u) -e USER_GID=$(id -u) -e TARGET_REPOSITORY="balena/amd64-supervisor" -e TARGET_TAG="v10.8.0" -e HELLO_REPOSITORY="amd64/hello-world" -e HELLO_PLATFORM="linux/amd64" -e PRIVATE_REGISTRY="${PRIVATE_REGISTRY}" -e PRIVATE_REGISTRY_USER="${PRIVATE_REGISTRY_USER}" -e PRIVATE_REGISTRY_PASSWORD="${PRIVATE_REGISTRY_PASSWORD}" -e PARTITION_SIZE="192" -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /home/build/build/tmp/work/genericx86_64-poky-linux/docker-disk/balena_amd64-supervisor-v10.8.0-r0/build:/build --name ${_container_name} ${_image_name}'
| ERROR: Function failed: do_compile (log file is located at /home/build/build/tmp/work/genericx86_64-poky-linux/docker-disk/balena_amd64-supervisor-v10.8.0-r0/temp/log.do_compile.1212)
ERROR: Task (/home/build/build/../layers/meta-balena/meta-balena-common/recipes-containers/docker-disk/docker-disk.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3543 tasks of which 3492 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/build/build/../layers/meta-balena/meta-balena-common/recipes-containers/docker-disk/docker-disk.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
[000000081][LOG]Build for genericx86-64 failed. Check failed log in build/tmp/log/cooker/genericx86-64 .
[000000081][LOG]If build for genericx86-64 succeeded, final image should have been generated here:
[000000081][LOG]   build/tmp/deploy/images/genericx86-64/resin-image-flasher-genericx86-64.resinos-img
[000000081][LOG]Done.

Interestingly the surface-go built seems to be fine, and it produces a *img file.

docker run --rm -it --privileged -v $(pwd)/build/balena-intel:/home/build balena-builder:$RELEASE ./balena-yocto-scripts/build/barys --rm-work -d --machine surface-go

Hi
Are you building on Arch or the builder that Alex linked above?

So I’m using the builder, like above, on my Arch dev machine.

Are you consistently running into this issue with the x86 builds?

*genericx86-64

Yes, just tried to rebuild again:

RELEASE=warrior

docker run --rm -it --privileged -v $(pwd)/build/balena-intel:/home/build balena-builder:$RELEASE ./balena-yocto-scripts/build/barys --rm-work -d --machine genericx86-64

| time=“2020-04-14T13:41:59.195531558Z” level=info msg=“pickfirstBalancer: HandleSubConnStateChange: 0xc420460770, CONNECTING” module=grpc
| time=“2020-04-14T13:41:59.195765535Z” level=info msg=“pickfirstBalancer: HandleSubConnStateChange: 0xc420460770, READY” module=grpc
| time=“2020-04-14T13:41:59.256113936Z” level=error msg=“AUFS was not found in /proc/filesystems” storage-driver=aufs
| Error starting daemon: error initializing graphdriver: driver not supported
| Docker started.
| Pulling balena/amd64-supervisor:v10.8.0…
| Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
| Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
| WARNING: /home/build/build/tmp/work/genericx86_64-poky-linux/docker-disk/balena_amd64-supervisor-v10.8.0-r0/temp/run.do_compile.1803:1 exit 1 from ‘DOCKER run --privileged --rm -e BALENA_STORAGE=aufs -e USER_ID=(id -u) -e USER_GID=(id -u) -e TARGET_REPOSITORY="balena/amd64-supervisor" -e TARGET_TAG="v10.8.0" -e HELLO_REPOSITORY="amd64/hello-world" -e HELLO_PLATFORM="linux/amd64" -e PRIVATE_REGISTRY="{PRIVATE_REGISTRY}" -e PRIVATE_REGISTRY_USER="{PRIVATE_REGISTRY_USER}" -e PRIVATE_REGISTRY_PASSWORD="{PRIVATE_REGISTRY_PASSWORD}" -e PARTITION_SIZE=“192” -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /home/build/build/tmp/work/genericx86_64-poky-linux/docker-disk/balena_amd64-supervisor-v10.8.0-r0/build:/build --name {_container_name} {_image_name}’
| ERROR: Function failed: do_compile (log file is located at /home/build/build/tmp/work/genericx86_64-poky-linux/docker-disk/balena_amd64-supervisor-v10.8.0-r0/temp/log.do_compile.1803)
ERROR: Task (/home/build/build/…/layers/meta-balena/meta-balena-common/recipes-containers/docker-disk/docker-disk.bb:do_compile) failed with exit code ‘1’
NOTE: Tasks Summary: Attempted 4402 tasks of which 4266 didn’t need to be rerun and 1 failed.

Summary: 1 task failed:
/home/build/build/…/layers/meta-balena/meta-balena-common/recipes-containers/docker-disk/docker-disk.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
[000000092][LOG]Build for genericx86-64 failed. Check failed log in build/tmp/log/cooker/genericx86-64 .
[000000092][LOG]If build for genericx86-64 succeeded, final image should have been generated here:
[000000092][LOG] build/tmp/deploy/images/genericx86-64/resin-image-flasher-genericx86-64.resinos-img

Hi, looks like your host machine doens’t support AUFS

| time=“2020-04-14T13:41:59.256113936Z” level=error msg=“AUFS was not found in /proc/filesystems” storage-driver=aufs
| Error starting daemon: error initializing graphdriver: driver not supported

You can overwrite it here: https://github.com/alexgg/docker-balena/blob/master/Dockerfile#L26

You can check what your docker is using by checking docker info

So I changed my containers extra opts to

ENV DOCKER_EXTRA_OPTS ‘–storage-driver=overlay2’

as this is what my docker uses:

❯ docker info
Client:
Debug Mode: false

Server:
 Containers: 110
  Running: 1
  Paused: 0
  Stopped: 109
 Images: 622
 Server Version: 19.03.8-ce
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.6.3-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.28GiB
 Name: rog
 ID: QZD3:7Z3E:3KAP:JO47:B2HS:QNQG:DDVE:YHVS:LJYY:Q67J:OGRT:5YKU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

But that made no difference to the genericx86-64 build - keep in mind it works fine for the surface-go build.

But that made no difference to the genericx86-64 build - keep in mind it works fine for the surface-go build.

@sthysel, I’ve gathered “some evidence” (as opposed to certainty) that the NUC images require the AUFS driver, whereas surface-go require the overlay2 driver:

These settings:

ENV DOCKER_EXTRA_OPTS ‘–storage-driver=aufs’
ENV DOCKER_EXTRA_OPTS ‘–storage-driver=overlay2’

… would tell Docker which storage drivers to load, and the Linux kernel needs the module to be loaded as well, but it is not a choice: the NUC image needs AUFS, the surface-go image needs overlay2.

The following Docker documentation indicates how to check that the host OS kernel supports AUFS: https://docs.docker.com/storage/storagedriver/aufs-driver/#configure-docker-with-the-aufs-storage-driver

$ grep aufs /proc/filesystems
nodev   aufs

Try ensuring that the host OS / kernel and Docker have support for AUFS when building with the NUC image.