Pull, EOF, sigsegv

I have observed the same behaviour as described in https://github.com/balena-os/balena-engine/issues/177 when trying ‘balena-engine pull busybox’.

Error message:

Using default tag: latest
error during connect: Post http://%2Fvar%2Frun%2Fbalena-engine.sock/v1.39/images/create?fromImage=busybox&tag=latest: EOF

Including allot of systemd error output as described in issue

Balena engine: ‘Docker version v18.9.7, build 87ef8e0dca’
Uname: Linux bbox 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Lsb release:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.1 LTS”
Network: Behind proxy

I have tried ‘docker pull busybox’ which works as expected. Docker is version ‘Docker version 18.09.7, build 2d0083d’

I am unsure if it related to https://stackoverflow.com/questions/49043181/eof-error-while-pulling-images-from-docker as ‘docker pull busybox’ is working as expected

Did you try the workaround mentioned in that issue thread?

‘GODEBUG=netdns=go balena-engine pull busybox’ does not work.

Tried balena-engine on another bionic ubuntu computer on the same network, where it worked as expected.

@aliasbits the env war needs to be set on the balena-engine-daemon process… You can add a drop-in unit to the systemd service if you’re using that.
Something like systemctl edit balena-engine.service and then:

[Service]
Environment=GODEBUG=netdns=go

It’s interesting that another ubuntu machine works, do you mind looking at the glibc versions of those two machines for me? :thinking:

Edit: fix spelling of “environment”

Service environment update does not change anything. Still not working.

Both machines are using ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27

Thanks for the input, I will see if I can get a bionic box to test this myself

@aliasbits sorry I didn’t really get around to try this myself yet… Did you continue to experience the issue?

Since you say it only happens on one of those machines, what is different about them? Especially from a network perspective, since that seems to be the part that fails.

In the github issue you linked balenaEngine would crash because of the libc issue. Do you experience this as well? Maybe if you could paste the output of journalctl --no-pager -eu balena-engine I can have a look…

cheers,
Robert

balena-engine_log_outout.log (68.2 KB)

Looking at the logs you pasted this has to be https://github.com/balena-os/balena-engine/issues/170

This is how I fixed it:

First reproduce this on a fresh install:

qemu@ubuntu:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

qemu@ubuntu:~$ sudo balena-engine pull busybox
Using default tag: latest
error during connect: Post http://%2Fvar%2Frun%2Fbalena-engine.sock/v1.39/images/create?fromImage=busybox&tag=latest: EOF

Edit the service as mentioned above:

qemu@ubuntu:~$ sudo systemctl edit balena-engine
qemu@ubuntu:~$ sudo systemctl restart balena-engine
qemu@ubuntu:~$ systemctl show balena-engine | grep netdns=go
Environment=GODEBUG=netdns=go

Try again:

qemu@ubuntu:~$ sudo balena-engine pull busybox
Using default tag: latest
latest: Pulling from library/busybox
7c9d20b9b6cd: Pull complete
Total:  [==================================================>]  760.8kB/760.8kB
Digest: sha256:fe301db49df08c384001ed752dff6d52b4305a73a7f608f21528048e8a08b51e
Status: Downloaded newer image for busybox:latest

I found the issue as my

systemctl show balena-engine | grep netdns=go

did not show any environment.

I copied your balena-engine conf override without type checking which was missing a ‘n’ in environment.

Sorry for the noise.

sorry for that :stuck_out_tongue: glad we figured it out o/