Balena exec failures

Product support
1

We have been seeing this for many months, however with the growth of our app and the supervisor this has become very common recently.

# balena exec -it 2ec2246c603b /bin/bash
rpc error: code = Unknown desc = invalid character 'e' looking for beginning of value

We see this all the time on many devices. When running Balena exec with less than a huge amount of available memory, the exec fails (-t needs to be omitted to get the error message) with ‘rpc error: code = Unknown desc = oci runtime error: exec failed: container_linux.go:263: starting container process caused "process_linux.go:95: writing config to pipe caused “write init-p: broken pipe”’

root@dcea8b0:~# free -m
             total       used       free     shared    buffers     cached
Mem:        504284     445464      58820      10616       6424     112424
-/+ buffers/cache:     326616     177668
Swap:            0          0          0

It’s not exactly memory pressure, but perhaps fragmentation? I can see we are not the first people to hit this issue either, there a few posts on this forum regarding this particular error.

Any of the following actions typically fix the issue (temporarily):

  • Killing the daemon running in the container (ruby). This application has a RSS of just under 100MB
  • restarting the container with balena restart
  • restarting balena with systemctl restart balena

After some debugging the issue has been traced to the Something closing (?) setnsProcess’ parentPipe before it gets to write its config during start()

Final reported error: process_linux.go#L95 which is called by container_linux.go#L263. My co-worker who is currently take a look into this issue is writing notes on it here: https://github.com/HalleyAssist/balena#investigating-the-write-init-p-broken-pipe-error

This issue is not unique to us, a quick google reveals other threads here and github issues. It may not even be unique to balena with some references to a similar error occurring in moby.

4

Hi,

Thank you for this very well written report, I’ll bring this to the team and we’ll get back to you.

Best, Kostas

8

Any updates?

9

Hi @splitice,

We don’t have any updates yet - by the way, what is the device type where you noticed these balena exec issues?

10

Qemu64 (standard) and our NanoPI NEO Air H3 (resin-allwinner repo) devices.

11

Another week down, any update yet? This continues to be a highly disruptive error for us.

root@dcea8b0:~# balena exec -it 0f8df2bae355 /bin/bash
rpc error: code = Unknown desc = invalid character 'e' looking for beginning of value
root@dcea8b0:~# free
             total       used       free     shared    buffers     cached
Mem:        504284     439580      64704      12632       4556      73200
-/+ buffers/cache:     361824     142460
Swap:        65532      62708       2824