We have been seeing this for many months, however with the growth of our app and the supervisor this has become very common recently.
# balena exec -it 2ec2246c603b /bin/bash
rpc error: code = Unknown desc = invalid character 'e' looking for beginning of value
We see this all the time on many devices. When running Balena exec with less than a huge amount of available memory, the exec fails (-t needs to be omitted to get the error message) with ‘rpc error: code = Unknown desc = oci runtime error: exec failed: container_linux.go:263: starting container process caused "process_linux.go:95: writing config to pipe caused “write init-p: broken pipe”’
root@dcea8b0:~# free -m
total used free shared buffers cached
Mem: 504284 445464 58820 10616 6424 112424
-/+ buffers/cache: 326616 177668
Swap: 0 0 0
It’s not exactly memory pressure, but perhaps fragmentation? I can see we are not the first people to hit this issue either, there a few posts on this forum regarding this particular error.
Any of the following actions typically fix the issue (temporarily):
- Killing the daemon running in the container (ruby). This application has a RSS of just under 100MB
- restarting the container with
balena restart - restarting balena with
systemctl restart balena
After some debugging the issue has been traced to the Something closing (?) setnsProcess’ parentPipe before it gets to write its config during start()
Final reported error: process_linux.go#L95 which is called by container_linux.go#L263. My co-worker who is currently take a look into this issue is writing notes on it here: https://github.com/HalleyAssist/balena#investigating-the-write-init-p-broken-pipe-error
This issue is not unique to us, a quick google reveals other threads here and github issues. It may not even be unique to balena with some references to a similar error occurring in moby.