Terminal giving "rpc error" over and over again

Hi, @imrehg
I still have this issue. usually, this happens when I sleep/wake up my mac book.
here’s how to :

  1. open multiple application terminals(2 or 3) on resin.io dashboard
  2. sleep my mac book (‘sleep’ menu under Apple icon)
  3. wake up my mac book about 10 mins later
  4. refresh browser; now I can see terminals are trying to respawn shell but failed with an error
    (rpc error: code=2 desc=invalid character ‘e’ looking for beginning of value)

most time, at least 1 terminal had succeeded to reconnect, but sometimes, all fail.
If I keep open/close terminal multiple times, then it suddenly succeeded to connect ! :slight_smile:

here’s my environment :

  • browser: google chrome 60.0.3112.90
  • mac book model: mac book air (mid-2011), Sierra
  • Raspberry Pi 1 model B, USB Wifi

Please let me know if you need more information.

Thanks.

I’m having this consistently happen on my raspberry pi zero w.

I ssh’d in via dev mode, and can reprpdouce with
~# docker exec -it ef802e4585c4 bash -v
rpc error: code = 2 desc = invalid character ‘e’ looking for beginning of value

Trying it without -t causes a full error message to be spit out
~# docker exec -i ef802e4585c4 bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:248: starting container process caused “process_linux.go:100: writing config to pipe caused “write parent: broken pipe””

https://github.com/moby/moby/issues/31383 appears to be related

Running
journalctl -u docker

Shows the following.

Oct 12 06:05:55 919df58 dockerd[653]: time=“2017-10-12T06:05:55.795635713Z” level=error m
sg="Error running exec in container: rpc error: code = 2 desc = invalid character ‘e’ loo
king for beginning of value"
Oct 12 06:05:55 919df58 dockerd[653]: time=“2017-10-12T06:05:55.804043677Z” level=error m
sg="Handler for POST /v1.27/exec/0f8cc2d6819cd45c107da6b771fe463e4b5f6c3217cab116eee03a5f
2e0d0f2b/resize returned error: rpc error: code = 2 desc = containerd: process not found
for container"
Oct 12 06:06:06 919df58 dockerd[653]: time=“2017-10-12T06:06:06.042497940Z” level=error m
sg="Error running exec in container: rpc error: code = 2 desc = invalid character ‘e’ loo
king for beginning of value"
Oct 12 06:06:06 919df58 dockerd[653]: time=“2017-10-12T06:06:06.055061886Z” level=error m
sg="Handler for POST /v1.27/exec/8949392bfeb6556727c1418a877e4859a7c6aeea14857c2c7f8e7cd8
79e1956a/resize returned error: rpc error: code = 2 desc = containerd: process not found
for container"
Oct 12 06:06:35 919df58 dockerd[653]: time=“2017-10-12T06:06:35.159834262Z” level=error m
sg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec f
ailed: container_linux.go:248: starting container process caused “process_linux.go:100:
writing config to pipe caused \“write parent: broken pipe\”””

Given that docker 17.03.1 was released in febuary, kicking up the version may be the easiest way to test the fix.

sorry for the issue, could you give our team support access on the device? Will PM you to ask you for your device dashboard URL

I just also started to have this problem now. Worked fine last week.
I have Resin OS 2.2.0+rev1 (prod) on a raspberry P1. it has been running for around a week on same software that I deployed on Oct 29th 2017. Application works fine but unable to use terminal. Haven’t tried any fixes yet

I also just started getting this message. Also running Raspberry Pi 1, running Resin OS 2.2.0+rev1(prod). Started in the last few days. Sometimes I can log in, most times not - today not at all.
I have 2 units that are showing this problem. I think (need to find a way to verify) the ones that are failing were built as 2.2.0+rev1 rather than being upgraded to 2.2.0 (originally built with 2.0.0+rev3).

Would you mind granting support access, under actions, and then letting us know the device URL.

Any update here? I gave you 12 hours of access - not sure if anything material occurred?

Hi, sorry for the delay. Could you grant us a bit longer access, and provide the device’s dashboard URL? Otherwise we can’t know what device to look at.

OK, I have granted support access for 1 week. The UUID of a failing node is 608030f24d65e80db139536335940ad2.
I originally responded to you by email and apparently I have an email configuration issue as you never received it (it had the UUIDs in that email)

Hey Guys

we are getting the same error, after local push:

- Creating 'local-app' container
- Starting 'local-app' container
rdt push failed. Error: Error while starting container local-app: Error: (HTTP code 500) server error - invalid character 'c' looking for beginning of value  Error: Error while starting container local-app: Error: (HTTP code 500) server error - invalid character 'c' looking for beginning of value
    at /usr/local/lib/node_modules/resin-cli/node_modules/resin-sync/build/docker-utils.js:309:15
    at tryCatcher (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)

looks like the problem disapperas if we don’t want to write /data from the Dockerfile

Having the same issues… output of the Raspberry PI 3 is completely white, and getting ‘’ when trying to login trough SSH. UUID is: a9f5170544e4e86f18897e0001bd42a5. Granded support access.

Any update on this. I still get the error occasionally and it can be
quite frustrating…

Roy

Internal discussions have yielded a couple of interesting nuggets of wisdom, so I’m passing them on.

it’s not a single issue - any problems that docker has starting the container will appear with that error message

so basically it could mean a bunch of things, like:

  • the image is for a different architecture
  • the CMD or ENTRYPOINT are invalid, not executable, or nonexistent
  • the image has a file at a folder that is bind mounted in the container
  • docker is out of memory

one of the users talked about writing /data from the Dockerfile… that’s one of the things that breaks

2 Likes

Are you still interested in devices doing this just got it again on a device that I really need to get into with a shell.
“rpc error: code = 2 desc = invalid character ‘e’ looking for beginning of value”

Raspberry pi 1
Resin OS 2.2.0+rev1 (prod)
Supervisor Version 6.1.2

Hey @JoachimLindborg, would it work for you updating the resinOS version to the latest one? Would recommend trying that, since the 2.2.0+rev1 release has been quite a while ago and a lot of things changed since.

I’ve got reason to believe this issue is still occurring (we have been seeing it since early 2018 when we started with resin).

I’ve created a new issue with the results of our debugging: Balena exec failures

I hope this helps anyone who finds this thread via Google.

How would one be able to verify it docker is out of image. It seems like there are no docker commands possible from the host.

Hi. In newer resin OS versions we use our own docker container engine called “balena”.