Terminal giving "rpc error" over and over again

rglenn · November 6, 2017, 1:24pm

Hi everyone. I’m trying to use the terminal after pushing an application up, however I’m running into issues. The terminal in the dashboard gives the following in a continuous loop, and never actually gives me a prompt:

Connecting to e96081[REDACTED]...
Spawning shell...
rpc error: code = 2 desc = invalid character 'e' looking for beginning of value
SSH session disconnected
SSH reconnecting...
Spawning shell...
rpc error: code = 2 desc = invalid character 'e' looking for beginning of value
SSH session disconnected
SSH reconnecting...
Spawning shell...

Hardware is a Pi 1 running Resin OS 2.0.8+rev1 (prod)

Has anyone encountered this before? Any suggestions for fixing it? Thanks!

michal · July 19, 2017, 8:37am

Hi Randy,

I sent you a private message asking for some more details.

daghemo · August 6, 2017, 10:40am

Same problem here.

That is, same error reported by @rglenn, using a Raspberry Pi 1 B board with Resin OS 2.0.8.
Also tried with the new 2.2.0, with the same result.
Also tried with resin-cli version 5.10.2 and 6.3.0, with the same result.

Switching back to Resin OS 2.0.6 seems to solve the problem: I can connect to my Raspberry Pi 1 B.

Raspberry Pi 0 W, known to be based on the Raspberry Pi 1 image, is not affected by the problem.
Also, obviously, Raspberry Pi 3 is not affected by the problem.

imrehg · August 6, 2017, 10:48am

hey @daghemo That’s very strange, thanks for letting us know. We’ll be checking it out and try to reproduce the issue. What base image are you using? Is it possible to share the code that you are deploying on the device?

The main difference between 2.0.6 and newer versions is the newer using Docker 17.03.

daghemo · August 6, 2017, 11:36am

Hi @imrehg, my Dockerfile starts with:

FROM resin/%%RESIN_MACHINE_NAME%%-python

Yes, me too I suspect that the error is related to Docker.

imrehg · August 7, 2017, 9:26am

Just set up a Raspberry Pi 1 with resinOS 2.2.0+rev1 and a basic Python project. That did not give me those issues, thus so far cannot replicate this.
Can you share the code that you are deploying, or if you have a device doing that, send us the dashboard link of the device in private message? That would help quite a bit to triage this issue!

phynias · August 9, 2017, 9:56pm

just setup a pi3 with 2.2.0+rev1 and this is happening to me.

EugeneGwon · August 17, 2017, 5:21am

It happens me, too.
after deploy my application, it works very well. but a day after, It shows me ‘rpc error’.
It works again after restart my application.

imrehg · August 27, 2017, 9:25am

@phynias, @EugeneGwon, do you still have these issues? If you hit them, could you send us a heads up here to have a chance to debug when it’s happening? Or if you have a repeatable way to replicate it, that would be also helpful. It’s not great that the application needs to be restarted for any reason like this.

EugeneGwon · August 28, 2017, 2:07am

Hi, @imrehg
I still have this issue. usually, this happens when I sleep/wake up my mac book.
here’s how to :

open multiple application terminals(2 or 3) on resin.io dashboard
sleep my mac book (‘sleep’ menu under Apple icon)
wake up my mac book about 10 mins later
refresh browser; now I can see terminals are trying to respawn shell but failed with an error
(rpc error: code=2 desc=invalid character ‘e’ looking for beginning of value)

most time, at least 1 terminal had succeeded to reconnect, but sometimes, all fail.
If I keep open/close terminal multiple times, then it suddenly succeeded to connect !

here’s my environment :

browser: google chrome 60.0.3112.90
mac book model: mac book air (mid-2011), Sierra
Raspberry Pi 1 model B, USB Wifi

Please let me know if you need more information.

Thanks.

greenwalrus · October 12, 2017, 6:09am

I’m having this consistently happen on my raspberry pi zero w.

I ssh’d in via dev mode, and can reprpdouce with
~# docker exec -it ef802e4585c4 bash -v
rpc error: code = 2 desc = invalid character ‘e’ looking for beginning of value

Trying it without -t causes a full error message to be spit out
~# docker exec -i ef802e4585c4 bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:248: starting container process caused “process_linux.go:100: writing config to pipe caused “write parent: broken pipe””

greenwalrus · October 12, 2017, 6:16am

https://github.com/moby/moby/issues/31383 appears to be related

Running
journalctl -u docker

Shows the following.

Oct 12 06:05:55 919df58 dockerd[653]: time=“2017-10-12T06:05:55.795635713Z” level=error m
sg="Error running exec in container: rpc error: code = 2 desc = invalid character ‘e’ loo
king for beginning of value"
Oct 12 06:05:55 919df58 dockerd[653]: time=“2017-10-12T06:05:55.804043677Z” level=error m
sg="Handler for POST /v1.27/exec/0f8cc2d6819cd45c107da6b771fe463e4b5f6c3217cab116eee03a5f
2e0d0f2b/resize returned error: rpc error: code = 2 desc = containerd: process not found
for container"
Oct 12 06:06:06 919df58 dockerd[653]: time=“2017-10-12T06:06:06.042497940Z” level=error m
sg="Error running exec in container: rpc error: code = 2 desc = invalid character ‘e’ loo
king for beginning of value"
Oct 12 06:06:06 919df58 dockerd[653]: time=“2017-10-12T06:06:06.055061886Z” level=error m
sg="Handler for POST /v1.27/exec/8949392bfeb6556727c1418a877e4859a7c6aeea14857c2c7f8e7cd8
79e1956a/resize returned error: rpc error: code = 2 desc = containerd: process not found
for container"
Oct 12 06:06:35 919df58 dockerd[653]: time=“2017-10-12T06:06:35.159834262Z” level=error m
sg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec f
ailed: container_linux.go:248: starting container process caused “process_linux.go:100:
writing config to pipe caused \“write parent: broken pipe\”””

Given that docker 17.03.1 was released in febuary, kicking up the version may be the easiest way to test the fix.

imrehg · October 12, 2017, 4:12pm

sorry for the issue, could you give our team support access on the device? Will PM you to ask you for your device dashboard URL

JoachimLindborg · November 5, 2017, 8:45am

I just also started to have this problem now. Worked fine last week.
I have Resin OS 2.2.0+rev1 (prod) on a raspberry P1. it has been running for around a week on same software that I deployed on Oct 29th 2017. Application works fine but unable to use terminal. Haven’t tried any fixes yet

royillingworth · November 5, 2017, 9:37pm

I also just started getting this message. Also running Raspberry Pi 1, running Resin OS 2.2.0+rev1(prod). Started in the last few days. Sometimes I can log in, most times not - today not at all.
I have 2 units that are showing this problem. I think (need to find a way to verify) the ones that are failing were built as 2.2.0+rev1 rather than being upgraded to 2.2.0 (originally built with 2.0.0+rev3).

sqweelygig · November 6, 2017, 1:20pm

Would you mind granting support access, under actions, and then letting us know the device URL.

royillingworth · November 8, 2017, 2:05am

Any update here? I gave you 12 hours of access - not sure if anything material occurred?

imrehg · November 11, 2017, 11:11am

Hi, sorry for the delay. Could you grant us a bit longer access, and provide the device’s dashboard URL? Otherwise we can’t know what device to look at.

royillingworth · November 11, 2017, 1:01pm

OK, I have granted support access for 1 week. The UUID of a failing node is 608030f24d65e80db139536335940ad2.
I originally responded to you by email and apparently I have an email configuration issue as you never received it (it had the UUIDs in that email)

PeterKAlvin · December 5, 2017, 2:57pm

Hey Guys

we are getting the same error, after local push:

- Creating 'local-app' container
- Starting 'local-app' container
rdt push failed. Error: Error while starting container local-app: Error: (HTTP code 500) server error - invalid character 'c' looking for beginning of value  Error: Error while starting container local-app: Error: (HTTP code 500) server error - invalid character 'c' looking for beginning of value
    at /usr/local/lib/node_modules/resin-cli/node_modules/resin-sync/build/docker-utils.js:309:15
    at tryCatcher (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/resin-cli/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)