Our team has been able to locate and look into the transient GRPC communication issue which occurred on one of our builder instances from 22:00 to 13:00 BST. As it only occurred on one of our build instances it only affected a % of all builds, and fortunately repeat builds seem to have gone through during that timeframe.
We have resolved the issue, and will continue to work to prevent repetition.
I am encountering this error when trying to connect to a container via SSH on one of our devices currently. Can’t say for sure if it is device specific or not yet. But I suspect its a different issue, since the device is generally doing weird stuff, like randomly restarting all containers, currently.
Today I got this message again. Is April 18 a “magic date”?
Looking on masseges in this thread. Seams, every 18th day of each month could be a problem
UPD:
yesterday I got failed build with this message: “E: Some index files failed to download. They have been ignored, or old ones used instead.”
I have tried to build the same app on another balena account - and it was successfull.
I’m so sorry for the inconvenience. Our team has been able to locate and look into the issue. In our builders, docker-containerd process fails, and the parent process (dockerd) fails to detect the failure or respawn the child process, resulting in this rpc error when trying to reach it
We have resolved the issue, and will continue to work to prevent repetition.
Taking a quick look, it seems to be a bug in the docker daemon (or one of it’s supporting modules, such as containerd). A restart of the daemon does fix things, but we’re investigating the root cause too. You should be able to push to any arm host now. Thanks again for the report!
...
[main] ---> Running in d8bc5984986e
[main] Removing intermediate container d8bc5984986e
[Info] Uploading images
[main] grpc: the connection is unavailable
[Success] Successfully uploaded images
[Error] Some services failed to build:
[Error] Service: main
[Error] Error: grpc: the connection is unavailable
[Error] Not deploying release.
Remote build failed
If you need help, don't hesitate in contacting us at:
GitHub: https://github.com/balena-io/balena-cli/issues/new
Forums: https://forums.balena.io