I’m trying to create a new device,and the following error comes up,
Failed to download image 'registry2.balena-cloud.com/v2/.....@sha256:...'
due to '(HTTP code 500) server error - Get
https://registry2.balena-cloud.com/v2/v2/.../manifests/sha256:..: net/http: TLS handshake timeout '
maybe it’s related to /v2/v2
Hello @danielgnzlzvll could you please clarify where you are seeing this error? Is it a new device that is trying to download the application?
Hi @nazrhom, thanks for your reply.
We have an app for testing, in this app we have a device to verify that everything works correctly before sending it to the production app, in this app, we have created a new device, downloaded the image of balenaOS, we have writed it on the SD card, and we have waited for the device to download the image of the app. The latter after several hours has not worked properly. The following error always occurs.
22.05.19 08:37:44 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:11:58 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/v2/601c9cc0e84f15cf796c0e21f7212187/manifests/sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6: Get https://api.balena-cloud.com/auth/v1/token?account=d_95fd5736453aed163c5718e8aac0244d&scope=repository%3Av2%2F601c9cc0e84f15cf796c0e21f7212187%3Apull&service=registry2.balena-cloud.com: net/http: TLS handshake timeout '
09.09.19 15:12:01 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:12:17 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6: No such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6 '
09.09.19 15:12:22 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:12:32 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: net/http: TLS handshake timeout '
09.09.19 15:12:40 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:12:53 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6: No such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6 '
09.09.19 15:13:09 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:13:23 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6: No such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6 '
09.09.19 15:13:56 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:20:59 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6: No such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6 '
09.09.19 15:22:04 (-0500) Downloading image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6'
09.09.19 15:23:06 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6: No such image: registry2.balena-cloud.com/v2/601c9cc0e84f15cf796c0e21f7212187@sha256:159685922ea94cd578d6aae8e3149a6f072bd451932d0bdec72707b2374444f6 '
Thanks for the follow-up. It seems the device is having troubles downloading the application container, is it on the same network as the previous one? Would it be possible to grant support access to the device experiencing this issue so we can have a look at the supervisor logs and try and figure out what is causing this?
yes.
UUID= 748f535de12534707a27f2aefb0ad1b4
thanks
Hey @danielgnzlzvll the device should have correctly downloaded the application now. It looks like the supervisor was hitting some error which made it unable to pull the image, I simply restarted it and the first pull was successful. I opened an issue on the supervisor repo about this (https://github.com/balena-io/balena-supervisor/issues/1084) so we can hopefully dig further into what may have caused it.
I think there is something more in the background, but thank you very much for your help.
You are welcome Do you want to send me the UUID of the other device so I can have a quick look at that as well?
Hi @nazrhom this is another device with the same problem. We have tried to restart it but it doesn’t fix it.
uiid: 4ad387a9d54dd5459c99d29d3752adc3
Thanks.
Hey @juandhm I tried restarting the supervisor on the device you linked (just to be clear this is not the same thing as restarting the device, sorry if this was unclear) it looks like its making some progress (around 50% now) with the download but it did report the same TLS error once on startup. I have to go soon but another team member or I will have a look at the device tomorrow to be sure the download went through.
We still need to understand what causes this; I attached the github issue to this ticket so that we can notify you as soon as figure out the cause.
Hi @juandhm,
The device you linked is offline at the moment. Then I see that it was running the application fine earlier. So seems like Giovanni’s workaround fixed the issue for now. You could track the above linked github issue to see our progress about this error. We will also send a reply when we close that issue.
Let us know if we could help further.
This a frustrating issue, For more that 3 hours I tried to provision a device and it has not been successful.
18.09.19 14:18:42 (-0500) Supervisor starting
18.09.19 14:19:08 (-0500) Applying configuration change {"SUPERVISOR_POLL_INTERVAL":"6000000","SUPERVISOR_DELTA":"1"}
18.09.19 14:19:08 (-0500) Applying boot config: {"avoid_warnings":"1","fixup_file":"1","hdmi_safe":"1","start_x":"1","disable_splash":"1","dtparam":["i2c_arm=on","spi=on","audio=on"],"gpu_mem":"16"}
18.09.19 14:19:08 (-0500) Applied boot config: {"avoid_warnings":"1","fixup_file":"1","hdmi_safe":"1","start_x":"1","disable_splash":"1","dtparam":["i2c_arm=on","spi=on","audio=on"],"gpu_mem":"16"}
18.09.19 14:19:09 (-0500) Applied configuration change {"SUPERVISOR_POLL_INTERVAL":"6000000","SUPERVISOR_DELTA":"1"}
18.09.19 14:19:10 (-0500) Rebooting
18.09.19 14:20:31 (-0500) Supervisor starting
18.09.19 14:20:33 (-0500) Creating network 'default'
18.09.19 14:20:33 (-0500) Creating volume 'resin-data'
18.09.19 14:20:33 (-0500) Creating volume 'app'
18.09.19 14:20:33 (-0500) Creating volume 'conf'
18.09.19 14:20:33 (-0500) Creating volume 'log'
18.09.19 14:20:33 (-0500) Creating volume 'videos'
18.09.19 14:20:37 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 14:33:03 (-0500) Supervisor starting
18.09.19 14:33:04 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 14:39:35 (-0500) Supervisor starting
18.09.19 14:39:37 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 14:47:22 (-0500) Supervisor starting
18.09.19 14:47:26 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 14:54:28 (-0500) Supervisor starting
18.09.19 14:54:29 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 14:56:11 (-0500) Supervisor starting
18.09.19 14:56:14 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 15:02:55 (-0500) Supervisor starting
18.09.19 15:03:02 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 15:40:27 (-0500) Supervisor starting
18.09.19 15:40:30 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 15:47:36 (-0500) Supervisor starting
18.09.19 15:47:37 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 15:54:27 (-0500) Supervisor starting
18.09.19 15:54:29 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:01:43 (-0500) Supervisor starting
18.09.19 16:01:44 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:08:52 (-0500) Supervisor starting
18.09.19 16:08:58 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:16:44 (-0500) Supervisor starting
18.09.19 16:16:47 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:18:56 (-0500) Supervisor starting
18.09.19 16:18:58 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:30:28 (-0500) Supervisor starting
18.09.19 16:30:34 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:31:40 (-0500) Rebooting
18.09.19 16:33:48 (-0500) Supervisor starting
18.09.19 16:33:51 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:33:55 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f: No such image: registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f '
18.09.19 16:33:58 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:49:40 (-0500) Supervisor starting
18.09.19 16:49:41 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
18.09.19 16:56:42 (-0500) Supervisor starting
18.09.19 16:56:44 (-0500) Downloading image 'registry2.balena-cloud.com/v2/7527ad7835996fa7a9a0dc678674ed5c@sha256:a2c5e7941c5b2e0126642a1e73d9ff9efa74b10db83f033cfd2f785eb937a13f'
This many “supervisor starting” note is not expected, can you maybe enable support access to the device, and posts the UUID here, so our support agents can take a look? It looks as if the supervisor was crashing, and restarting the download…
thet support is enable and the
UUID=86dc2fb88f41aa8789b8f891e9438b28
Checked out the device, and it seems like a watchdog timeout is happening on the device while it is downloading, restarting the container engine and thus starting from the beginning again.
I see that it’s a 2.43.0+rev1 Raspberry Pi3 device, it’s a new release, I’ll check with our OS/Devices team to see what do they think.
In the meantime, if you need to be unblocked, if possible I would recommend downloading a 2.38.0 OS version, and try using that?
And thanks very much for the report, we are looking into this, as soon as the OS time is back to work tomorrow.
Another option, though, is trying a different SD card, if you have one at hand!
The device feels very very slow (in terms of read/write), and it might be because of the SD card, which then would trigger much easer such watchdogs kicking in, and killing the process,
I tried with previus versions too.
I can try other SD, but i’m also interested on change this watchdog, how i can do that?
thaks a lot for your reply.
If the previous version is like that, I would much more suspect the SD card.
Unfortunately the watchdog settings are not adjustable, so won’t be able to give it a longer timeout, etc. I’m asking our team about this in general too, as it seems like your SD card is veeery slow (as I would suspect to be faulty, if it was mine), but it shouldn’t completely prevent the device from working. But will have to wait for their response.
Hi there @danielgnzlzvll,
Would you mind reenabling support access for that device, and give us a head’s up once you do? We’d like to investigate a bit more on device if possible.
Thank you!
Oh sorry, i cant, i deleted the device to minimize the cost. but i can provisioning other with the same SD card or I can report it when the issue is presented again