Supervisor is unable to start service

[bithell]

• Python script initiated a reboot of the entire device using api (it may have done this multiple times)
• Device rebooted
• Supervisor was unable to initialise script again
• I manually rebooted from cloud console
• Device rebooted
• Script started fine

Anyone seen this behaviour before?

[samothx]

Hi @bithell, I find it hard to image how this behaviour would be triggered by the balena env. Could there be any state in the weatherStationCoreLink container that makes it fail to start ?
The way to analyse this would be to look at the device while it is in the failing state. Can you trigger the reboot and does it show that behaviour when you do ?

[bithell]

@hedss

s you did within the container (that ‘GET’ is not a suported method, which is fine as it at least shows there’s something listening on that port), or something like * connect to ::1 port 80 failed: Connection refused which would suggest that the container hasn’t exposed port 80 and therefore can’t be contacted.

If you could share the relevant parts of the Dockerfile you’re using, that would also be extremely helpful!

Hi @samothx - no when I trigger the reboot in the dashboard it starts fine

[jviotti]

Can you please enable support access and share the device url/uuid with us so we can take a closer look?

[bithell]

Sure thing its f453f0e67eb717e9b5930cd9452d2d1f

[jviotti]

Hey there,

Is the device on the problematic state right now? If not, can you make device get into the state where the supervisor fails to start the service? I’d be interesting to check the device logs as that happens

[bithell]

It’s not in that state right now sorry, and not really sure how to get it there - it’s happened a few times today (and yesterday) but I don’t really have a sure fire way to get it there other than by initiating a restart pretty soon after the script starts

[jviotti]

No worries! Please keep an eye on it and once it happens, leave it in that state and ping us here. We are always watching the forums so we’ll be able to jump into the device as soon as it happens again and get all the necessary information.

[bithell]

@jviotti it’s just gone into it now

[jviotti]

Awesome, I’m taking a look now

[jviotti]

Hey @bithell,

I have a theory for what’s going on. It seems that the supervisor starts all the other containers before binding to the HTTP port, so I think that if your container is fast enough, the supervisor API might not be available yet. I’m double checking that this is indeed the case, and if so we can update it to ensure that the API is available before the other containers start.

For now, can you try updating your script to retry various times if the connection is refused, waiting a bit before each retry? I believe that the supervisor will eventually start responding

[bithell]

@jviotti

I have a theory for what’s going on. It seems that the supervisor starts all the other containers before binding to the HTTP port, so I think that if your container is fast enough, the supervisor API might not be available yet. I’m double checking that this is indeed the case, and if so we can update it to ensure that the API is available before the other containers start.

For now, can you try updating your script to retry various times if the connection is refused, waiting a bit before each retry? I believe that the supervisor will eventually start responding

Sure thing - I’ve set it to loop around waiting 10 minutes each time

[bithell]

@jviotti - seems to be stuck in a state of trying to start the service but it can’t. Any chance you could take a look?

[imrehg]

Hey @bithell, can you remind us of the device UUID to check, and enable support access please? Thanks!

[bithell]

Sure thing it’s f453f0e67eb717e9b5930cd9452d2d1f

[CameronDiver]

Taking a look now.

[CameronDiver]

Could you also make sure that support access is enabled, it does not seem to be currently.

[bithell]

Apologies, done now @CameronDiver

[CameronDiver]

Hey, it seems like the device still does not have support access enabled, could you check you enabled it on this device? https://dashboard.balena-cloud.com/devices/f453f0e67eb717e9b5930cd9452d2d1f/summary

[bithell]

Apologies, I’ve tried again