# supervisor busy.. Cannot reach supervisor

**URL:** <https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640>\
**Category:** General\
**Created:** [July 2, 2021, 2:18pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640 "2021-07-02T14:18:43Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 2, 2021, 2:18pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/1 "2021-07-02T14:18:43Z")

</div>

I am getting this error,

 ![2021-07-02 09_17_58-balena dashboard _ dark-cherry - Brave](https://global.discourse-cdn.com/balena/original/2X/e/e69781f79830144c1fff155a71c7f2bb2364a530.png)

what will cause the supervisor to be busy… I had it working, then it did not want to work

---

<div class="post-metadata">

**Author:** ![zwhitchcox](https://avatars.discourse-cdn.com/v4/letter/z/8e7dd6/32.png) [@zwhitchcox](https://forums.balena.io/u/zwhitchcox)\
**Post date:** [July 2, 2021, 9:34pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/3 "2021-07-02T21:34:45Z")

</div>

Hi, can you please provide more context? Can you show the journal logs for the supervisor with `journalctl --no-pager -a -u resin-supervisor` on the HostOS?

Sorry you’re having trouble.

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 3, 2021, 2:32pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/5 "2021-07-03T14:32:28Z")

</div>

I am having a very hard time trying to sleep the balena fin with the co-procesor. Sometimes it works, sometimes doesnt.

And when it does not work is after i push an update to my code. the balenafin restarts and then when i try to sleep it, it gives the supervisor busy error and it cant go to sleep anymore until i flash another version of the finabler

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 3, 2021, 3:49pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/6 "2021-07-03T15:49:05Z")

</div>

> [@zwhitchcox](#):
>
> journalctl --no-pager -a -u resin-supervisor

 ![2021-07-03 10_48_28-balena dashboard _ dark-cherry - Brave](https://global.discourse-cdn.com/balena/original/2X/f/f89bdf62035119b3376cce4e3a5a1ad6a8c14ba4.png)  
 ![2021-07-03 10_47_40-balena dashboard _ dark-cherry - Brave](https://global.discourse-cdn.com/balena/original/2X/4/41d7e17925d10f073fa8ba24421072e6b78f606c.png)

still same problem with the finabler

This is the most important feature that I need working for my project.

---

<div class="post-metadata">

**Author:** ![cywang117](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/cywang117/32/6307_2.png) [@cywang117](https://forums.balena.io/u/cywang117)\
**Post date:** [July 5, 2021, 11:47pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/7 "2021-07-05T23:47:03Z")

</div>

Hi,

If the Supervisor’s GET /v1/device endpoint is returning anything other than “Idle” for its status field in the response, “supervisor busy…” will be logged. I see that the request to the Supervisor’s API is actually getting rejected for other reasons here though. To debug, could you try running `curl -X GET -H "Content-Type:application/json" "$BALENA_SUPERVISOR_ADDRESS/v1/device?apikey=$BALENA_SUPERVISOR_API_KEY"` from your finabler container to see if the container can access the Supervisor API? This will let us know if the error is in the finabler or the network itself. You can use `balena exec -it finabler /bin/bash` to access its terminal.

Regards,  
Christina

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 6, 2021, 12:31pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/9 "2021-07-06T12:31:20Z")

</div>

> [@cywang117](#):
>
> curl -X GET -H “Content-Type:application/json” “$BALENA\_SUPERVISOR\_ADDRESS/v1/device?apikey=$BALENA\_SUPERVISOR\_API\_KEY”

@cywang117  
I was able to get into the ssh container and did the request you recommended and i got a good response and i was also able to reboot the balenafin from the finabler container via ssh with the command

curl -X POST --header “Content-Type:application/json”   
“[http://127.0.0.1:48484/v1/reboot?apikey=$BALENA\_SUPERVISOR\_API\_KEY](http://127.0.0.1:48484/v1/reboot?apikey=$BALENA_SUPERVISOR_API_KEY)”

any other thoughts?

---

<div class="post-metadata">

**Author:** ![phil-d-wilson](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/phil-d-wilson/32/3730_2.png) [@phil-d-wilson](https://forums.balena.io/u/phil-d-wilson)\
**Post date:** [July 6, 2021, 1:51pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/10 "2021-07-06T13:51:51Z")

</div>

Jorge,

Can you let me know what supervisor version you are running? I’ll try to recreate this issue.

Phil

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 6, 2021, 1:55pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/11 "2021-07-06T13:55:05Z")

</div>

@phil-d-wilson

12.8.10

Please if you get it to work, do some updates on your sample code on your other container (the one that will send the signal) or message me and i can transfer the device to you, because after many updates is when the finabler stopped working and doing the supervisor busy thing.

---

<div class="post-metadata">

**Author:** ![phil-d-wilson](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/phil-d-wilson/32/3730_2.png) [@phil-d-wilson](https://forums.balena.io/u/phil-d-wilson)\
**Post date:** [July 6, 2021, 2:42pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/12 "2021-07-06T14:42:58Z")

</div>

Jorge,

I can’t recreate here. This is the reply from my curl:

```auto
root@d6e683b:/usr/src/app# curl -X GET -H "Content-Type:application/json" "$BALENA_SUPERVISOR_ADDRESS/v1/device?apikey=$BALENA_SUPERVISOR_API_KEY"
{"api_port":48484,"ip_address":"192.168.86.58","os_version":"balenaOS 2.58.3+rev1","mac_address":"B8:27:EB:B0:DC:AB 48:A4:93:03:0C:26 48:A4:93:03:0B:26","supervisor_version":"12.8.10","update_pending":false,"update_failed":false,"update_downloaded":false,"commit":"ab77c473fea5568f102f18f16c7e99e3","status":"Idle","download_progress":null}root@d6e683b:/usr/src/app# 

```

---

<div class="post-metadata">

**Author:** ![phil-d-wilson](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/phil-d-wilson/32/3730_2.png) [@phil-d-wilson](https://forums.balena.io/u/phil-d-wilson)\
**Post date:** [July 6, 2021, 2:43pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/13 "2021-07-06T14:43:15Z")

</div>

Can you please post me your reply from the same curl command?

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 6, 2021, 2:55pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/14 "2021-07-06T14:55:13Z")

</div>

![2021-07-06 09_54_16-balena dashboard _ dark-cherry - Brave](https://global.discourse-cdn.com/balena/original/2X/9/9a846d7717a208d9893be2bbf585da17d6f9e4ed.png)

this is what i have @phil-d-wilson

---

<div class="post-metadata">

**Author:** ![phil-d-wilson](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/phil-d-wilson/32/3730_2.png) [@phil-d-wilson](https://forums.balena.io/u/phil-d-wilson)\
**Post date:** [July 6, 2021, 3:00pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/15 "2021-07-06T15:00:43Z")

</div>

Jorge,

Ah, so your device is “stuck” in the “Downloaded” state. The finabler won’t do anything unless the device is in the “idle” state. Please could I have support access on the device, and I’ll take a look.

Phil

---

<div class="post-metadata">

**Author:** ![phil-d-wilson](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/phil-d-wilson/32/3730_2.png) [@phil-d-wilson](https://forums.balena.io/u/phil-d-wilson)\
**Post date:** [July 6, 2021, 3:58pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/21 "2021-07-06T15:58:04Z")

</div>

Jorge,

Up front: I’ve resolved the problem for you. What I don’t yet know is why the device got into the problem state.

What I found was that there were references to old versions of the finabler block in the local database that the supervisor uses to track state. I removed these old references, and the curl command started returning “idle” for the state, rather than “downloaded”. This allowed the finabler to do it’s job.  
This is not an issue with the finabler - it is working as designed: not trying to sleep or flash the device if it is in the process of updating. This is an issue with the supervisor which we are looking into. I will update this ticket with a way to track this issue, FYI.

Your device is now sleeping soundly. 🙂

Phil

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 6, 2021, 4:03pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/22 "2021-07-06T16:03:32Z")

</div>

Thank you @phil-d-wilson.

Glad you got it solved and rule out the root cause…

---

<div class="post-metadata">

**Author:** ![alanb128](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/alanb128/32/2839_2.png) [@alanb128](https://forums.balena.io/u/alanb128)\
**Post date:** [July 6, 2021, 5:30pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/25 "2021-07-06T17:30:05Z")

</div>

Hi Jorge, we have a GitHub issue to track this: [Incorrect release commit · Issue #1579 · balena-os/balena-supervisor · GitHub](https://github.com/balena-os/balena-supervisor/issues/1579) - I’ve linked it to this ticket so you should be alerted when the issue is closed.

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 7, 2021, 7:04pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/27 "2021-07-07T19:04:20Z")

</div>

Hi @phil-d-wilson

Well I broke it again. I pushed an update to my python code container and broke the supervisor . it is again in “stuck” in the “Downloaded” state.

By the way, since your fix, it worked great.

thanks

---

<div class="post-metadata">

**Author:** ![cywang117](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/cywang117/32/6307_2.png) [@cywang117](https://forums.balena.io/u/cywang117)\
**Post date:** [July 8, 2021, 12:24am UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/29 "2021-07-08T00:24:55Z")

</div>

Hi @jorgesea,

Device dfbe2a2ddcde3e58ff7b0ebd12464673 should have returned to the normal state again. This is definitely a case of the issue described in [Incorrect release commit · Issue #1579 · balena-os/balena-supervisor · GitHub](https://github.com/balena-os/balena-supervisor/issues/1579). The Supervisor currently has a bug that we are looking into, where the device’s Supervisor database contains outdated image entries which result in the device entering an invalid state, as you saw here with `status: Downloading`. I’ve cleaned up some entries in the device database and you should be able to use the sleep endpoint now:

```auto
root@dfbe2a2:~# balena exec -it finabler_3797341_1854038 /bin/bash
root@dfbe2a2:/usr/src/app# curl -X GET -H "Content-Type:application/json" "$BALENA_SUPERVISOR_ADDRESS/v1/device?apikey=$BALENA_SUPERVISOR_API_KEY" | jq .
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 340 100 340 0 0 6415 0 --:--:-- --:--:-- --:--:-- 6415
{
  "api_port": 48484,
  "ip_address": "192.168.86.99",
  "os_version": "balenaOS 2.80.3+rev1",
  "mac_address": "B8:27:EB:8D:51:E8 48:A4:93:02:BC:90 48:A4:93:02:BB:90",
  "supervisor_version": "12.8.10",
  "update_pending": false,
  "update_failed": false,
  "update_downloaded": false,
  "commit": "391ba0abd058f8d6f538f238ff411155",
  "status": "Idle",
  "download_progress": null
}

```

Could I ask what your patterns for pushing application updates are? I haven’t seen this Supervisor issue occur this frequently, so this would help us pinpoint the problem. Thanks!

Regards,  
Christina

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 8, 2021, 1:25pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/31 "2021-07-08T13:25:46Z")

</div>

@cywang117

Thanks Christina.

I have windows. I push the update via balena cli. just the simple  
`balena push `

Is there any other information you want me to check in my computer?

I bet if i push another update to the python container, i might break it again.

---

<div class="post-metadata">

**Author:** ![pipex](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/pipex/32/4840_2.png) [@pipex](https://forums.balena.io/u/pipex)\
**Post date:** [July 8, 2021, 10:08pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/32 "2021-07-08T22:08:33Z")

</div>

Hi Jorge,

Thanks for the information. I’m currently working on this issue and I’m getting close to a fix. Looking at the data shared by my colleagues, this lines up with my understanding of the causes of the issue. I think you are probably making changes to your `tanklevel` service but not making changes to the `finabler` service. While this is perfectly fine, this seems to cause issues for the supervisor to apply the target state under some conditions. If you get stuck in the same state again, I would appreciate if you could share the supervisor logs under those conditions, particularly since you have been able to reliably reproduce this (you can get them using `journalctl -u resin-supervisor -a --no-pager`). If you are in a hurry and need to push changes and not get this issue, you can make sure you make a change to the `finabler` container as well (it can be changing a comment, a version number). In any case I’m sorry about the issues this is causing and we’ll hopefully have a fix soon.

---

<div class="post-metadata">

**Author:** ![jorgesea](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@jorgesea](https://forums.balena.io/u/jorgesea)\
**Post date:** [July 12, 2021, 3:18pm UTC](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640/34 "2021-07-12T15:18:25Z")

</div>

@pipex

Hi Thanks for the help.

So i did this. I updated the tank level container and added a consolo.log line to the finabler. but my python script crashed because of an error in the tank level.

then i updated only the tank level container and it broke the supervisor to the bug hehe.

then i updated both and still the supervisor has the bug. here is the journal

[balena multiple.txt](https://forums.balena.io/uploads/short-url/A6iv24fyHEKx2YYGkZXdtVVL6jx.txt) (78.5 KB)

[Next page](https://forums.balena.io/t/supervisor-busy-cannot-reach-supervisor/333640.md?page=2)
