Started getting errors - balena.exceptions.RequestError: b'Unauthorized'

Product support
1

Hi,
I’m using the connector Balenablock. Recently I changed the image location to bh.cr/balenalabs/connector-aarch64 from balenablocks/connector:raspberrypi4-64. Unfortunately there was a problem with this image resulting in an ModuleNotFound Errors (I made a post about this on the forum and it seems this is a known problem indicated in the GitHub repo).

No problem I thought, I just changed the image location back to balenablocks/connector:raspberrypi4-64 and did a push up to my fleet which all seemed to work.

However now I’m getting the following rather strange error which I’m not convinced is entirely related just to the connector block. I’ve never seen this before and don’t see that I’ve changed anything my end.

22.11.22 21:24:09 (+0000)  connector  balenaBlocks connector version: 1.1.2
22.11.22 21:24:09 (+0000)  connector  Changing hostname to ec1db25
22.11.22 21:24:09 (+0000)  connector  Generating config
22.11.22 21:24:09 (+0000)  dashboard  Interim dashboard sync skipped: No schema found.
22.11.22 21:24:09 (+0000)  influxdb  [httpd] 172.17.0.5 - - [22/Nov/2022:21:24:09 +0000] "GET /query?db=balena&q=show+field+keys HTTP/1.1" 200 70 "-" "Python-urllib/3.9" 007211f3-6aac-11ed-804d-0242ac110003 865
22.11.22 21:24:11 (+0000)  connector  balenablocks/connector
22.11.22 21:24:11 (+0000)  connector  ----------------------
22.11.22 21:24:11 (+0000)  connector  Intelligently connecting data sources with data sinks
22.11.22 21:24:12 (+0000)  connector  Traceback (most recent call last):
22.11.22 21:24:12 (+0000)  connector    File "/app/./autowire.py", line 96, in <module>
22.11.22 21:24:12 (+0000)  connector      config = autowire.GetConfig()
22.11.22 21:24:12 (+0000)  connector    File "/app/./autowire.py", line 62, in GetConfig
22.11.22 21:24:12 (+0000)  connector      services = self.GetServices()
22.11.22 21:24:12 (+0000)  connector    File "/app/./autowire.py", line 33, in GetServices
22.11.22 21:24:12 (+0000)  connector      device = self.balena.models.device.get_with_service_details(device_id, True)
22.11.22 21:24:12 (+0000)  connector    File "/root/.local/lib/python3.9/site-packages/balena/models/device.py", line 309, in get_with_service_details
22.11.22 21:24:12 (+0000)  connector      raw_data = self.base_request.request(
22.11.22 21:24:12 (+0000)  connector    File "/root/.local/lib/python3.9/site-packages/balena/base_request.py", line 197, in request
22.11.22 21:24:12 (+0000)  connector      raise exceptions.RequestError(response._content)
22.11.22 21:24:12 (+0000)  connector  balena.exceptions.RequestError: b'Unauthorized'
22.11.22 21:24:14 (+0000) Service exited 'connector sha256:3652f9a3f1dc9dd7253dba82b9fac518f78ccd1c874d95c1c25cbc6d190f6768'

I tried creating a new fleet, creating a new device, trying a previous OS version, all to no avail. I even powered up a device in this fleet that I know was working fine and was pinned to a working release but even this is now showing this error!

Does anybody have any ideas what might be causing this or anything I can try?

Many thanks
Mark.

2

I’m wondering if this error means that the ‘connector’ is not able to get data from my sensor which makes its data available on port 7575 (as per connectors requirements) and yet I haven’t made any changes to the sensor code… very strange.

3

Phew looks like a fixed this. In case anybody comes across in a search in the future - I went back to a clean slate, created a new project in PyCharm using my main branch code, setup a new fleet and went back to Python 3.7 images. No errors now, everything is working.

So at the moment I’m not quite sure what caused this but I’ll keeping a close eye on things as I move forward now to see if I can isolate the cause. I’ll update here once I can be certain.

4

Hi @markysparks, glad you got your service working again. When you changed the image location on your app, did you change any other configurations? The balena.exceptions.RequestError looks like an authentication failure to the Balena API, so I’m wondering if maybe you could have mistakenly removed the io.balena.features.balena-api label in your docker-compose. Let us know if this sounds familiar

1 Like
6

Hi @pipex, thanks for the info - great idea to check this.

I checked the project that I’d had the problem with but unfortunately it all seems OK:


  connector:
    image: balenablocks/connector:raspberrypi4-64
    restart: always
    depends_on:
      - influxdb
      - hmt333
    labels:
      io.balena.features.balena-api: '1' # necessary to discover services
      io.balena.features.supervisor-api: 1  # necessary to discover services in local mode
    privileged: true # necessary to change container hostname

This is the same as in my current working project so still a bit of mystery to me why this happened.

7

Hmm, it’s mysterious indeed. Since you point out the issue persisted even for new devices on the fleet, this really points out to some issue with the release, but I’m not really sure what it could be then. Could you maybe download the docker-compose from the failing release and paste it to the ticket? You can remove any sensitive information if you like, as long as you keep the format.

8

Sure, here we are - though this all started when I tried to point the connector image to bh.cr/balenalabs/connector-aarch64 from balenablocks/connector:raspberrypi4-64 though that maybe co-incidence of course. I changed the image location back to balenablocks/connector:raspberrypi4-64 as you can see in this docker-compose file. I don’t see why I would have changed anything else.

Also I’m not sure now if this docker-compose is in exactly the same state when I had the problems now as I needed to get it working again quickly and I have the deleted the old GitHub repo, didn’t think I would be needing it anymore! - sorry.

I do remember checking through the GitHub commits at the time and not seeing any changes to the docker-compose apart from that image location, that’s what I found so strange about this, I really tried to look for some unintended change but couldn’t find anything. It was at the point I went back to a ‘clean slate’ just to get it working again.

I know I also tried going back to previous working releases as well but the problem persisted, which was again very strange.

version: '2.1'
volumes:
  settings:
  dashboard-data:
  sensor-data:
services:

  scheduler:
    restart: always
    build: ./scheduler
    privileged: true

  wifi-connect:
    image: bh.cr/balenalabs/wifi-connect-aarch64
    restart: always
    network_mode: host
    privileged: true
    labels:
      io.balena.features.dbus: "1"
      io.balena.features.firmware: "1"

  hmt333:
    privileged: true
    build: ./hmt333
    restart: always
    expose:
      - '7575'
    volumes:
      - 'settings:/data'

  metoffice-wow-prod:
    build: ./metoffice-wow-prod
    restart: always
    depends_on:
      - hmt333
    volumes:
      - 'settings:/data'

  influxdb:
    image: influxdb@sha256:73f876e0c3bd02900f829d4884f53fdfffd7098dd572406ba549eed955bf821f
    container_name: influxdb
    restart: always
    environment:
      - INFLUX_DATA_DIR=/data
      - PERSISTENT=1
    volumes:
      - 'sensor-data:/data'

  connector:
    image: balenablocks/connector:raspberrypi4-64
    restart: always
    depends_on:
      - influxdb
      - hmt333
    labels:
      io.balena.features.balena-api: '1' # necessary to discover services
      io.balena.features.supervisor-api: 1  # necessary to discover services in local mode
    privileged: true # necessary to change container hostname

  dashboard:
    image: balenablocks/dashboard:raspberrypi4-64
    restart: always
    volumes:
      - 'dashboard-data:/data'
    ports:
      - '3000:80'

  configuration:
    build: ./configuration
    restart: on-failure
    privileged: true
    depends_on:
      - hmt333
    ports:
    - '8080'
    volumes:
      - 'settings:/data'

# The following 'browser section' should be commented out for the HMT datahub
  browser:
    # image: balenablocks/browser:raspberrypi4-64
    # image: bhcr.io/balenablocks/browser-aarch64 # where <arch> is one of aarch64, arm32 or amd64
    image: bh.cr/balenalabs/browser-aarch64
    privileged: true # required for UDEV to find plugged in peripherals such as a USB mouse
    network_mode: host
    environment:
      - LAUNCH_URL=http://0.0.0.0:3000
      - ENABLE_GPU=0
      - KIOSK=1
      - LOCAL_HTTP_DELAY=8
      - PERSISTENT=1
      - WINDOW_SIZE=800,600
      - WINDOW_POSITION=0,0
    volumes:
      - 'settings:/data' # Only required if using PERSISTENT flag (see below)
9

Thanks again for this info. I also don’t see anything wrong with the composition. Unfortunately I don’t have any more ideas here, you mention that you saw the problem even on a new device which made me suspect the release, however that doesn’t explain why rolling back to a working release.

If you see this issue again, please make sure to check the variables BALENA_API_KEY and SUPERVISOR_API_KEY when inspecting the container. You can get that information SSHing into the device and running

 balena inspect $(balena ps -qa --filter 'name=connector_') | jq .[].Config.Env

The BALENA_API_KEY should match what’s in /mnt/boot/config.json. To test the supervisor API key you can use

curl http://127.0.0.1:48484/v1/device?apikey=<SUPERVISOR_API_KEY>

That will give us a better idea of where is the authentication issue.

1 Like
11

That’s very useful info thanks. If I see the problem again I will make sure to check this.

13

I have the same issue.
The device was working fine for 4 months and then the Connector started with the same error as shown here. I believe I’ve had the issue before and the only solution was to create the same project as a new project and flash it to the usb for the device, as mentioned in the solution here.

This time, I’ve been trying to fix the issue, but to no luck.
I can confirm that the BALENA_API_KEY from inspect is the same as /mnt/boot/config.json.
I have tried clearing all data from the device, Rebooting the device, Restarting the connector, Upgrading the OS, Upgrading Supervisor, Running a new build and Pushing it to the device. Even after upgrading the connector, the same error persists.
I can’t get ‘curl’ to work in the Host OS Terminal.