Unable to login to openbalena instance

ericmjl · June 12, 2020, 12:36pm

Hello Balena team!

First off, thank you for putting balena out as an open source project!

I have followed the docs exactly to get Balena up and running on DigitalOcean, but I am always hitting a wall getting stuck at logging in.

In particular, I was most unsure about CNAME records. My CNAME records are set up, and I think they are set up correctly. Base domain is mydomain.org (I am masking the actual domain in this post), and I have CNAME records for api, s3, vpn, and registry on Google Domains, with api pointing to api.mydomain.org etc. SSL is enabled for mydomain.org, which forwards directly to my DigitalOcean box’s IP address.

However, even with all of this set up, I still get the following error logging in.

ENOTFOUND: request to https://api.mydomain.org/login_ failed, reason: getaddrinfo ENOTFOUND api.mydomain.org api.wors
hip-manager.org:443

I understand that the issue may need some digging before I can get past this blocker to using open-balena. What would be the best starting points for debugging? To be clear, I’m not expecting to resolve this quickly, but would like to take this chance to learn more about the right process of debugging the installation process. I might then take the chance to contribute some docs back.

bversluijs · June 12, 2020, 1:58pm

Hi,

I’m not a member of the Balena team, but I’ve been using Balena quite some time and trying things with openBalena. So just sharing my thoughts.

The first thing I would do is check my DNS settings, if they’re correct. First by checking your DNS provider, and then by using ping cmd and check the IP it tries to resolve is the correct IP

ping api.mydomain.org

If it is, my next step would be checking if the docker containers are running on the VPS (or Droplet, in DigitalOcean’s case) by using:

docker ps

If they are, you could check the response from https://api.mydomain.org/. It should show something. If it’s a 503 error, it means. it could reach the haproxy container, if it shows:

Cannot GET /

It means the API is running, but / is not found (404). If that’s the case, the API is online, so probably working fine.

I don’t know if you were looking for these kind of answers. But I hope they help debugging the problem!

JuanFRidano · June 12, 2020, 2:03pm

Hello!
Glad to hear that you are enjoying open-balena.

if you are unsure whether your CNAME record have been set-up correctly, run the following commands from the computer where you are trying to log in.

dig mydomain.org
and
dig api.mydomain.org

if dig api.mydomain.org has no ;; ANSWER SECTION: then the problem relies on your DNS configuration

Remember that CNAME record changes can take up to 72 hours to go into effect.
feel free to post back the output of the dig commands so that we can help you further

ericmjl · June 13, 2020, 3:27am

Thanks for your help, @bversluijs!

I have an intriguing observation. I can ping [mydomain].org, but pinging api.[mydomain].org gives me:

$ ping api.[mydomain].org
cannot resolve api.[mydomain].org: Unknown host

$ curl api.[mydomain].org
curl: (6) Could not resolve host: api.[mydomain].org

At the same time, running docker ps shows me that the necessary containers are apparently running:

root@openbalena:~# docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                                           NAMES
b993d71af1be        openbalena_haproxy                    "/docker-entrypoint.…"   16 hours ago        Up 16 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 222/tcp, 5432/tcp, 0.0.0.0:3128->3128/tcp, 6379/tcp   openbalena_haproxy_1
11c8330cfdd2        balena/open-balena-vpn:v8.10.0        "/usr/bin/entry.sh"      16 hours ago        Up 16 hours         80/tcp, 443/tcp, 3128/tcp                                                                       openbalena_vpn_1
41c0dfae76a3        balena/open-balena-api:v0.19.5        "/usr/bin/entry.sh"      16 hours ago        Up 16 hours         80/tcp                                                                                          openbalena_api_1
219929aec5de        balena/open-balena-registry:v2.11.1   "/usr/bin/entry.sh"      16 hours ago        Up 16 hours         80/tcp                                                                                          openbalena_registry_1
d6518d97720f        balena/open-balena-s3:v2.8.5          "/usr/bin/entry.sh"      16 hours ago        Up 16 hours         80/tcp                                                                                          openbalena_s3_1
398a48e1fe5a        redis:alpine                          "docker-entrypoint.s…"   16 hours ago        Up 16 hours         6379/tcp                                                                                        openbalena_redis_1
1bd8dacab6b5        openbalena_cert-provider              "/entry.sh /usr/src/…"   16 hours ago        Up 16 hours         80/tcp                                                                                          openbalena_cert-provider_1
e30ae561b38d        balena/open-balena-db:v2.0.3          "docker-entrypoint.s…"   16 hours ago        Up 16 hours         5432/tcp                                                                                        openbalena_db_1

Also thanks for responding, @JuanFRidano!

Running the dig commands shows me that I do have an ;; ANSWER SECTION: on both [mydomain].org and api.[mydomain].org. I think the CNAME records are correct, though to troubleshoot, here is the output:

$ dig [mydomain].org
; <<>> DiG 9.9.7-P3 <<>>[mydomain].org
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19807
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;[mydomain].org.           IN      A

;; ANSWER SECTION:
[mydomain].org.    15      IN      A       [CORRECT IP ADDRESS HERE]

;; Query time: 11 msec
;; SERVER: 192.168.2.1#53(192.168.2.1)
;; WHEN: Fri Jun 12 23:23:08 EDT 2020
;; MSG SIZE  rcvd: 64

$ dig api.[mydomain].org

; <<>> DiG 9.9.7-P3 <<>> api.[mydomain].org
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28202
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;api.[mydomain].org.       IN      A

;; ANSWER SECTION:
api.[mydomain].org. 15     IN      CNAME   api.[mydomain].org.

;; Query time: 226 msec
;; SERVER: 192.168.2.1#53(192.168.2.1)
;; WHEN: Fri Jun 12 23:25:06 EDT 2020
;; MSG SIZE  rcvd: 66

Could it be I need to open up some ports or configure NGINX?

bversluijs · June 13, 2020, 8:14pm

Ah, I see the probable cause of the problem.
You’ve ping’ed the api.[mydomain].org and it should return a IP address, but it couldn’t resolve the host. This means it’s a DNS problem. It can’t reach the server because it doesn’t know which server to talk to.

And after the dig commands, the problem is definitely your DNS. You’ve set the following DNS:

api.[mydomain.org] IN CNAME api.[mydomain].org

The CNAME record is just something like: For this domain (api.[mydomain].org), use the same DNS configuration as the value IP (in this case also api.[mydomain].org. So it’s pointing to itself, which stays in a constant loop.

What I like to do when setting up openBalena, is creating a subdomain like:

balena.[mydomain].org

And setting all my A (IPv4 of the server) and AAAA (IPv6 of the server) records to that subdomain. Setting up openBalena with that domain, so:

./scripts/quickstart -U <super_username> -P <super_user_password> -d balena.[mydomain].org

And setting the DNS CNAME’s like this:

api.balena.[mydomain].org IN CNAME balena.[mydomain].org
registry.balena.[mydomain].com IN CNAME balena.[mydomain].org
vpn.balena.[mydomain].com IN CNAME balena.[mydomain].org
s3.balena.[mydomain].com IN CNAME balena.[mydomain].org

You’re probably ready to set-up your devices now!
Also, it could take up to 72 hours for DNS changes to take effect. This is almost never the case and depends on what you’ve set-up in your DNS, but it could take a while.

ericmjl · June 13, 2020, 8:29pm

Thank you, @bversluijs! Setting up my domains was indeed the issue. I ended up using the following setup, which might be non-standard, but at least worked:

I first set up a subdomain, balena.my_special_domain.com, which forwarded to my OpenBalena server on Digital Ocean.

I then added four “A” records for my_special_domain.com:

api.balena
registry.balena
s3.balena
vpn.balena

Under Google Domains, those are the names, and the data all point to my_special_domain.com.

I think this was the only way I could make it work, as I do use my_special_domain.com for other purposes. That said, could it be that I would run into issues downstream? I’m not sure, so I’m open to hearing whether there could be anything bad that happens as a result of using this scheme.

In any case, I’ve solved the login issue, so thank you both for responding and helping! I ran into another issue, which pertains to the device being unable to ping my OpenBalena server, but I will let the problem simmer for a bit first, maybe there’s something I’m missing in my head.