We just turned on Deep Packet Inspection on our campus and now as a result I am having connection error problems to my devices. I have installed the certificate which appears successful but there definitely appears to be problems caused that are preventing me from entering the shell of my devices. Any thoughts?
Hi,
It could be that the tool you set up to inspect packages is also blocking traffic from/to one or more of our infrastructure services. Did you try deactivating it to see if it’s actually related to the DPI tool?
Let us know so we can dig more into it!
We have a FortiGate with Deep Packet Inspection turned on. This is some good material on the subject:
I know this is the issue because I had the problems as soon as it was turned on and when I put them on a subnet where DPI was not turned on and I used a laptop to manager them on that same subnet, everything was good. I have gone ahead and placed the sites below as exceptions, but I am not sure that will be accepted by the higher ups.
*.balena-cloud.com*.docker.com*.docker.io
Unfortunately, even with the exceptions, I have a machine that is having difficulty updating.
21.02.19 07:46:40 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/5b07f50ea66dd7232649a9ec8e655525@sha256:397a53450887ca0a06f383058f43c6ebe2bee9589e9ca73dc8bacc39457bbb97' due to 'error pulling image configuration: Get https://resin-production-registry2-cloudformation.s3.amazonaws.com/prod/docker/registry/v2/blobs/sha256/d3/d3a634802d12342356970af07dbfc6fd00bca11fe5387441bc6ea558fb916c35/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPVZA7LDCHZ6P54A%2F20190221%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190221T124640Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=a6ad97fbcd6920cc60bde7f30983d729523f066b2c39f68d7f3b5acfafb340c4: x509: certificate signed by unknown authority'
21.02.19 07:46:41 (-0500) Downloading image 'registry2.balena-cloud.com/v2/5b07f50ea66dd7232649a9ec8e655525@sha256:397a53450887ca0a06f383058f43c6ebe2bee9589e9ca73dc8bacc39457bbb97'
21.02.19 07:46:42 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/5b07f50ea66dd7232649a9ec8e655525@sha256:397a53450887ca0a06f383058f43c6ebe2bee9589e9ca73dc8bacc39457bbb97' due to 'error pulling image configuration: Get https://resin-production-registry2-cloudformation.s3.amazonaws.com/prod/docker/registry/v2/blobs/sha256/d3/d3a634802d12342356970af07dbfc6fd00bca11fe5387441bc6ea558fb916c35/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPVZA7LDCHZ6P54A%2F20190221%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190221T124642Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=679ea715dd165381eb7b97465a52a3e81ff0261693d1d0cfec0e380924b800f1: x509: certificate signed by unknown authority'
21.02.19 07:46:43 (-0500) Downloading image 'registry2.balena-cloud.com/v2/5b07f50ea66dd7232649a9ec8e655525@sha256:397a53450887ca0a06f383058f43c6ebe2bee9589e9ca73dc8bacc39457bbb97'
21.02.19 07:46:44 (-0500) Failed to download image 'registry2.balena-cloud.com/v2/5b07f50ea66dd7232649a9ec8e655525@sha256:397a53450887ca0a06f383058f43c6ebe2bee9589e9ca73dc8bacc39457bbb97' due to 'error pulling image configuration: Get https://resin-production-registry2-cloudformation.s3.amazonaws.com/prod/docker/registry/v2/blobs/sha256/d3/d3a634802d12342356970af07dbfc6fd00bca11fe5387441bc6ea558fb916c35/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPVZA7LDCHZ6P54A%2F20190221%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190221T124644Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=c24ef1eb1b369a136291cc1d0af0dfd869152cc78362a9a5da97493cffa02891: x509: certificate signed by unknown authority'
I am thinking I might also have to add resin-production-registry2-cloudformation.s3.amazonaws.com
as an exception
Hello!
Thanks for getting back to us. Just for clarification, is this the only machine within the same network that’s having issues updating an application with the DPI enabled, or are all balena provisioned devices showing the same issue?
Additionally, are deltas enabled for this device (and for other devices)?
I’ve a few ideas about what may be causing this, but I’d like to ensure I have enough information first.
Thanks and best regards, Heds
I’ve looked into this a bit, and you should be able to add the resin-production-registry2-cloudformation.s3.amazonaws.com as another exception for the download to work. It’d still be good to know if other devices aren’t affected, although I would suspect from the error message that they are.
I’ve also raised an internal issue to improve our docs for network endpoints and DPI cases.
Hope this helps!
Best regards, Heds
I did that and it seems to work OK, but it would be much better to create a targeted firewall rule to allow access to a particular set of services from a particular group of machines. Unfortunately, the firewall rule does not allow wildcard domains. They need to be fqdns or ip addresses/subnets. Is there a list of fqdns for all balena services?
I have also found that it is not possible to enter a shell of a device. I have to go on our guest Wifi, which does not have DPI turned on to enter the shell of our devices. I would love to help you guys work out such scenarios, but I am guessing this does not happen often.
Hello there @rpelletier, thank you for the info. Nice to hear that you have a workaround for now.
We don’t have a full set of FQDNs but we are going to be looking at this fairly soon. We have internal discussions going on to work out example cases and build future docs about the topic.
Your help offer is appreciated! We will get back to you as soon as we have more info or more questions.
Hi @gelbal, I’ve just landed on this topic searching through the forum.
I’m facing an issue with the firewall of a client who is unwilling to open the port on generic domains and is asking for specific FQDN.
Since this topic has been silent in the last months I’ll try to ask and cross my fingers.
Do you have developed a full set of FQDN?
Hi @g-michelle-galindo
unfortunately it does not look like we have that list. One of our engineers is looking into this and will confirm tomorrow. Meanwhile he suggested to use wildcards.
You shlould get a more complete answer and hopefully some helpful advice tomorrow.
Hey @g_michele_morello and anyone else reading this, sorry about the wait. This topic touches many parts in our architecture so it requires further discussion and investigation on our end.
We’re discussing this internally, and we will update this thread when we have some more information.
Have there been any updates on this topic? We’re running into similar issues now for some of our devices.