Preloading application - all port bindings fail!

#1

Hey Folks,

I’ve been playing with the resin preload tool. I would love for it to work on my app, but I can’t quite get it to work.

On my application started from the preloaded image, all of the port bindings needed for my app fails! The resin log gives a clue:

Screen Shot 2017-07-08 at 10.00.23 PM.jpg750×445 145 KB

Looks like there was two simultaneous attempt to start the device. That’d explain why all the port bindings seem to fail - the container that I’m ssh-ing may be the one the second one shown on the log.

What do you guys think? Any ideas why this may be happening on the preloaded image?

#3

Hi. Can you detail how you diagnosed your app is not working as expected?
As for the logs shown here, not sure that is really the problem, I think that the problem is just how the supervisor currently displays the logs for a preloaded app but in the background it should be doing the right work. @pcarranzav can you chime in about these logs here?

#4

Hi @daniel.lee - I think it might be indeed that you have a duplicated container issue.
It appears that the supervisor applies the config.txt changes and triggers a reboot, but before the device reboots it actually starts the container. There’s a problem in how we store the device state that might cause an additional container to be started if the device reboots before we store some information about the container into an internal database - this is going to be fixed in resin-supervisor#465 which is a rather large work in progress.
In the meantime, I’ve just PRd this resin-supervisor#472 which should avoid the supervisor starting the container before the reboot, effectively solving this particular issue for you. Hopefully we can get this out soon.

#5

@floion It was a wild guess, but from what I hear from @pcarranzav it might be true.

Thanks for looking at the issue guys - I’ll wait for a new release and let you guys know how it works out.

#6

@pcarranzav

Hey! I tried preloading my application with Resin OS 2.2.0+rev1 (prod) which includes your change!

Sadly, I’m still seeing failed to bind to port XXXX. And I still think that two instance of my application started by the supervisor.

04 PM
Screen Shot 2017-08-02 at 2.38.04 PM.png1348×772 73.7 KB

I still love the idea of preloading and would love to be able to do it for my app! Let me know if I can be useful in getting there.

#7

Sorry @daniel.lee, your post just made me realize that the PR I had made was insufficient. I just PRd this change https://github.com/resin-io/resin-supervisor/pull/484 that now should really fix the issue for you. Hopefully it will be merged and deployed soon.
Cheers! And thanks for your patience :slight_smile:

#8

That’s fanstastic! Appreciate your work :+1:

#10

Hi @daniel.lee, just a heads up that this update is available in the latest 2.3.0+rev1 resinOS version! (changelog)

#11

Can’t wait to try this out!

#12

Have you had a chance to try the release out, and does it fix the problem? We’re really keen to know.

resin.io

#14

Hey @imrehg @Sqweelygig. I tried preloading (resin preload), but it doesn’t seem to work!

Steps:

  1. Download Resin OS 2.3.0+rev1 (prod) from my resin application dashboard
  2. Add a network manager config file to the image to make our 3g modem work
  3. preload using resin preload
  4. Burn the image, insert to device

Observations:

  1. The device shows up in the wrong application (what?). Our team manages two applications; one for prod and another for testing/staging. I download the staging image and preloaded the staging app (i.e. resin preload --app {Staging App Id}), and yet the device shows up as a device in the production app. I have no idea why.

  2. The app fails too boot.

27.07.17 22:23:22 (-0700) Applying config variable RESIN_SUPERVISOR_POLL_INTERVAL = 600000
27.07.17 22:23:22 (-0700) Applied config variable RESIN_SUPERVISOR_POLL_INTERVAL = 600000
27.07.17 22:23:22 (-0700) Applying boot config: {"dtoverlay":"pi3-miniuart-bt"}
27.07.17 22:23:22 (-0700) Applied boot config: {"dtoverlay":"pi3-miniuart-bt"}
27.07.17 22:23:23 (-0700) Rebooting
27.07.17 22:23:17 (-0700) Applying config variable RESIN_SUPERVISOR_POLL_INTERVAL = 600000
27.07.17 22:23:17 (-0700) Applied config variable RESIN_SUPERVISOR_POLL_INTERVAL = 600000
27.07.17 22:23:17 (-0700) Installing application 'registry2.resin.io/XXXstaging/REDACTED'
27.07.17 22:23:18 (-0700) Installed application 'registry2.resin.io/XXXstaging/REDACTED'
27.07.17 22:23:18 (-0700) Starting application 'registry2.resin.io/XXXstaging/REDACTED'
27.07.17 22:23:20 (-0700) Started application 'registry2.resin.io/XXXstaging/REDACTED'
27.07.17 22:23:20 (-0700) Systemd init system enabled.
27.07.17 22:23:20 (-0700) systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
27.07.17 22:23:20 (-0700) Detected virtualization 'other'.
27.07.17 22:23:20 (-0700) Detected architecture 'arm'.
27.07.17 22:23:20 (-0700) Set hostname to <REDACTED>.
27.07.17 22:23:21 (-0700) Killing application 'registry2.resin.io/XXXstaging/REDACTED'
27.07.17 22:23:21 (-0700) Downloading delta for application 'registry2.resin.io/XXX/REDACTED'
27.07.17 22:23:22 (-0700) Failed to download application 'registry2.resin.io/XXX/REDACTED' due to 'getaddrinfo EAI_AGAIN delta.resin.io:443'
30.08.17 11:19:55 (-0700) Application exited 'registry2.resin.io/XXXstaging/REDACTED'
30.08.17 11:19:55 (-0700) Killed application 'registry2.resin.io/XXXstaging/REDACTED'

Notice that staging app is started, killed, and then prod app is downloaded (only to fail!). I’ll be happy to hand over the device url with the permission for you guys to take a look at the supervisor.

#15

Hi @daniel.lee

Yeah it would be good if we could take a look, please can you grant access for 24 hours.

To grant access: navigate to that app or device, find Actions in the sidebar, click “Grant support access”, and choose the timespan. Then please reply to this support message with the URL targeting the app or device to investigate. (For example: https://dashboard.resin.io/apps/11111/devices/22222/summary)

#16

@joe done! I msg you the link to the device.

#19

Hi @daniel.lee,

Actually run across this device today, running the hostOS updates for your fleet.

First, the important thing: are you using iptables in your application? Because it seems like that the iptable rules are changed when the user application starts, and that breaks networking (dns resolution) in the docker containers (both the application and the supervisor).

ResinOS 2.3.0 contains a change that fixes the docker bridge’s IP address. This was needed, because automatic IP for docker0 interface run into issues on some network setup where conflicting IP addresses were assigned to the interfaces. In the new setup that won’t happen. On the other hand, whoever was relying on the previous, automatic setting to be fixed, will have issues. That seems to be happening in your case!
Would recommend looking into your application and before preloading trying to create a regular 2.3.0 or newer device that runs your application correctly!

Regarding the other issue of the device showing up in the wrong application, that’s very strange, and looking into it!