Fin software image updated but not running

Product support
1

Hi,
One of the docker images, fin-debugger, has finished updating, but it won’t start and is missing when running balena ps from host OS. Is there any way to fix this without reflashing? If this happens in the field it won’t be possible to reflash. See below:

Screenshot%20from%202019-05-23%2009-48-09
Screenshot from 2019-05-23 09-48-09.png841×327 19.8 KB

Thanks
Josh

2

Hello,

You can ssh the host OS and try restarting the supervisor from here:
systemctl restart resin-supervisor

5

Hi @zvin,
I have tried that and it does not resolve the issue, I have tried rebooting the device too which also does not resolve it

6

Does anything happen if you press the “play” button next to the container name ?

7

I’ve tried that

9

@joshbot I’ve sent you a PM

12

Thanks, I’m having a look

16

@joshbot do you add or remove iptables rules in one of your containers?

17

@zvin yeah, traffic is being routed from wlan0 to eth0 or wwan0, depending on which is available, since we are using it as a hotspot

18

From the supervisor logs, the issue seems to be

[2019-05-23T10:32:24.308Z] Applying target state
[2019-05-23T10:32:25.017Z] Event: Network creation {}
[2019-05-23T10:32:25.115Z] Event: Service start {"service":{"appId":1309186,"serviceId":259646,"serviceName":"fin-debugger","releaseId":930655}}
[2019-05-23T10:32:25.516Z] Event: Network creation error {"error":{"message":"(HTTP code 500) server error - Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-cfca7e584020 -j DOCKER: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2) ","stack":"Error: (HTTP code 500) server error - Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-cfca7e584020 -j DOCKER: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n (exit status 2) \n    at /usr/src/app/dist/app.js:576:112035\n    at /usr/src/app/dist/app.js:576:111998\n    at m.buildPayload (/usr/src/app/dist/app.js:576:112008)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:576:111508)\n    at emitNone (events.js:91:20)\n    at IncomingMessage.emit (events.js:185:7)\n    at endReadableNT (_stream_readable.js:974:12)\n    at _combinedTickCallback (internal/process/next_tick.js:80:11)\n    at process._tickCallback (internal/process/next_tick.js:104:9)"}}
[2019-05-23T10:32:25.532Z] Scheduling another update attempt due to failure:  900000 { Error: (HTTP code 500) server error - Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-cfca7e584020 -j DOCKER: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
 (exit status 2)
    at /usr/src/app/dist/app.js:576:112035
    at /usr/src/app/dist/app.js:576:111998
    at m.buildPayload (/usr/src/app/dist/app.js:576:112008)
    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:576:111508)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  reason: 'server error',
  statusCode: 500,
  json: { message: 'Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-cfca7e584020 -j DOCKER: iptables v1.6.2: Couldn\'t load target `DOCKER\':No such file or directory\n\nTry `iptables -h\' or \'iptables --help\' for more information.\n (exit status 2)' } }
[2019-05-23T10:32:25.536Z] Apply error Error: (HTTP code 500) server error - Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD -o br-cfca7e584020 -j DOCKER: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
 (exit status 2)

So it looks like the DOCKER chain was removed somehow.
It is needed by docker to function properly ( see https://docs.docker.com/network/iptables/ ).
Is it possible that your code deletes it?

21

Yeah, I thought doing it inside the container would only apply the rules within the container, but it must be system-wide:

echo “Resetting forwarding rules…”

iptables -P INPUT ACCEPT

iptables -P FORWARD ACCEPT

iptables -P OUTPUT ACCEPT

iptables -t nat -F

iptables -t mangle -F

iptables -F

iptables -X

I could do with some way of undoing rules I have applied earlier as before this I am running
echo “Forwarding to $1”

echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -A FORWARD -i wlan0 -o "$1" -j ACCEPT

iptables -A FORWARD -i "$1" -o wlan0 -m state --state ESTABLISHED,RELATED -j ACCEPT

iptables -t nat -A POSTROUTING -o "$1" -j MASQUERADE

When I switch from wwan0 to eth0, or vice-versa, I want to update the ip tables rules accordingly and remove the previous ones that are forwarding to wlan0. This is because when the device switches to ethernet, I don’t want 4G data to be wasted unnecessarily. Is there a better way to reset the forwarding rules and then apply the new ones?

25

Hi, you may need to add cap_add: NET_ADMIN for the container you are making the iptables calls (or make it privileged).

Are you trying to enable Internet connection sharing with those settings on the wlan0 interface? If this is the case you may also try the ipv4 method shared of NetworkManager which could set it all up.

27

Thanks @majorz. I’ve just got it working by using iptables-save to save the state before, make changes and load it back again with iptables-restore if I need to restore it to how it was. It seems to work pretty well. If this still has issues I’ll try using NetworkManager’s method instead.