I was looking for a way to schedule my multiconainer fleet’s containers. I have 2 usecases :
Container B start after container A exits
Container C start after container B has started
As of today I am rewriting the entrypoint of each image to include some ping. Example for the second usecase : command: bash -c "rc=1;while [[ $rc -ne 0 ]]; do echo 'fluent-bit not started'; nc -vz fluent-bit 24224 &>/dev/null; rc=$?; sleep 1;done;echo 'fluent-bit started'; sleep 1 && /usr/src/app/start.sh"
Which, with correct indentation gives :
rc=1
while [[ $rc -ne 0 ]]; do
echo 'fluent-bit not started'
nc -vz fluent-bit 24224 &>/dev/null
rc=$?
sleep 1
done
echo 'fluent-bit started'
sleep 1 && /usr/src/app/start.sh
I think that you won’t disagree with me that this is far from optimal (even if its sufficient for my basics needs today… ).
Main issue (amongst other) is that the fact that the port is open won’t necessary be sufficient to describe “fluent-bit is started” …
I have spent some time to read about this and will be happy to discuss existance of better solutions with anyone here. Can anyone share good practice about container scheduling in multicontainer balena fleet ? @balena : Is docker 3.X upgrade in the roadmap? Do you have any better solution for the scheduling issue with docker 2.X?
For the second use case, you might want to look at the depends_on keyword, which imposes that a container is running before dependent ones are started: Compose file version 2 reference | Docker Documentation
I am not aware of a contrary keyword existing (Like waiting for X to finish) as most of the time containers are expected to keep running.
About your bash script solution, it is a decent approach, you can probably use it to cover the first use case: blocking containers B and C while A is running (which makes the depends_on solution useless).
What do you mean by “open port won’t be necessary sufficient”? What does it port service should reply? It if is HTTP you can check with CURL, or do some fancy Bash/Netcat-fu to send data and parse an expected answer.
For the second use case, you might want to look at the depends_on keyword, which imposes that a container is running before dependent ones are started
A container started does not necessarily mean that service in this container is up and running.
About your bash script solution, it is a decent approach, you can probably use it to cover the first use case: blocking containers B and C while A is running (which makes the depends_on solution useless).
That’s indeed what i implemented and works OK.
What do you mean by “open port won’t be necessary sufficient”? What does it port service should reply? It if is HTTP you can check with CURL, or do some fancy Bash/Netcat-fu to send data and parse an expected answer.
Sorry if that was not clear.
nc -vz fluent-bit 24224 &>/dev/null
does return an “OK exit code” as soon as port 24224 of fluent) bit container is open. That’s a good start but not sufficient to say “fluent bit is up and running”.
Your proposal to implement a specific route with specific answer seems to answer my needs, even if that will induce some container specific development.
I will take some time soon to test it, although I was wondering how much cleaner would HEALTCHECK and docker3.0 features allow to reach the same goal.
I’m not fully aware of how HEALTHCHECK works as I’ve never used it myself. But from what I understand, you have to provide it with a command-line returning 0 (OK exit status) when your app it healthy, which is exactly what you are proposing with a specific route for your application. The only difference is that HEALTHCHECK command is run periodically, and will trigger your specific response (container restart for instance) if your application is unhealthy.
For instance, the example given in Docker documentation is:
Which will use curl to check the app is running, every 5minutes, and will consider it unhealthy if it takes more than 3 seconds to reply. Resulting only in a new health status indicator, which has to be processed by your orchestrator (mainly useful with Kubernetes I suppose, this part is out of my current skills-set).
I’m not sure this would do anything cleaner for your use-case, as you would have to create a route/specific code to health-check anyway and wouldn’t be able to trigger other containers startup from it (running B when A exits for instance).
Let me know if some parts are not clear, I understand it can be a bit tricky to figure out.
That was pretty clear thank you.
I am not really aware of how far docker can offer such features, that’s what made me doubt of my “rewrite entrypoint” technique, that’s all.
In conclusion, I will have to write specific code to define “container ready AND containerized app fully launched and ready as well”.
Thanks for your time again.
PS: I will try to keep you updated if I find any elegant and more generic way to orchestrate balena container launch.