@dtischler et al., thanks for the responses. After more testing I’ve found that the issue may be using the alpine distro balenalib/raspberrypi3-alpine-python vs balenalib/fincm3-python. Performance seems to improve with live push when using the fincm3 image for containers but I will continue to verify. I was able to run 10 flask containers operating at 3 http requests per second each to the same client container at 60% CPU usage, with logging, without issue, which was a big improvement.
Most importantly, live push was much more responsive, even when running all these requests and logging. This leads me to think our use of our alpine distro may have been causing some of the live push build fragility. I tried to rebuild with Alpine and the build would not complete. Shooting in the dark here, really just the results of my heuristic tests.
Also, I believe we’re running into the limitations of http requests. They are running at 30ms not 3ms, which from my research is actually typical. Not the most performant option for inter-container communication for a robotic system and we’re researching moving to Redis which from the official site benchmarks should be closer to 3ms which gives a lot more room in a 20hz cycle.
In general the BalenaFin and Balena OS are a pleasure to work with, really just figuring out the best setup for us in this application. Any insight on the Alpine vs FinCM3 topic would be helpful, and will post final results when we find a stable scenario for our control system.