I have a web application running in a browser, on a balena-based RPI3.
From time to time, the screen shows the “Aw, Snap!” error from Chromium, which means that the browser tab process has crashed.
I have tried to look in the balena dashboard logs, but nothing about the crash is shown. I have also extracted the Xorg.0.log file from the container itself, but it also has no clues for me.
I suspect that the browser tab is either killed because of memory throttling, or because of some issue on the web-app I run. But so far I have traces to follow, to see if I’m right.
I have not been able to reproduce the issue locally, only in the production which is on a remote site.
Details:
Hardware: RevPi 3+
OS: balenaOS 2.53.9+rev1
Browser image: Docker Hub (chromium-browser --version says Chromium 78.0.3904.108)
Web app: An online hosted React based app, communicating via MQTT with other systems.
The balena image on the machine is running 4 services: Browser, MQTT server (mosqitto), a node.js app to turn off display backlight when there’s no activity on MQTT, and a service that receives a webhook from an external device and forwards it to MQTT.
I’m also concerned whether I’m using an obsolete/outdated/broken browser image. I believe that I chose the balenaplayground/balenalabs-browser image because it worked for me and the other images didn’t, but I’m not completely sure. But it bothers me that I cannot find the related github project and I cannot see how the image is composed. Neither balena playground · GitHub or balenaLabs · GitHub seems to have the balenalabs-browser repo.
So maybe i’m better of with trying another browser image? Like the balenablocks/browser which is also used by balena-dash?
Hi @esbenvb, I believe that you’ll be interested in balenaHub, our collection of edge and IoT projects created by fleet owners and product builders. In our block collection you’ll find the browser block, which is available on GH. This is the main idea, to use it as a block of your application.
I am not sure what’s the issue with your application, but you could share the health checks and the device diagnostics in case it insists so that we can take a look and continue the troubleshooting.
Thanks for your responses. I just changed the package to balenablocks/browser, as you both suggested. And updated the OS to balenaOS 2.69.1+rev1 However, the problem persists.
There’s a couple things I’d check first. First of all, I’ve experienced this problem in the past with a browser process killed after an out of memory (OOM) condition. You can check dmesg for something like Out of memory: Killed process to see if this is an issue. On a device with only 1 GB of memory, this is quite likely.
If you don’t see any signs of an OOM condition, I’d suspect a GPU crash. Check about:gpu on your device to see if GPU acceleration is enabled. If it is, disabling GPU acceleration might improve stability at the cost of performance, which may not matter in your situation. You can disable GPU acceleration with the launch flag --disable-gpu.
I have also tried a dmesg, and found something that might be a clue.
At 516289 I get the error Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x0277b6b6
which seems to match the time where the users reported the issue of the “Aw, snap” error page.
At 524136 I restarted the browser package, and at 613656 I disabled the GPU using the flag you just suggested.
So, can the error that I found, be helpful to us in finding the cause?
This looks to be a bug in either Chromium, or the graphics driver. Has disabling the GPU helped at all? Is there a newer version of Chromium available to try?
Sorry for my poor understanding, but I’m not 100% sure how to make sure that I get the right version of the browser image into my container, using docker compose?
Also, if you expose the port in the docker compose as rahul described - have you got your devices public url enabled? If you have got it enabled, then I think you should be able to access the port through http://<PUBLIC_DEVICE_URL>:35173
I also tried port scanning all ports of the device on my LAN and I can easily access other services running on it (MQTT and some node.js HTTP endpoints) but not the debugging port of the browser.
Public device URLs (v.Current) only forward ports 80 and 8080 to the device, so as long as something is listening there on these ports, the request will get through.
Alternatively, if you are on the same network as the device, just change your container to host networking and whatever ports open on it will be accessible via the local LAN IP.
But unfortunately, it still does not give me any response on port 8080. Neither on the public URL or the local 192.168.1.200… Other services like mosquitto works fine on the RPI.
It really looks like chromium is not listening at all.
However I cannot connect to this port from outside, neither using 35173 directly or mapping it to 8080, when accessing the local IP or the public device URL.
Just for the testing, I managed to access the chrome debugger, by doing a balena ssh ... and from the balena machine, SSH out to my server with a tunnel that forwards port 35173, but that’s not a viable solution when debugging in my production environment.
I think you need to add --headless to your flags for it to work.
What we may end up doing is adding a DEBUG envar which will expose the port and add the headless flag when it’s enabled. I’ll try to have another look/test of this when I’m on support later.
In terms of chromium itself, I think that the --remote-debugging-port=35173 flag works as it should. When doing tricks with SSH tunnels, I can access the remote debugger.
My main problem now is that the host OS and the Docker does not expose the 35173 port to the LAN side or to the public URL, even though I try mapping it on port 80 or 8080…
