Update: the SD card (8Gb Kingston) shows as being 32Mb in size containing 3Mb worth of files. Scanned it with windows stock disk check and showed now errors. I’ll scan it with something better later but I don’t want to fix the SD and lose my evidence before knowing what caused it.
Thank you for getting back. With the third device (with the same error), are you able to retrieve logs from the device? Any logs from the device will be helpful to debug and understand why are the devices in such state.
As a side measure, if possible try to run some of the device healthchecks (Balena Device Debugging Masterclass - Balena Documentation) which can help pin-point if there is something specific failing. Let us know if something doesn’t work or you are not able to retrieve logs here.
Regards,
N
Note that if your application sends anything to stdout, it will end up being picked up by the HostOS and written to logs; if persistent logging is enabled (not sure about the case when it’s disabled), this will write to SD.
If I’m not mistaking, the default configuration for systemd-journald in Balena does not set a minimum disk space to be left free; it does set a limit to the total size of logs.
What you’re seeing in Windows, is only the boot partition, which is the only FAT partition.
From the Windows Disk Management tool, you should be able to see the other partitions (but won’t be able to use them).
These other partitions are Ext3 and/or Ext4; you will need external tools like Ext2Fsd to see their contents in Windows.
If it’s an option, try to get a Linux host (can be through a live cd/usb) to view/edit them; this will be a lot more stable than doing it through Windows.
Now on to the fun part, issues that have actually bitten me and are easily overlooked 
- Have you double checked your configuration against the local conditions?
You said one of your setups was connected through 3G.
In some places 3G is actively being phased out in favor for 4G and 5G since about 2020.
You may want to check some coverage maps.
In the other case you are connected through WiFi, is the router/firewall setup to allow connections?
- Have you double checked your configuration against remote conditions?
Something else that happened around the end of 2020 that could affect things (but probably shouldn’t block access through BalenaCloud) is that LetsEncrypt changed their root certifactes.
This change meant my HTTP traffic still worked, but all HTTPS traffic failed.
- Can you provide more information about this “minor update” you mentioned?
Maybe something changed that affects your DNS settings or the way your modem is initialized?
It happened to me that one of my modems needed a usb_mode_switch to go into the proper mode and an update of that package actually broke/fixed things.
In a similar way, it has happened to me that /etc/resolv.conf wasn’t properly populated anymore.
Wow @TJvV, thanks for the comprehensive answer. I’ve not spent much more time on this, I sent new SD cards up to the units in question and they’re still going, but probably just a matter of time before the same happens. Sorry for my late reply, this is the first time I’ve been back on here since July. I still have that SD card somewhere, I’ll check it out with Ext2Fsd and post here what I find.
Persistent logging was never activated on any of my devices, so that rules out disk space filling up due to that.
To answer your questions:
-
We’re in Africa, where 3G is still cool. We have 4G in the cities and the tiny mobile routers I use switch between 3G and 4G without issues, and speed is pretty good for what we do, so no issues there. And if a break in connectivity does occur, it always comes back again at some stage.
-
The unit I had handy I was able to give a fresh SD card and that fixed it, so I doubt a security certificate issue was to blame.
-
I can’t remember exactly what the update was but it would truly have been something minor. It was in the way that the units dealt with strings coming from an rfid reader, and again, the issue was resolved with a fresh SD card.
Thanks again for your advice and interest.