Update: the SD card (8Gb Kingston) shows as being 32Mb in size containing 3Mb worth of files. Scanned it with windows stock disk check and showed now errors. I’ll scan it with something better later but I don’t want to fix the SD and lose my evidence before knowing what caused it.
Thank you for getting back. With the third device (with the same error), are you able to retrieve logs from the device? Any logs from the device will be helpful to debug and understand why are the devices in such state.
As a side measure, if possible try to run some of the device healthchecks (Balena Device Debugging Masterclass - Balena Documentation) which can help pin-point if there is something specific failing. Let us know if something doesn’t work or you are not able to retrieve logs here.
Note that if your application sends anything to stdout, it will end up being picked up by the HostOS and written to logs; if persistent logging is enabled (not sure about the case when it’s disabled), this will write to SD.
If I’m not mistaking, the default configuration for systemd-journald in Balena does not set a minimum disk space to be left free; it does set a limit to the total size of logs.
What you’re seeing in Windows, is only the boot partition, which is the only FAT partition.
From the Windows
Disk Management tool, you should be able to see the other partitions (but won’t be able to use them).
These other partitions are Ext3 and/or Ext4; you will need external tools like Ext2Fsd to see their contents in Windows.
If it’s an option, try to get a Linux host (can be through a live cd/usb) to view/edit them; this will be a lot more stable than doing it through Windows.
Now on to the fun part, issues that have actually bitten me and are easily overlooked
- Have you double checked your configuration against the local conditions?
You said one of your setups was connected through 3G.
In some places 3G is actively being phased out in favor for 4G and 5G since about 2020.
You may want to check some coverage maps.
In the other case you are connected through WiFi, is the router/firewall setup to allow connections?
- Have you double checked your configuration against remote conditions?
Something else that happened around the end of 2020 that could affect things (but probably shouldn’t block access through BalenaCloud) is that LetsEncrypt changed their root certifactes.
This change meant my HTTP traffic still worked, but all HTTPS traffic failed.
- Can you provide more information about this “minor update” you mentioned?
Maybe something changed that affects your DNS settings or the way your modem is initialized?
It happened to me that one of my modems needed a
usb_mode_switch to go into the proper mode and an update of that package actually broke/fixed things.
In a similar way, it has happened to me that
/etc/resolv.conf wasn’t properly populated anymore.
@heinburgh Did you manage to resolve the issue? Let us know if you’re still having problems!
@tjvv Thanks for jumping in to help here too, much appreciated!