Hey @jmptable… Thats a nasty one, its something we have been debugging and working of for a little while. We did a thorough investigation of the issue and found that the reason it was getting filesystem errors was due to the eMMC subsystem being unable to fulfil read/write requests coming from the filesystem. And the reason the eMMC subsystem was failing was because it couldn’t allocate memory, and the reason it couldn’t allocate memory was due to memory fragmentation. This was indicated by dmesg logs like the following…
[ 3511.434696] edma 49000000.edma: edma_prep_slave_sg: Failed to allocate a descriptor
[ 3511.442410] omap_hsmmc 481d8000.mmc: prep_slave_sg() failed
[ 3511.448149] omap_hsmmc 481d8000.mmc: MMC start dma failure
[ 3511.922619] mmcblk0: unknown error -1 sending read/write command, card status 0x900
[ 3511.930544] blk_update_request: 1839 callbacks suppressed
[ 3511.936050] blk_update_request: I/O error, dev mmcblk0, sector 6357616
…and it was further verified by inspecting /proc/buddyinfo.
This issue seems to be happening at least since 2011[1] on beaglebones and keeps happening even with new kernels[2]. The suggested workaround in most threads was to set the sysctl parameter vm.min_free_kbytes to 8192[3]. This causes the kernel to more aggressively reclaim memory and reduce fragmentation. However, this didn’t help and we were still hitting the problem on your device.
You can try the following to fix it:
add the following contents in /etc/sysctl.d/fix-mmc-bbb.conf
vm.min_free_kbytes=8192
vm.dirty_ratio=5
vm.dirty_background_ratio=10
Then reboot the device.
We hope to release new images with a bunch of fixes for this in the coming week or so.
[1] https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/746137
[2] https://groups.google.com/forum/#!topic/beagleboard/tdCUVMicDrk
[3] http://www.keypressure.com/blog/yay-fedora-23-on-beaglebone-black/