Hey, we are running into a weird issue, one of our container filled up on /mnt/data/docker/aufs/diff
Similar issue reported on /mnt/data full; what to do?
It’s at about 29GB at the moment and the apps won’t start. Would you know why things would endup in that state?
Thanks,
Brice
@lekkas sure, the box was on Resin OS 2.12.5+rev1
Same here.
One of our devices running Resin OS 2.12.7+rev1 is filling up on /mnt/data/docker/aufs/diff. Then the folder seems to purge itself after a while, only to start filling up again.
All the other devices in our fleet behave as expected and run the same version of ResinOS, as can be seen here:
I just confirmed with a mix of df and du commands that /mnt/data/docker/aufs/diff is the only folder filling up. All other things are equal across the fleet.
Hey @lv82,
Thank you very much for reporting this. This is an issue we’ve been chasing for a while, we’ll follow up over email so that you can share some more information about your application.
@brice I’m also reaching out to you over email as well (since you reported the original issue 
so that you can give us more information about your application. Thanks!
Hey Laurent, what are you using to graph disk usage?
A slightly customized version of the monitoring dashboard from https://resin.io/blog/prometheusv2/
The Disk Fullness metric is provided out of the box with the shipping dashboards.
Awesome, thanks
Hi @brice, @lv82 (and anyone who may bump into this)
After checking @lv82’s affected device, we found out that the underlying cause was an application script that failed to mount an SD card. The application code continued writing data anyway, but that data ended up in the container’s filesystem, (instead of the SD, because of the failed mount), which caused /mnt/data/docker/aufs/diff to fill up.
So it looks like similar application-specific issue patterns are definitely worth looking out for, in cases where device storage is filling up unexpectedly.
Thank you @brice and @lv82 for helping us with the investigation.
@lekkas nice one, yea we do have a script to mount an external drive and record video, probably related.
@lekkas happened again for some reasons, is there an easy way for me to clean up without messing the container?
Thanks!
I just came here to ask a similar question and I saw this thread. Why so are so many layers being kept? It would seem that old layers are not deleted.
total 552
24 drwx------ 133 root root 20480 Jul 25 11:09 .
4 drwx------ 5 root root 4096 Mar 27 17:49 ..
4 drwxr-xr-x 5 root root 4096 Jul 5 13:21 042e3432134fc132f354c6cf216c79fd5a1107c75fd9d1db9c49ada88d5d0c37
4 drwxr-xr-x 12 root root 4096 Jul 5 19:03 04fd37949b5263a0fbf5899fff79e681aed131a722eb7d8982459b42b9e11ea8
4 drwxr-xr-x 12 root root 4096 Jul 4 12:09 05b5259938e6cf99412611e09126dffa6ed9964868c695f933920fbaae3a24c4
4 drwxr-xr-x 12 root root 4096 Jul 5 21:30 081846f8341f96f87d47f9624f871ed355f0c66c92cb7d36c0f38b6470bac287
4 drwxr-xr-x 8 root root 4096 Jul 5 12:34 0cc84f8468ebc3a42ef77927025065c5d3b57826bf046993248f6032bb994c13
4 drwxr-xr-x 8 root root 4096 Jul 5 12:18 11201f04b2b44310f6f4ce3b05020e65d02077267eaf0d373b83d87e6143e159
4 drwxr-xr-x 12 root root 4096 Jul 6 03:55 13fdabca79811ba70609996fc77304f242dc2700f4b65ee11a69b1c0548626b2
4 drwxr-xr-x 5 root root 4096 Jul 5 14:06 1561b399a8bb9ede7b7ad2e9b1af266580a773823689e6e7573e5c74ad2b2505
4 drwxr-xr-x 12 root root 4096 Jul 5 10:16 1671581743a04c22202998385fb4f3687cc45a373441174b9ed3ccc9cc7647ef
4 drwxr-xr-x 21 root root 4096 Jul 4 07:26 1941c2176146e2c31df875169fb795be8fd92deda01265324839fe50422271c2
4 drwxr-xr-x 12 root root 4096 Jul 5 15:31 1976cc59b2250d118425324dc3a449e865e7f8d28e1677420ec64c71b0f080e3
4 drwxr-xr-x 12 root root 4096 Jul 6 01:58 1a0c49f5d3afd4b358521fe747be7e3acd7574c5ff0274d40ac10dbd085e1024
4 drwxr-xr-x 9 root root 4096 Jul 4 07:27 1e97de0340e9c6aeb4eac95c27c010219f31654ceb0f12f561974a350f06d19d
4 drwxr-xr-x 12 root root 4096 Jul 4 10:52 1f5bfa990a85e7ce8bb373ddaca3eef92aee2931a4c8e69fa39050f07b8388b8
4 drwxr-xr-x 12 root root 4096 Jul 4 09:52 21f695901e1bf926d6bef2ae6f6a79857f47d21ba9703ee4a00194f137d30eaf
4 drwxr-xr-x 21 root root 4096 Jul 25 10:54 25e58a1645cd9d0eb5febfca630e5bf7a4b8269fc807fea11d37c82527cd8456
4 drwxr-xr-x 12 root root 4096 Jul 5 09:20 2631933fa6ec5c333943081a8ad9afe5911d2e17ba93fc90c0a84d2e37dbccc9
4 drwxr-xr-x 12 root root 4096 Jul 5 22:44 279984e7d5b1072a4570b5c69e093a9a69f33f76f16de21e89993d167bef71c7
4 drwxr-xr-x 8 root root 4096 Jul 5 12:48 28930b3124dfbe5874fa26bab2be89fafe3e2a3e4eb4a7bd0330411ff90c393a
4 drwxr-xr-x 12 root root 4096 Jul 5 11:27 2a1c61c2847ff0ac41f2d5474f47d518bb995bc5617d8c467c8d7e4bd4986458
4 drwxr-xr-x 12 root root 4096 Jul 5 09:33 2a90132f496207b161404a239d6fe681cd1942a570d0260499ce8a0cce8e9585
4 drwxr-xr-x 12 root root 4096 Jul 5 05:58 2a9fb2053458c5d1533f5ce45c9ce0913d9528b8c1975d2af2a7b88500229ac1
4 drwxr-xr-x 5 root root 4096 Jul 5 14:25 2c8b9a7f2ba8c1e4afe25440d4a9f80cfe7a9b0f5ec3402ca686ff1ee7055fab
4 drwxr-xr-x 20 root root 4096 Jul 4 07:16 2e6fe076b118f0dec809936019602fe100110e6c3a00e1003386866d64f2e80d
4 drwxr-xr-x 8 root root 4096 Jul 5 13:34 30db9fcede3f517e6d2e76df7368f317e9396e3db4fbfb41945b63275f33f37d
4 drwxr-xr-x 5 root root 4096 Jul 5 14:21 315fbcf486d583ded38457b630ed580d231b0b3e286fed4c1ec912822fbfbb1a
4 drwxr-xr-x 5 root root 4096 Jul 5 13:39 32acc92ba09f4f0b6d8b123d945b24b033f4f63e549059830610287a902b4ff5
4 drwxr-xr-x 5 root root 4096 Jul 5 12:58 370094448f5f796726d942a2d79ebe5c4d97b74ce5f01bc26c60ceb920123f7a
4 drwxr-xr-x 8 root root 4096 Jul 5 21:01 378e129ff40df9c2363193f86f482fc6d4924ac2966765e6a01cd4ed332e4795
4 drwxr-xr-x 12 root root 4096 Jul 6 02:18 3806ab40ca36d1d5af0b724b9cafa7f224e28a9f19047c0da93801919b05997b
4 drwxr-xr-x 12 root root 4096 Jul 5 09:58 383974a1fda571b65108319bea57a9c9c83e2aed585872179cf700412cd63050
4 drwxr-xr-x 12 root root 4096 Jul 5 04:21 389b3cee137139db7b886146ba96439041134a39e5845498d7b8662ce2484a2c
4 drwxr-xr-x 12 root root 4096 Jul 4 10:34 3b46c92b53d7d699560431fcc8337d3b41468ff048ec934f85d53f2847ff436e
4 drwxr-xr-x 12 root root 4096 Jul 4 12:36 3c0baaf8250b5bb2e77fc4114906af5080a3be7cd33c19684fec52219f761eba
4 drwxr-xr-x 4 root root 4096 Jul 5 14:33 3f8f1b71d94604410dfe6f1f057eb3b8ef317b3347ad2b2fe17c37571204b00b
4 drwxr-xr-x 12 root root 4096 Jul 4 10:05 40d58b1dcffb89e0545cf63caf2c01d6017447fd53d43266eafc2dccda9daf98
4 drwxr-xr-x 5 root root 4096 Jul 5 13:03 4463bf44170277b94e30035b34a3a2d926c44536a10e18782a2d60629fa826c2
4 drwxr-xr-x 6 root root 4096 Jul 5 13:50 44aef7ed30b74a6993a2545b71992f5e25f125a6cc2922be59acb663383dc6fe
4 drwxr-xr-x 8 root root 4096 Jul 5 13:59 453567e2114f70095ba4252df973d0d1e1dcbbf26f63a6e09f67ae5f9f1f3be6
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 45a9a9da67725733848de3cb078aaf23106608d2e2dbc4475a6298a73d8cee44
4 drwxr-xr-x 12 root root 4096 Jul 5 04:34 46e3bd51c248c35bce0a8cd227925f4bb319894cf45451ed2fce92801da285e8
4 drwxr-xr-x 12 root root 4096 Jul 5 11:02 473c66091aa630043da0024f30f672212d3a6bc508167cb46b592ee29c19a429
4 drwxr-xr-x 12 root root 4096 Jul 4 11:09 47c725cd9982eb297f620cfa9baa0fc78dfb8950d22c8b4f27d5ee1e7a37eb33
4 drwxr-xr-x 12 root root 4096 Jul 5 11:55 4b4cfa48e7cb54849f5f798950df2f47cf75baa2b13b7058ba67f09c1ce3dc4d
4 drwxr-xr-x 3 root root 4096 Mar 27 17:50 4cf1204d54c5e7d67eea705366ad60f9e3251081b904bf32fca85b1d2a200535
4 drwxr-xr-x 8 root root 4096 Jul 5 14:02 4fbc7ef9df8bdc67b62f693b2f0671ac280b496ae5bd35f1dcd44fb03d01fa55
4 drwxr-xr-x 2 root root 4096 Jun 26 23:32 50b9ee66c61b3d4b3cc5964785770489cf768bf83dbb4e4bb4f26943095b069d
4 drwxr-xr-x 12 root root 4096 Jul 5 16:49 56005354ba0b7430a00a8e1932c2257862d27f217da954689ad71238976850ab
4 drwxr-xr-x 12 root root 4096 Jul 6 01:10 57968544a9d12f367c627a3cd5e35b0b5bf1c0fd897302dd3ada7ee6219f815a
4 drwxr-xr-x 12 root root 4096 Jul 5 08:00 58a884a2cb10ad4a8e5aa420792953145574817eb5cb11a48b6954b7dce41a3e
4 drwxr-xr-x 12 root root 4096 Jul 5 08:36 5ec4f218da3e4342e711269b05f5c4ea40d080813bb4f11c30e9e5e3d2c1a6e1
4 drwxr-xr-x 12 root root 4096 Jul 5 05:04 5f605fa46b69f2078922d9df6ef02a6473439c63e0893dfd7bde8ee128e51834
4 drwxr-xr-x 3 root root 4096 Mar 27 17:50 6029249585b9a1b91bd6c328744a5d920a62e13a18d3602b454d7d90f84bb51d
4 drwxr-xr-x 12 root root 4096 Jul 5 17:26 6031719f98a7fde7c6cc4042117c0dcc152589e1a91bedb97958f75feb58dde5
4 drwxr-xr-x 12 root root 4096 Jul 6 01:37 60b0cfd8437423e2d1aed1ba69742749c386d05c64d09f0603e86ae5c710f0c7
4 drwxr-xr-x 12 root root 4096 Jul 4 13:07 6188a42c274af12ef16b808d1bfa6e38b9ac660a05fcb9dfd83924d8b34a26f3
4 drwxr-xr-x 12 root root 4096 Jul 4 11:53 6bfd160520f30e9f99efdb266afb7253090c26cccc265fb90b76b613a82ec532
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 6c69761c7a9c6407da111a4b4525cda31d6a8c9d4677bfb2b185b690ea51ce13
4 drwxr-xr-x 4 root root 4096 Jul 5 14:19 6d895b2f7f1f3c5631e63726f5cc46ef52e3c4b4772eb54d27a6ec0245521c7f
4 drwxr-xr-x 5 root root 4096 Jul 5 13:11 715e48aeadcf65a7fcbc43782ecd7c56e59736dbcea8e687e4d2ecdbd0da14c9
4 drwxr-xr-x 14 root root 4096 Jul 25 14:07 732695148ba81e90369193e08db2dc0b99c8f490f44090003880e1f1e0e11740
4 drwxr-xr-x 6 root root 4096 Jul 25 11:09 732695148ba81e90369193e08db2dc0b99c8f490f44090003880e1f1e0e11740-init
4 drwxr-xr-x 12 root root 4096 Jul 6 04:58 76855621b84da18d4958918e927fa7ddadc7c59beb24bf8c54a447a58b15a7c8
4 drwxr-xr-x 12 root root 4096 Jul 5 07:15 77218d66a3af51ba92a2da02b89e1b05b4dae1531d5d3b14d0f0bf76deed94e9
4 drwxr-xr-x 12 root root 4096 Jul 5 22:24 77480aee57886bedad6e961f4f1dc43a6fc8b179b9807a577499b0922a3ad930
4 drwxr-xr-x 9 root root 4096 Jul 4 07:28 7a758286b3cb2c448f382b2c77b2aa65c6fb9afb9e75f54857dd1182827e9c5f
4 drwxr-xr-x 12 root root 4096 Jul 5 10:39 7b2d4523ae3bb5fd6e32d37811e7dcec6bbe88aebb1adeb65813318b4dcb6246
4 drwxr-xr-x 2 root root 4096 Jul 5 14:44 8382efca74f4bb47af73a898c79784bf98bd402df54513989d698c6c918e1932
4 drwxr-xr-x 12 root root 4096 Jul 5 05:42 86187afdf6d3a7671b3f65dddd0502f8ef59d63b79be657dce5f46aff953a974
4 drwxr-xr-x 12 root root 4096 Jul 6 02:57 8913a4686c4c34ca32364ede2c6c5706f66293ff1529bbd37dbb73665ded89d9
4 drwxr-xr-x 5 root root 4096 Mar 27 17:50 89d2af1ff5778ddb0b86f133c6227530217d2424b4e6d57491b605067b20d453
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 89d51c3821d24698528f86573d4084abb422c50c12c3b51ccecc2047ae3f17b0
4 drwxr-xr-x 5 root root 4096 Jul 5 14:38 8d42e5d9c9ac68654e01a8f37e791fb8a1ff8179390b1cbf60d3603a1672ffbf
4 drwxr-xr-x 8 root root 4096 Jul 4 03:41 8d95914d0a69cb58a0397cb66309381b797834a2728068c94f25e9f15a2c735a
4 drwxr-xr-x 12 root root 4096 Jul 5 22:04 8eefbf77d7955c496bc5872b2d08fbc90a2b49053c25fe9fd71f4a8f44edfcaa
4 drwxr-xr-x 3 root root 4096 Jul 4 07:10 90d82fe637ffedc823c984b356e07837ca1dd278c6c2774c161e9f0e152e5b57
4 drwxr-xr-x 6 root root 4096 Jul 5 13:55 926bf2b1db2913e85c0614c3abff7f667bf055a26a9aa043f7f78a4ec1b0aa57
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 945d303efe784c7b276b4b15ba8ed4c743f0cf123b903b7ecac545685812a57a
4 drwxr-xr-x 20 root root 4096 Jun 29 02:36 94b783f550a1ae4062fa10606fa61f08978ba94675fd92541f506c4b805aa718-removing
4 drwxr-xr-x 3 root root 4096 Mar 27 17:50 9852169bc4daaeee26b581ebf82b1fdf4e1322c045427f5dc60115b0f60a36df
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 9a4dbbac7770ad8b17b95e15a50a6370edbf6ae10e5c85475e6c22c1e83a4098
4 drwxr-xr-x 5 root root 4096 Jul 5 13:42 9cffcc3fc4e09ba0357a6e773570316e186cdf07e716d71ef4b1df0818eb39fe
4 drwxr-xr-x 12 root root 4096 Jul 6 00:14 9e2c7f25881751be98ef7c2d5f3e327f67a109a66d91b238e654bad813ce2930
4 drwxr-xr-x 12 root root 4096 Jul 5 23:28 a00ca7eb868021ea7581919a08f47c00922edbf11cffce075604cca9003c803b
4 drwxr-xr-x 12 root root 4096 Jul 5 17:54 a3a51e1f759461406c85ee6e81713263edca780b5ad1af8eed768530038ddd55
4 drwxr-xr-x 12 root root 4096 Jul 4 12:25 a3d6c00f9e0e64e04528c0e902091fc94939e6c7ede62dbba17d09738f80544d
4 drwxr-xr-x 5 root root 4096 Jul 5 12:41 a411269b1390686cdb9acea4def74280b6397e50915de160e916eab2bd113586
4 drwxr-xr-x 12 root root 4096 Jul 5 08:55 a6a5ff3018316a7454427538b0e43224e0fee13003452f9a737f0ef1846e6ead
4 drwxr-xr-x 8 root root 4096 Jul 5 12:25 a9be5b8c6ce1a82a2cdebfe81d75f6ff53680dbb420181e6f8110232424e2a6c
4 drwxr-xr-x 12 root root 4096 Jul 5 06:30 abfe7f63d23342a74cc3a7a3e4cac6e165e69a4394ef73b09ee9853a66d8acbe
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 ac02fcb3438b946209a45cc9a8426eda71fc6d40504d3d4972d16ec05e7fb511
4 drwxr-xr-x 12 root root 4096 Jul 5 20:18 ac7bbd2b42a6b35125ad0cadd7b27a9c11481b4b0cc9d074ea82ca28d432014c
4 drwxr-xr-x 12 root root 4096 Jul 5 19:14 b0852a1687e8026359dfbe72bb6aec52f52e4e34d06e863f3e5b08328140c96f
4 drwxr-xr-x 8 root root 4096 Jul 5 13:28 b2f1bbdab760237bf0a2f1559bb1f37616db3a7d7a7a9cf0b03d215487bc198c
4 drwxr-xr-x 12 root root 4096 Jul 6 00:40 b30e32e6d1244c188466efb416aef3c2c0cedab0eb6df01777b3d907f320e5bc
4 drwxr-xr-x 5 root root 4096 Jul 5 14:12 b41d6f6756c67e1ff04f1dd6fc19252e5141dd7b928ec9f0e5397203b559d7ce
4 drwxr-xr-x 3 root root 4096 Mar 27 17:50 b510710243346c514e7dbde93e8d0d37f39a95b554d81cbf4919f76436cf2635
4 drwxr-xr-x 5 root root 4096 Jul 5 14:15 ba568d6dfabfd45e34d6baa1e6ccf3e72616aa5bba8db000157a66c8ef6735c9
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 bd1a3d9f7662c84e9355094d5ff4455e5316000ead247cafa84b572efb6a8c13
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 bf3e40f9dcd16d325489875bf653d5f5ecd1f08e397073bd01ad9c2bc81ea815
4 drwxr-xr-x 12 root root 4096 Jul 6 03:30 c07698a793f27f7b7f596c9389afdf59dd7fcbcf4fcf61ae1640ec35f94faa46
4 drwxr-xr-x 5 root root 4096 Jul 5 12:41 c20d1c20bdf87b52ad66876473a350bd30fccdb0147c7df694780c60142ef3ab
4 drwxr-xr-x 12 root root 4096 Jul 5 05:22 c224ddc4492086fa41252c8f65ff771f6132fc1549710374cab6fc491ceb9505
4 drwxr-xr-x 12 root root 4096 Jul 5 15:00 c305e4e3cc2e9555a17a29f8fcb28357808c29b3f3a38eb9909c92e403a9f199
4 drwxr-xr-x 12 root root 4096 Jul 5 07:36 ccb48c8b5f153c75b700f6bdef6566fb2994c5b4d43442004b297536950326ea
4 drwxr-xr-x 12 root root 4096 Jul 5 11:41 cec09ed6e2b4767ab23c4ad801c15ec0dddb203232e4b3801207f761b0e1ff8c
4 drwxr-xr-x 5 root root 4096 Jul 5 14:10 d0108b6d01d4f3138c2b48c91e743ed271cb712ad4b00f3455ed23f05cc7a32c
4 drwxr-xr-x 12 root root 4096 Jul 5 18:23 d2500150bbfd64de603b1119fee963cdcc9b6be30f14c4a46dd4836c30549d4f
4 drwxr-xr-x 4 root root 4096 Jun 26 04:51 d324cfc76705a7c363a1da3ff36c808eb4b2e30b149fcae4575a7574fd281dd0-removing
4 drwxr-xr-x 12 root root 4096 Jul 5 16:22 d3d93f9be7fa2bdb21a8a6395c181dbe21bd206dbe014019c7ffd8c908697457
4 drwxr-xr-x 8 root root 4096 Jul 5 20:35 d60259209849c2b2890a1b564b10d16e8ea9e9b355a2ae9df52806c33ad5b9e5
4 drwxr-xr-x 12 root root 4096 Jul 6 02:37 d662c681ecc65c4d6f224a74309bf22339c7d4fb4767f342c0a1e2c5edd200d7
4 drwxr-xr-x 12 root root 4096 Jul 4 11:32 dec1cc31f02a376ccc2a630855bf87df90ba7dbc3c6add9d882b1a20739658c3
4 drwxr-xr-x 12 root root 4096 Jul 5 06:53 e097a116d4f0f349ca09e5602669276459226950fbc3b00c6bed3e8adb53fdbf
4 drwxr-xr-x 12 root root 4096 Jul 5 23:39 e192fa73718fdd63710acf9751f2cf6b3477a2ed522a5d58b6b4baa9be961f9b
4 drwxr-xr-x 3 root root 4096 Jul 4 07:27 e23eb16b6617d54b6b45d402255171cffaf56a4ab44daf31376c9f933d05a17a
4 drwxr-xr-x 12 root root 4096 Jul 5 09:49 e3895ca17c5ef6da3f6e25d8c6e3d5128b78b6d3e821470d77566c37a499e298
4 drwxr-xr-x 3 root root 4096 Jul 5 04:18 e4a6014e3192dbc5101da7c294c8da054ff0587356b1cf9c80195f96352e5369
4 drwxr-xr-x 2 root root 4096 Jun 29 02:22 e9d8a955ab97552657da9310e3ee66bd7b2c0dd9f18c688a15bb29f40adfd49c
4 drwxr-xr-x 12 root root 4096 Jul 4 12:59 ea75ee72261b464efd3fa85c12c7121efde2938d7b06558c89dc326f7b4e9d37
4 drwxr-xr-x 12 root root 4096 Jul 5 19:45 ead7454ef24ba46c86fca2ea4315f62f96271b91abb993f630ef6f1e84fb60a1
4 drwxr-xr-x 9 root root 4096 Apr 29 11:41 ebb6dff5590bfa8cdbfce3353f465e7c929aaa00a10b52233909d4e9c5aa2dcd
4 drwxr-xr-x 6 root root 4096 Apr 29 11:41 ebb6dff5590bfa8cdbfce3353f465e7c929aaa00a10b52233909d4e9c5aa2dcd-init
4 drwxr-xr-x 12 root root 4096 Jul 6 04:09 ec4665d39d70929d79f71a8aced09141f6c0c2daabab13d9c76651e93246f9d4
4 drwxr-xr-x 17 root root 4096 Mar 27 17:50 f0e8d4902e4fea002c0985d517f0c5f6df8b7befc99d100185ee18d5271fdd70
4 drwxr-xr-x 12 root root 4096 Jul 5 11:14 f2647d8e6937fe8734d8ebc1c1a8a65042cc0b7dce96d3a8b5c7aa430febf7f4
4 drwxr-xr-x 12 root root 4096 Jul 6 05:14 f73940f0656cb694c1506fd7fdc56318fa5924f7c28012daaac1c51774308ea6
4 drwxr-xr-x 9 root root 4096 Jul 4 07:27 fd515885bd83c4cd7efa44381c0768ef27ca5c478b4b8884e3b224f117765e5c
4 drwxr-xr-x 5 root root 4096 Jul 5 13:15 fd66e1608b95366200a2727d66b344735c97f972cb4fdead2e158d2c2059842b
4 drwxr-xr-x 12 root root 4096 Jul 5 15:59 fecde781ac5dbbf6d977e1abf897b4ecf22ad49baa261abf4acbda75d78349a6
4 drwxr-xr-x 12 root root 4096 Jul 6 04:32 ff0a0d70e22212321577d8e02d1ca37b14cacd05246a6035a314e486ad00ab1c
In our case this is a 32GB debug machine our actual hardware is 8GB and there is 22.5GB of layers here (~300mb project)
We’re seeing this as well. /mnt/data/docker/aufs filled the disk with diffs, now updates aren’t able to download.
bump
Hi @SplitIce is your application writing a lot of data in the container?
Can you provide the dashboard link for the device please?
@floion There would be temporary data written, yes. Data that we fully expect to loose on reboot.
edec7670a3f5e81134d6602eee189bb3c7b2f6fe is currently one of the devices that we have not yet cleared.
What do you mean by not yet cleared? How do you clear the data?
On some I’ve rm’ed some using a a really scrappy bash script that looks for dangling containers that contain /run.sh
A couple were just testing devices and were entirely cleared.