I’ll add some logging to the
balena_cli node module next. Hopefully that’ll give us some clues as to what’s happening.
@nckswt, I’ve done just that, but so far could not reproduce the issue (assuming that the issue is that files are not being ignored, and instead being added to the tar stream pushed to balenaCloud).
I patched the CLI code to produce extra debug messages as in this diff: https://github.com/balena-io/balena-cli/compare/v11.21.0...investigate-dockerignore-v11.21.0?expand=1
In that patch, I also added a strategic “throw new Error()” to prevent the CLI from actually sending anything to balenaCloud, as I only wanted to test what files are included in the tar stream, and what files are filtered out.
I tested with CLI versions 11.21.0 and 11.25.13. My test project folder container the following test files:
$ find .
docker-compose.yml files have the same contents as the ones you shared above. The result of running the patched CLI was:
$ ./bin/balena-dev push test-rpi --detached --source ~/balena/other/support/nckswt
addIgnoreFile type=0 fullPath="/Users/paulo/balena/other/support/nckswt/.dockerignore"
FileIgnorer filter TRUE relFile=".dockerignore"
FileIgnorer filter TRUE relFile="docker-compose.yml"
FileIgnorer filter TRUE relFile="push-this.txt"
FileIgnorer filter FALSE relFile="data/data.txt"
FileIgnorer filter FALSE relFile="node_modules/modules.txt"
FileIgnorer filter FALSE relFile="data/sub1/sub1.txt"
FileIgnorer filter FALSE relFile="devices/node_modules/node_modules-devices.txt"
FileIgnorer filter FALSE relFile="node_modules/sub2/sub2.txt"
FileIgnorer filter FALSE relFile="robot/node_modules/node_modules-robot.txt"
FileIgnorer filter FALSE relFile="robot/node_modules/sub3/sub3.txt"
tarDirectory packing file '/Users/paulo/balena/other/support/nckswt/.dockerignore'
tarDirectory packing file '/Users/paulo/balena/other/support/nckswt/docker-compose.yml'
tarDirectory packing file '/Users/paulo/balena/other/support/nckswt/push-this.txt'
Note the TRUE and FALSE values above, where TRUE indicates files that are included in the tar stream, and FALSE indicates files that are filtered out. It appears to behave as intended. Something you may also notice in the output above is that excluded folders and subfolders are still traversed, even if all files in them are ultimately ignored. This behaviour is related to the fact that the
dockerignore file allows specifying exceptions to filter rules using the exclamation mark (docs), something like “filter out all node_modules folders, except for this one file which is several subfolders down”. As implemented, each file is matched against each rule. I wonder if this alone could explain the time difference when you delete excluded folders; it’s what James hinted at, but I would expect not, given your
find test results.
I wonder how to more closely reproduce the issue. I’ve created a zip file of the very simple project structure above, which perhaps you could try using / playing with as well – especially if you’re also able to add debug messages: https://drive.google.com/file/d/1bD5gPGcIapa3KDlzZI00pkhmUHIAauDV/view?usp=sharing
Otherwise, maybe you could try sending us your own small zip file with steps to reproduce the issue, including details about your operating system (which version of Linux?). Thank you for helping with the investigation.