Layers not caching on Resin build servers

Product support
,
1

I’m trying to figure out why my container layers aren’t caching when I build a container via resin.io. Some layers are particularly hefty (lasting over 15 minutes to build a layer).

I deploy the code to resin via git push resin -f <branch>:master. Here’s sample output from the first few layers of the build:

[Info]     Starting build for nick2/nicktesting, user nick2
[Info]     Dashboard link: https://dashboard.resin.io/apps/232687/devices
[Info]     Building on 'local'
[Info]     Pulling old image for caching purposes
[Info]     Dockerfile application detected
[Info]     Starting Dockerfile build
[Build]    Step 1 : FROM resin/raspberrypi3-buildpack-deps:jessie-20170610
[Build]     ---> 2641bb7dcc50
[Build]    Step 2 : ENV INITSYSTEM "on" TERM "xterm" PYTHONIOENCODING "UTF-8" ROS_DISTRO "indigo" ROS_LANG_DISABLE "genlisp" TEST_WS "/usr/test"
[Build]     ---> Running in 812f50032107
[Build]     ---> c79e007935c4
[Build]    Removing intermediate container 812f50032107
[Build]    Step 3 : ENV TEST_RC "${TEST_WS}/test_install/scripts/onboard/test_setup.sh"
[Build]     ---> Running in 1df3866f806f
[Build]     ---> af0b8fd70a9c
[Build]    Removing intermediate container 1df3866f806f
[Build]    Step 4 : RUN echo "deb http://packages.ros.org/ros/ubuntu trusty main" > /etc/apt/sources.list.d/ros-latest.list
[Build]     ---> Running in a03ded573ce6
[Build]     ---> 93f9d3fd21dd
[Build]    Removing intermediate container a03ded573ce6
...

For comparison, if I run the exact same Dockerfile on a raspberry pi, here’s the output the second time it runs:

Sending build context to Docker daemon  4.096kB
Step 1/20 : FROM resin/raspberrypi3-buildpack-deps:jessie-20170610
 ---> 2641bb7dcc50
Step 2/20 : ENV INITSYSTEM "on" TERM "xterm" PYTHONIOENCODING "UTF-8" ROS_DISTRO "indigo" ROS_LANG_DISABLE "genlisp" TEST_WS "/usr/test"
 ---> Using cache
 ---> 495619d6f7f1
Step 3/20 : ENV TEST_RC "${TEST_WS}/test_install/scripts/onboard/test_setup.sh"
 ---> Using cache
 ---> d71f70e0f167
Step 4/20 : RUN echo "deb http://packages.ros.org/ros/ubuntu trusty main" > /etc/apt/sources.list.d/ros-latest.list
 ---> Using cache
 ---> 7e7c3cf1d868

So, this is the same Dockerfile, but it doesn’t seem to be caching through resin, but is caching when building manually. I do see the following at the end of the resin build:

[Info]     Uploading failed build to registry for caching purposes...
**[**[==================================================>] 100%

What can I do to figure out what’s going on and why caching isn’t working? Is there something I’m missing?

3

Hey Nick,

Don’t worry, you’re not doing anything incorrectly! We (ahem, well me…) updated the code which handles caching to speed things up and make it a bit cleaner. Unfortunately there was an edge case on failed builds which caused some of them to not be used as caching. This has now been fixed and should be making its way to production in the next day or two!

Cameron

4

Thanks for the heads-up! Mind posting an update when it’s been pushed to production?

5

I’ve already requested to be notified so I can let you know :slight_smile:

6

Hey @nckswt

Just to let you know that fix is fully tested and in production now. Please let me know if you find any issues!

Thanks,
Cameron

7

Awesome! Yup, looks like it’s working just fine. Thanks for the info and the fix! :smile:

8

Hi,
I’m not sure if it’s related, but in the Dockerfile below, I’ve just added a node package (in the 3rd RUN below) and it has triggered a full rebuild (more than 7 min).

# base-image for node on any machine using a template variable,
# see more about dockerfile templates here:http://docs.resin.io/pages/deployment/docker-templates
# Note the node:slim image doesn't have node-gyp
FROM resin/%%RESIN_MACHINE_NAME%%-node:6

# Defines our working directory in container
RUN mkdir -p /usr/src/app/
WORKDIR /usr/src/app

# Install dependencies
RUN apt-get update \
 && apt-get install -y apt-utils \
 && apt-get install -y \
      git-core \
      build-essential \
      gcc \
      python \
      python-dev \
      python-pip \
      python-virtualenv \
      python-rpi.gpio \
      python3-rpi.gpio \
      --no-install-recommends \
 && rm -rf /var/lib/apt/lists/*

# This install npm dependencies on the resin.io build server,
# making sure to clean up the artifacts it creates in order to reduce the image size.
RUN JOBS=MAX npm install -g \
      node-red \
	  node-red-contrib-resinio \
	  node-red-node-pisrf \
 && npm cache clean \
 && rm -rf /tmp/*

# This will copy all files in our root to the working  directory in the container
COPY ./app ./

# Enable systemd init system in container
ENV INITSYSTEM=on

# server.js will run when container starts up on the device
CMD ["bash", "/usr/src/app/start.sh"]
10

Hey @Tristan107

I’ve tried to reproduce the issues that you’ve been having, but have been unable to. It’s quite likely that this is a one-off issue, as if for any reason the builder is unable to pull the cached image (for example because of high registry load) it will continue with the build anyway.

In these situations you as a user have the choice of restarting the build to try and re-download the cache, or to continue without any cache.

I think that what you experienced was most likely an instance of this, but if this problem continues, please do make a new thread and I’ll be more than happy to get to the bottom of what is going on.

In the meantime, I’ll add a message to the builder which details if a cached image was not pulled correctly, to avoid confusion in the future.