Faster Docker Builds on CI

Docker layer caching ​

What's Docker layer caching and why are subsequent builds of the same image usually faster, even if some things like the sources of your app changed?

Docker layer caching reuses unchanged steps from previous builds, but only if earlier layers stay the same. Since Docker processes instructions top-down, changing an early step invalidates the cache for all later layers.

Fix layer order ​

First of all, you want to make sure that the layer order doesn't interfere with efficient layer caching. Keep stable steps (like dependency installs) at the top and the ones that regularly change (like your app's code) at the bottom for faster builds.

I've often seen frequently changing steps way too far up at the top in a Dockerfile. This will make builds slow, even on your local machine.
Example: If you COPY the sources of your app into the image before installing required packages, Docker will be forced to repeat the installing packages part, which is usually quite slow, every time even a single line changes in your app's sources.

FROM ruby:3.2-alpine
WORKDIR /app

# Copy all source files,
# including frequently changing ones,
# into the image
COPY . . 

# Install gems, this will re-run
# every time ANY file changes.
RUN bundle install

CMD ["ruby", "app.rb"]

Think about the underlying frequency of changes for each step in your Dockerfile. Arrange the steps in your Dockerfile from least likely to change to most likely to change. This will enable Docker to do layer caching as efficient as possible, speeding up your local and CI builds.

Here's a revised version of the example from above, with a few adjustments:

• Gemfile and Gemfile.lock – ie. the list of ruby packages to install; this is similar to package-lock.json in a JavaScript project, or a requirements.txt file from a Python package – are copied into the image on their own.
  Packages change infrequently. Separating them from the rest of the app's sources enables Docker to cache the installed packages. Changes you make to the app's sources, which probably happen frequently, won't invalidate the cached packages layer anymore. As long as we don't add, update or remove packages, Docker will rely on the layer cache.
• The app's sources are copied into the image last, after packages were installed.

We don't have to wait for packages to be downloaded, built and installed every time we build a new image now. Yay!

FROM ruby:3.2-alpine
WORKDIR /app

# Copy only Gemfile and Gemfile.lock first
COPY Gemfile Gemfile.lock ./

# Install gems (this layer is cached unless Gemfile changes)
RUN bundle install

# Copy the rest of the source code
COPY . .

CMD ["ruby", "app.rb"]

Yet this alone is usually not enough to make your CI builds go faster.
You may still notice that Docker rebuilds the entire image, executing all the steps, every time, on each CI run, regardless of something having changed or not.

At this point, slow CI builds are likely caused by layer caching not working on your CI environment.

Get layer caching on CI ​

The easiest and quickest way to get layer caching working on CI is to use inline caching. Docker will include the information needed for caching individual layers with your image. That's nice, because you do not need any special tools or dedicated cache storage with inline caching. You just build your image as usual, and push it to your registry.
The only thing needed for making this work is a little adjustment to the docker build command, telling Docker to use inline caching, and making sure a previously built image is available on the CI worker by pulling it first.

stages:
  - build

variables:
  COMMIT_REF_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  LATEST_IMAGE: $CI_REGISTRY_IMAGE:latest

build:
  # Build Docker image for Merge Requests _and_ default branch
  stage: build
  image: docker:cli
  services:
    - name: docker:dind
  before_script:
    # Login to registry - using GitLab's own registry here:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    # Pull images for caching:
    # * latest image for first MR run, builds on default branch
    - docker pull $LATEST_IMAGE || true
    # * image for the current MR to speed up subsequent builds
    - docker pull $COMMIT_REF_IMAGE || true
  script:
    # Fun part - mind cache-to and cache-from options
    - docker build --cache-to type=inline --cache-from="$LATEST_IMAGE" --cache-from="$COMMIT_REF_IMAGE" -t "$COMMIT_REF_IMAGE" .
    - docker push "$COMMIT_REF_IMAGE"
  rules:
    # run on merge requests
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    # run on main branch
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      
push-latest:
  # On default branch: tag image "latest" and push to registry
  stage: build
  image: docker:cli
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker pull $COMMIT_REF_IMAGE
    - docker tag $COMMIT_REF_IMAGE $LATEST_IMAGE
    - docker push $LATEST_IMAGE
  rules:
    # only on main branch
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: on_success

Conclusion ​

Docker layer caching speeds up builds by reusing unchanged layers. For best results, order your Dockerfile from least to most frequently changing steps, and use inline caching in CI. Small changes can cut build times dramatically.

FAQs ​

Q: Why does Docker rebuild everything when only one file changes?
A: Docker caches layers sequentially. If an early step (like COPY . .) changes, all later layers must rebuild. Move stable steps (e.g., dependency installs) to the top.

Q: How can I check if a layer is cached?
A: Run docker build—cached layers show Using cache. Use --no-cache to force a full rebuild and compare times.

Q: Does inline caching work everywhere?
A: Yes, inline caching works with most CI platforms (GitLab, GitHub Actions, etc.), as long as your registry is accessible. If not, you might want to consider external cache storage.