Why Your Docker Build Is Slow - Layer Caching Internals

You change one line in your app. Just one line. A comment, maybe you're documenting something you figured out. You save the file, trigger a build, and then you wait. Four minutes. Four minutes while npm downloads the same 245 packages it downloaded yesterday. The same ones it downloaded the day before that.

And the worst part? You've been told the fix. Move your COPY above the RUN. You've done it. It helped, sort of. But somewhere along the way, a new service got added, someone restructured the Dockerfile, and suddenly you're back to four-minute builds and nobody knows exactly why.

The reason nobody knows is because nobody explained what Docker is actually doing. Not the abstraction, what it's actually doing. What it compares. What it stores. What "cache" even means in this context.

That's what this blog is about.

TL;DR

• A Docker layer is a filesystem diff stored by its SHA256 hash, not an instruction, not a timestamp.

• When a layer's hash changes, every layer below it in the stack rebuilds automatically. This is the cascade.

• COPY . . before your install step hands Docker a hash that changes every time any file in your project changes, including comments.

• The fix is ordering instructions from least-likely-to-change to most-likely-to-change. That's physics, not preference.

• BuildKit cache mounts add a second level: even when the layer cache misses, your package manager's download cache survives.

Part 1: What is a Layer, Actually

Before we look at why things go wrong, we need to understand what a layer is. Not the tutorial version. The real version.

The common explanation is: "each instruction in your Dockerfile creates a layer." That sentence is technically accurate and practically useless. It tells you what, not what it means.

Here's what it means.

When Docker finishes executing a RUN instruction, it doesn't take a snapshot of the whole filesystem. That would be enormous. Instead, it walks the filesystem, finds every file that was added, modified, or deleted since the previous layer, and compresses that set of changes into a tar archive. Just the diff. Not the full system, just what changed.

That archive is the layer.

And then it does something important: it computes a SHA256 hash of that archive's contents. Not of the instruction string. Not based on when you built it. Based on the actual bytes of the files inside. That hash becomes the layer's identity, its name, essentially.

Docker stores every layer on disk at /var/lib/docker/overlay2/, in a directory named after that hash. Content-addressable storage. If two different images happen to share the same base Ubuntu layer, the same bytes, and the same hash, they share one directory on disk, not two.

The cache is just Docker asking one question before every layer: do I already have a directory in overlay2 with this exact hash? If yes, skip. If no, build it, hash it, store it.

A layer is not an instruction. It's a filesystem diff with a SHA256 fingerprint. The cache is Docker asking: "do I already have a layer with this hash?"

That's the whole mechanism. Everything else is a consequence of it.

Part 2: The Cascade

Once you understand what a layer is, the cascade rule makes complete sense.

If any layer's hash changes, every layer below it in the stack is automatically invalid. Not as a policy choice. Not something Docker decided. It's a mathematical consequence: each layer's hash is computed from its own content and a reference to its parent's hash. Change the parent, and the child's hash changes even if the child's files are identical. Change the child, and the grandchild changes. It cascades all the way down.

This is why your instruction order matters. It's not a Dockerfile style guide. It's the shape of a hash chain.

Now, how exactly does Docker decide if a specific layer's hash has changed? It depends on what kind of instruction you're looking at.

For a RUN instruction, Docker compares the instruction string character by character. Change one space, cache miss. For COPY and ADD, it goes deeper, it computes a checksum of the actual file contents being copied. If any file changed by even a single byte, the hash changes.

Sit with that second one for a moment.

COPY . . copies your entire project directory. Every file. So if Docker is computing a checksum of everything in that directory, then any change to any file, your app logic, your test files, your .gitignore, a trailing newline someone accidentally added to the README, changes the hash. Cache miss. And everything below that COPY in the Dockerfile? Rebuilt. Every time.

Part 3: Watching It Happen

I don't want you to take my word for this. Docker tells you exactly what it's doing, if you ask it the right way.

Here's a Node.js Dockerfile written the way most tutorials write it:

# Dockerfile.bad — the naive approach
FROM node:22-alpine

WORKDIR /app

COPY . .           # copies everything first
RUN npm install    # installs after

EXPOSE 3000
CMD ["node", "index.js"]

Build it with --progress=plain. This flag makes BuildKit show you its full decision log instead of the pretty summary. Every step. Every cache decision. This is the flag you should use whenever a build is confusing you.

docker build --progress=plain -t demo-bad -f Dockerfile.bad .

Cold cache, everything ran. 22 seconds to run npm. Fine. Now make the smallest possible change and rebuild:

echo "// updated" >> index.js

docker build --progress=plain -t demo-bad -f Dockerfile.bad .

Look:

• no label = executed = miss

• CACHED = hit

There it is. FROM - cached. WORKDIR - cached. Then COPY . . - miss. One comment in index.js changed the checksum of the directory, which changed the layer hash, which meant Docker had no matching entry in overlay2, which meant it rebuilt the layer. And then npm install, whose parent layer is now different, gets rebuilt automatically. It has no choice. The chain broke.

38 seconds. 215 packages. For a comment.

38 seconds. 215 packages. For a one-line comment change. That's not npm being slow - that's your Dockerfile order turning a guaranteed cache hit into a guaranteed miss.

🟢 Try it yourself

Write that bad Dockerfile for any project you have. Run the build twice - the second should be nearly instant. Then make any trivial change to a source file and rebuild with --progress=plain. Watch COPY miss and drag npm install down with it.

echo "# test" >> index.js
docker build --progress=plain -t demo-bad -f Dockerfile.bad .

The output makes the problem visible. CACHED is green. Everything else is your build time.

Part 4: Opening the Layer Store

Most Docker content stops at the build log. But I want to show you what actually lives on your disk, because once you've seen it, the model becomes impossible to forget.

First, check what storage backend you're actually running:

docker info | grep -A2 "Storage Driver"

The answer determines which path to look at. There are two possibilities.

If you see Storage Driver: overlay2 - You're on the classic Docker backend. Your layers are stored as directories in /var/lib/docker/overlay2/, each named by their SHA256 hash:

sudo ls /var/lib/docker/overlay2/ | head -5
3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b
7d3a2e1f8b4c9d6e5f2a1b0c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2
a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

Each directory is a layer. The name is the SHA256 hash - no metadata file, no database lookup. The name is the fingerprint. Go inside one:

sudo ls /var/lib/docker/overlay2/3a36935c.../
diff/   link   lower   merged/   work/

diff/ is the layer itself, the filesystem delta. Only the files this instruction added or changed. Go inside it:

sudo ls /var/lib/docker/overlay2/3a36935c.../diff/
usr/  etc/  var/  tmp/

Those are the only files this layer added or modified. Not the full filesystem, just the delta from its parent. The layer literally is a diff directory.

The lower file is the parent chain:

sudo cat /var/lib/docker/overlay2/3a36935c.../lower
l/6Y5IM2XC7TSNIJZZFLJCS6I4I4:l/ANOTHER_PARENT_HASH

That colon-separated list is what makes cascade invalidation unavoidable; change a parent's hash, and every child that references it now has a broken reference, so its own hash changes too. The merged/ directory is what a running container actually sees. OverlayFS stacks all the diff/ directories, lower layers read-only, the container's writable layer on top, and presents a single coherent filesystem. The container has no idea it's looking at ten stacked diffs. It just sees /usr and /app and everything it expects.

If you see overlayfs with driver-type: io.containerd.snapshotter.v1 - You're on the modern containerd image store, Docker's default since Engine 25+. The /var/lib/docker/overlay2/ path doesn't exist on this setup. BuildKit manages its cache under /var/lib/docker/buildkit/:

sudo ls /var/lib/docker/buildkit/
sudo ls /var/lib/docker/buildkit/containerd-overlayfs/

The key difference: instead of directories named by hash, containerd uses metadata_v2.dba SQLite database to track the snapshot chain. Same content-addressable model, same OverlayFS kernel primitive underneath. The layer relationship is recorded in a database rather than a lower file. The cachemounts directory is exactly what it sounds like; this is where RUN --mount=type=cache stores its persistent directories between builds.

To inspect BuildKit's cache directly on this backend:

# Equivalent to browsing overlay2/ on the classic backend
docker buildx du

# See individual cached build steps
docker buildx du --verbose

One command that works on both backends:

Regardless of your storage driver, this is the clearest proof of what the cache actually is:

$ docker image inspect demo-bad --format '{{json .RootFS.Layers}}'

You'll likely see more hashes than you have Dockerfile instructions, and that's the first thing worth understanding. RootFS.Layers shows every layer in the final image, including all the layers your base image brought with it. node:22-alpine isn't a blank slate; it's itself a multi-layer image. Those layers appear first in the list.

To map hashes back to actual instructions, use docker history:

docker history demo-bad

Three things stand out. First, CMD and EXPOSE show in the history but have zero size and don't appear in RootFS.Layers at all. They only modify image metadata, not the filesystem, so they produce no layer. Second, WORKDIR creates a directory entry but contributes near zero bytes. Third, the base image's layers are already there before your Dockerfile runs a single line. Every FROM instruction inherits a full layer stack. Now make a code change, rebuild, and run docker image inspect again. The base image hashes at the top of the list will be identical. The hashes from your COPY and RUN instructions will be new. Same cache misses that showed up in the build log, visible here as changed fingerprints in the manifest. Two different ways of seeing the exact same thing.

And to see OverlayFS stacking at the kernel level, this works on both backends while a container is running:

docker run -d --name probe ubuntu sleep infinity

# In another terminal:
cat /proc/mounts | grep overlay

Clean:

docker stop probe && docker rm probe

That lowerdir list is the read-only layer stack. upperdir is the container's writable layer. OverlayFS presenting multiple directories as one coherent filesystem, the kernel mechanism is identical whether you're on overlay2 or the containerd snapshotter. The storage organisation changed between versions. The kernel primitive underneath didn't.

The cache isn't magic. It's Docker asking: "does a cached result already exist for this exact input hash?" Present? skip. Absent? build it, hash it, store it. The storage path changed with newer Docker versions. The question didn't.

🟢 Try it yourself

First, check which backend you're on:

docker info | grep -A2 "Storage Driver"

Then run docker image inspect demo-bad --format '{{json .RootFS.Layers}}' and save the output. Change a source file, rebuild, run it again. Compare the two outputs, you'll see exactly which hashes changed and which stayed the same. That's the cascade, visible as fingerprints.

If you're on the containerd backend (overlayfs + io.containerd.snapshotter.v1):

docker buildx du --verbose
# look for your layer hashes from the image inspect output

If you're on the classic overlay2 backend:

sudo ls /var/lib/docker/overlay2/
# match one hash from image inspect to a directory here
sudo ls /var/lib/docker/overlay2/YOUR_HASH_HERE/diff/
# these are the only files that layer added or changed

On either backend, run a container and check /proc/mounts | grep overlay. The lowerdir chain is your layer stack, visible at the kernel level.

Part 5: The Fix, and Why It Works

The fix is not a Dockerfile trick. It's a consequence of everything we just looked at.

If cache invalidation cascades downward from wherever the hash first changes, the goal is to push that change point as late in the Dockerfile as possible. The thing that changes most often, your source code, should create a layer at the bottom of the stack. The thing that changes least often, your base image, is already at the top. The question is where your dependency install lives.

Your package manifest (package.json, requirements.txt, whatever it is) changes occasionally when you add or remove a dependency. Your source code changes constantly. So copy the manifest first, install against it, then copy the source. The install layer's hash only changes when the manifest changes, which is a small fraction of your total commits.

# Dockerfile.good - correct layer order
FROM node:22-alpine

WORKDIR /app

# only the manifest
COPY package.json package-lock.json ./  
# install - cached until manifest changes 
RUN npm ci                               

# source code last
COPY . .                                 

EXPOSE 3000
CMD ["node", "index.js"]

One thing worth noting: I switched from npm install to npm ci. The difference matters in a build context. npm install resolves version ranges and can produce different results on different runs. npm ci installs exactly what's in your lockfile, deterministic, faster, no surprises. When you have a committed package-lock.json, npm ci is the right tool.

Now rebuild after the same code change:

docker build --progress=plain -t demo-good -f Dockerfile.good .

echo "// updated again" >> index.js

# Rebuild it
docker build --progress=plain -t demo-good -f Dockerfile.good .

#3 [1/5] FROM node:22-alpine                    CACHED
#4 [2/5] WORKDIR /app                           CACHED
#5 [3/5] COPY package.json package-lock.json    CACHED
#6 [4/5] RUN npm ci                             CACHED
#7 [5/5] COPY . .                               ← only this misses

0.1 seconds. Was 38. Now 0.1. Same code change. 380x difference. Because the install layer's hash didn't change, package.json didn't change, so Docker found a matching cached result for that layer, skipped it entirely, and only rebuilt the source copy layer.

This is what layer ordering actually does. Not a code style preference. A 380x speed difference because you understood where the hash chain breaks.

The Dockerfile fix isn't about instruction order as an aesthetic choice. It's about placing the cache break as high as possible in the layer stack so the cascade invalidates the fewest layers below it.

🟢 Try it yourself

Rewrite your Dockerfile with the corrected order. Cold first build, then a source file change and rebuild. Measure both.

Then test in the other direction, change package.json and rebuild. The install step should miss cache now, which is correct. Dependencies changed; rebuilding them is the right call.

# Simulate adding a real dependency
npm install express --package-lock-only
docker build --progress=plain -t demo-good -f Dockerfile.good .
# npm ci re-runs — expected and correct

Both directions should make sense now: the cache is working as designed. The previous Dockerfile was feeding it wrong inputs.

Part 6: When Layer Ordering Isn't Enough

There's a gap in what we've fixed, and it's worth being honest about it.

Layer ordering protects you when your dependencies don't change. But dependencies do change. Add a package, remove one, upgrade one, and now package.json has a new hash, which means the COPY package.json layer has a new hash, which means RUN npm ci is invalidated, which means all 215 packages reinstall from the network. You've moved the pain point; you haven't eliminated it.

And in CI, there's a harder version of this problem: the build cache is cold on every single run. Fresh runner, no layer cache at all. Every build starts from zero, regardless of how carefully your Dockerfile is ordered.

This is what BuildKit's cache mounts solve. And it's a genuinely different mechanism from layer caching, worth understanding on its own terms.

A cache mount is a directory that persists on the build host across builds, completely separate from the layer system. You tell BuildKit: "mount a persistent directory here during this step." npm downloads packages to that directory. When the build finishes, BuildKit saves the directory in its own cache storage, not in the image, not in a layer. Just on the host.

Next build, even if package.json changed and the layer cache misses entirely, npm looks in that directory first. Most packages are already there. It downloads only what's new. The layer rebuilds, but the network round trips don't.

Here's what it looks like in a Dockerfile:

# syntax=docker/dockerfile:1
# ↑ first line required pragma for BuildKit frontend features

FROM node:22-alpine

WORKDIR /app

COPY package.json package-lock.json ./

RUN --mount=type=cache,target=/root/.npm \
    npm ci --cache /root/.npm
    # /root/.npm is npm's cache directory
    # BuildKit mounts a persistent dir there during this step
    # npm writes to it, reads from it, but it never enters the image

COPY . .

CMD ["node", "index.js"]

First build, fills the cache:

$ docker build --progress=plain -t demo-bk -f Dockerfile.buildkit .

#6 [4/5] RUN --mount=type=cache... npm ci
added 215 packages in 21s
=> writing cache    ← BuildKit persisting the npm cache directory

Now add a real new dependency, something that genuinely changes package.json, and rebuild:

# Add some packeges (I have added so many)
npm install express --package-lock-only

$ docker build --progress=plain -t demo-bk -f Dockerfile.buildkit .

11.2 seconds. Layer cache missed, package.json changed, as expected. But npm found 246 packages sitting in the cache mount from the last build, downloaded only the one new package, and finished in 11 seconds instead of 38.

One thing people always ask: Does the cache mount end up in my image? No. The mount exists only during the build step. Nothing from /root/.npm enters any layer. Your image size is identical to what it would be without the mount. The cache lives on the host, separately, in BuildKit's own storage, under /var/lib/docker/buildkit/containerd-overlayfs/cachemounts/ on modern Docker installs.

Layer cache is all-or-nothing, the whole layer rebuilds or it doesn't. Cache mounts are fine-grained: even when the layer misses, your package manager's download cache survives and absorbs most of the cost.

🟢 Try it yourself

Add # syntax=docker/dockerfile:1 as the first line of your Dockerfile (no blank line before it, BuildKit reads it as a parser directive, and blank lines break that).

Update your install step to use a cache mount:

npm:

RUN --mount=type=cache,target=/root/.npm \
    npm ci --cache /root/.npm

pip:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

apt:

RUN --mount=type=cache,target=/var/cache/apt \
    apt-get update && apt-get install -y curl

Warm the cache with a first build. Then add a single new dependency and rebuild. Count how many packages actually get downloaded. The rest came from the cache mount.

To see where BuildKit stored the cache mount on disk:

sudo ls /var/lib/docker/buildkit/containerd-overlayfs/cachemounts/

To clear the cache mount and verify:

docker builder prune --filter type=exec.cachemount
# rebuild — everything downloads again, mount is gone

The Mental Model, Complete

Here's the whole picture assembled.

A Docker image is a stack of filesystem diffs. Each diff is identified by its SHA256 hash, stored as directories in /var/lib/docker/overlay2/ on classic Docker setups, or tracked via SQLite in BuildKit's containerd snapshotter on modern installs. When Docker builds, it checks before each step: does a cached result for this hash already exist? If yes, skip. If no, build it, hash it, store it.

Cache invalidation cascades because the hash of each layer includes a reference to its parent's hash. Change a parent and every descendant's hash changes automatically; the math makes it unavoidable. This is why instruction order matters: it controls where in the chain the hash first changes, and the cascade erases everything below that point.

BuildKit adds a second level underneath this. Cache mounts are persistent directories on the host that build steps can read and write, completely outside the layer system. They give your package manager a warm download cache even when the layer cache misses. The two systems work at different granularities and together they cover each other's weaknesses.

Layer cache: coarse, all-or-nothing, gone in CI. Cache mounts: fine-grained, partial reuse, survives across builds on the same host.

The reason your build was slow was never npm's fault. It was never Docker's fault. It was a misunderstanding of what the cache actually is, and now you know.

Quick Reference

Next: two containers on the same machine. You run docker exec into one and ping the other. The packet leaves through eth0, crosses a virtual cable, hits a kernel-level switch, crosses another virtual cable, and arrives at the destination. Four hops. No physical hardware involved. Next post: veth pairs, the docker0 bridge, and a tcpdump capture that makes the full packet path visible, because the network is not magic, it's Linux networking from 2007 that Docker configured for you without asking.

root/cause · Not tutorials. Just the real picture.