A build that rebuilds from scratch on CI while being fully cached locally is usually not a Docker problem — it is that the checkout produced different file metadata.
# the cache key for COPY includes mode and path, not just content
$ git config core.fileMode false # stops chmod noise entering the tree
# and on CI, a shallow clone changes nothing about modes but
# a fresh workspace has no previous layers to match against:
$ docker pull registry.internal/app:latest
$ docker build --cache-from registry.internal/app:latest -t app:new .
--cache-from is the mechanism that makes a stateless CI runner able to reuse layers at all, and it requires pulling the previous image first — which is a cost worth measuring, since a large image can take longer to pull than the build saves. The mode sensitivity catches people on Windows and WSL most often, where a checked-out file arrives as 0755 on one machine and 0644 on another and the layer hash differs for reasons nobody can see.