CI can stay green after Upstream moves the pinned tag, because the clone step is cached #25

New issue

Closed

opened 2026-09-30 15:15:08 +00:00 by piscis · 1 comment

piscis commented

2026-09-30 15:15:08 +00:00

Owner

Problem

ci builds on the runner's shared dind daemon, and BuildKit keeps its cache across runs. The Dockerfile's clone-and-verify step (git clone --branch "$UPSTREAM_VERSION", then compare HEAD with UPSTREAM_COMMIT) is cached by its command and build args. So when neither the Pin nor the Dockerfile changes, CI reuses the old layer and never clones again.

If Upstream moved the tag of the pinned Upstream Version, CI would stay green, and the next uncached build (a new runner, a pruned cache, the publish workflow) would be the first to fail.

A PR that changes the Pin isn't affected: new build args mean a cache miss, and a mismatch turns the run red (run 6 in #6).

Found by /code-review on #20 / #23.

Options

Rebuild the build stage in CI without the cache (docker buildx build --no-cache-filter build). The Go module and build cache mounts keep it fairly quick, but every run then needs Upstream and the Alpine mirror.
Check the tag outside the build, e.g. git ls-remote "$UPSTREAM_REPO" "refs/tags/$UPSTREAM_VERSION^{}" compared with UPSTREAM_COMMIT, as a cheap CI step or make target.
Accept it, if the publish workflow (#7) always builds without cache.

Acceptance criteria

Moving the Upstream tag without changing the Pin turns ci red on the next run, even with a warm cache

## Problem `ci` builds on the runner's shared dind daemon, and BuildKit keeps its cache across runs. The Dockerfile's clone-and-verify step (`git clone --branch "$UPSTREAM_VERSION"`, then compare `HEAD` with `UPSTREAM_COMMIT`) is cached by its command and build args. So when neither the Pin nor the Dockerfile changes, CI reuses the old layer and never clones again. If Upstream moved the tag of the pinned Upstream Version, CI would stay green, and the next uncached build (a new runner, a pruned cache, the publish workflow) would be the first to fail. A PR that changes the Pin isn't affected: new build args mean a cache miss, and a mismatch turns the run red (run 6 in #6). Found by `/code-review` on #20 / #23. ## Options - Rebuild the `build` stage in CI without the cache (`docker buildx build --no-cache-filter build`). The Go module and build cache mounts keep it fairly quick, but every run then needs Upstream and the Alpine mirror. - Check the tag outside the build, e.g. `git ls-remote "$UPSTREAM_REPO" "refs/tags/$UPSTREAM_VERSION^{}"` compared with `UPSTREAM_COMMIT`, as a cheap CI step or `make` target. - Accept it, if the publish workflow (#7) always builds without cache. ## Acceptance criteria - [ ] Moving the Upstream tag without changing the Pin turns `ci` red on the next run, even with a warm cache

piscis referenced this issue

2026-09-30 15:15:18 +00:00

CI builds and smoke-tests the Image on every PR #6

piscis added the

needs-triage

bug

labels

2026-09-30 15:15:19 +00:00

piscis commented

2026-09-30 15:20:30 +00:00

Author

Owner

Triage

Decision: option 2. Check the tag outside the build with git ls-remote.

Option 1 (--no-cache-filter build) makes every run depend on network access from inside BuildKit, which is what broke in #22. It is also slow.
Option 3 (accept it) doesn't meet the acceptance criterion.

Implementation notes

make verify-pin: resolve $UPSTREAM_VERSION on Upstream with git ls-remote and compare it with UPSTREAM_COMMIT. Use the peeled ref refs/tags/$UPSTREAM_VERSION^{} and fall back to refs/tags/$UPSTREAM_VERSION for lightweight tags. v3.2.0 is annotated: the plain ref is the tag object 931a525… and ^{} is e30bb7e…, which matches the Pin. On a mismatch, fail with the same message as the Dockerfile check. Also fail if the tag doesn't exist.
ci.yml: add a step that runs make verify-pin before Build the Image. It runs in the job container, where DNS works.
One Upstream URL: it's currently hard-coded as ARG UPSTREAM_REPO in the Dockerfile. Define it once (pin.env or the Makefile) and pass it to the build as a build arg, so the two checks can't drift apart.
Keep the Dockerfile check. It still protects uncached and local builds.
#7: the publish workflow should run make verify-pin too.

Verifying

With a warm cache, push a throwaway commit that sets UPSTREAM_COMMIT to a wrong commit, and confirm ci goes red at the verify step before the build runs. Then revert it, as with 68e52e5/c76070e.

## Triage **Decision: option 2.** Check the tag outside the build with `git ls-remote`. - Option 1 (`--no-cache-filter build`) makes every run depend on network access from inside BuildKit, which is what broke in #22. It is also slow. - Option 3 (accept it) doesn't meet the acceptance criterion. ### Implementation notes - **`make verify-pin`**: resolve `$UPSTREAM_VERSION` on Upstream with `git ls-remote` and compare it with `UPSTREAM_COMMIT`. Use the peeled ref `refs/tags/$UPSTREAM_VERSION^{}` and fall back to `refs/tags/$UPSTREAM_VERSION` for lightweight tags. v3.2.0 is annotated: the plain ref is the tag object `931a525…` and `^{}` is `e30bb7e…`, which matches the Pin. On a mismatch, fail with the same message as the Dockerfile check. Also fail if the tag doesn't exist. - **`ci.yml`**: add a step that runs `make verify-pin` before `Build the Image`. It runs in the job container, where DNS works. - **One Upstream URL**: it's currently hard-coded as `ARG UPSTREAM_REPO` in the Dockerfile. Define it once (`pin.env` or the Makefile) and pass it to the build as a build arg, so the two checks can't drift apart. - **Keep the Dockerfile check.** It still protects uncached and local builds. - **#7**: the publish workflow should run `make verify-pin` too. ### Verifying With a warm cache, push a throwaway commit that sets `UPSTREAM_COMMIT` to a wrong commit, and confirm `ci` goes red at the verify step before the build runs. Then revert it, as with `68e52e5`/`c76070e`.

piscis added

ready-for-agent

and removed

needs-triage

labels

2026-09-30 15:20:30 +00:00

piscis referenced this issue from a commit

2026-09-30 15:23:18 +00:00

Check the Pin against Upstream with make verify-pin before the cached build (#25)

piscis referenced this issue from a pull request that will close it,

2026-09-30 15:23:33 +00:00

ci: check the Pin against Upstream with make verify-pin before the cached build #26