Skip to content

ci: harden the flash-attn cache and run unit-test on retargeted PRs - #8

Closed
curnane-lab wants to merge 3 commits into
mainfrom
cursor/ci-fa-cache-hardening-6929
Closed

curnane-lab wants to merge 3 commits into
mainfrom
cursor/ci-fa-cache-hardening-6929

Conversation

@curnane-lab

@curnane-lab curnane-lab commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Motivation

Follow-up to sgl-project#1078. These are gaps in the unit-test pipeline, each reproduced or traced before the fix:

  • Stacked PRs merge without unit-test. [3/4] capture: remove legacy SGLang patch support sgl-project/SpecForge#1065 and [4/4] build: align training with the pinned SGLang revision sgl-project/SpecForge#1066 were opened against their stacked parents and retargeted to main after the parent merged. Changing the base only fires pull_request: edited, which the default trigger types leave out, so both PRs show only lint and merged with the broken pinned-SGLang toolchain that ci: unbreak unit-test with the pinned SGLang toolchain sgl-project/SpecForge#1078 then had to fix.
  • A failed flash-attn wheel stays cached for good. ensure_flash_attention.sh built the patched wheel straight into the persistent SPECFORGE_CI_CACHE/flash-attn/<build_key>/wheels/, before installing it or running the ABI check. A wheel that fails the check, or a truncated one left when bdist_wheel is killed by the 30-minute step timeout, is reused by every later run with no rebuild, until someone deletes it on the runner. Reproduced with the existing stubs: after an ABI failure the wheel was still cached and the next run downloaded nothing.
  • The sdist is not checksummed. The script downloads the sdist from the URL in PyPI's JSON and patches it without checking the digests.sha256 from the same JSON. uv checked it when it fetched the sdist itself.
  • FLASH_ATTENTION_FORCE_BUILD is no longer covered by tests. Before ci: unbreak unit-test with the pinned SGLang toolchain sgl-project/SpecForge#1078 a test asserted it on the uv command. Now it is only passed to setup.py and nothing checks it. Without it, flash-attn's setup.py downloads a prebuilt wheel built against another torch, and the tests still pass.
  • safe.directory grows every run. /github/home persists across jobs on this runner (that is what keeps SPECFORGE_CI_CACHE alive), so git config --global --add appends another identical entry on every unit-test run.

Modifications

  • test.yaml
    • Add types: [opened, synchronize, reopened, edited]. The GPU jobs run for edited only when github.event.changes.base is set.
    • Title and body edits also fire edited. Those runs get their own concurrency group (keyed by run_id), so they neither cancel a running test nor replace a queued one under the workflow's cancel-in-progress.
    • Add the safe.directory entry only when git config --fixed-value --get does not already find it (the container has git 2.43).
  • ensure_flash_attention.sh
    • Build into the temporary source directory. Publish to the cache only after the wheel installs and passes check_flash_attention, by copying to a hidden staging file in the cache directory and renaming it atomically.
    • If a cached wheel fails to install or fails the check, delete it so the next run rebuilds.
    • Verify the sdist with sha256sum -c before extracting.
  • test_ci_dependencies.py
    • The curl stub serves a byte-stable tarball plus its sha256, and its setup.py asserts FLASH_ATTENTION_FORCE_BUILD=TRUE.
    • New cases cover: no wheel cached after a failed ABI check; a cached wheel that fails is dropped and rebuilt; a checksum mismatch stops before any install; trusting the workspace twice leaves one entry; the retarget trigger and the edit guards.

Related Issues

Follow-up to sgl-project#1078. The stacked-PR gap let sgl-project#1065 and sgl-project#1066 merge without unit-test.

Accuracy Test

Not applicable. CI scripts and tests only.

Benchmark & Profiling

Not applicable. CI scripts and tests only.

Testing

  • python -m unittest tests.test_scripts.test_ci_dependencies: 14 tests pass.
  • With the script and workflow reverted to ci: unbreak unit-test with the pinned SGLang toolchain sgl-project/SpecForge#1078 and the new tests kept, the flash-attn cache, checksum and safe.directory cases fail.
  • With FLASH_ATTENTION_FORCE_BUILD=TRUE removed from the script, 6 cases fail.
  • pre-commit run on the changed files passes. actionlint reports only the existing queue: max keys, which it does not know yet.
  • Not exercised on GitHub: a real retarget, and a body edit during a queued run. On the GPU runner, a passing wheel is already cached for the current build_key, so unit-test reuses it until torch or flash-attn changes.

Checklist

Open in Web Open in Cursor 

cursoragent and others added 3 commits November 7, 2026 07:04
ensure_flash_attention.sh built the patched flash-attn wheel straight into
the persistent per-build_key cache, before installing it or running the
ABI check. A wheel that failed the check, or a truncated one left by a
timed-out or cancelled bdist_wheel, stayed cached, and every later run
skipped the build and reinstalled it.

- Build into the temporary source directory and publish to the cache by an
  atomic rename only after the wheel installs and passes the ABI check.
  Drop a cached wheel that fails to install or fails the check, so the next
  run rebuilds it.
- Verify the sdist against the sha256 in PyPI's JSON before extracting and
  patching it; uv used to do this when it fetched the sdist itself.

Tests: the curl stub now serves a byte-stable tarball with its sha256, and
its setup.py asserts FLASH_ATTENTION_FORCE_BUILD=TRUE, which no test
covered after the build moved out of uv. New cases cover the failed-check,
dropped-cache and checksum-mismatch paths.

Co-authored-by: curnane-lab <curnane-lab@users.noreply.github.com>
HOME (/github/home) persists across jobs on the self-hosted runner, which
is what keeps SPECFORGE_CI_CACHE alive, so `git config --global --add
safe.directory` appended another identical entry on every unit-test run.
Skip the add when the exact path is already trusted.

Co-authored-by: curnane-lab <curnane-lab@users.noreply.github.com>
sgl-project#1065 and sgl-project#1066 were opened against their stacked parents and retargeted
to main once the parent merged. Changing the base fires only an edited
event, which the default pull_request types omit, so unit-test never ran
on either before merge and the broken pinned-SGLang toolchain reached main.

Add edited to the trigger types and run the GPU jobs for it only when the
base changed. Title and body edits also fire edited; give those runs their
own concurrency group so they do not cancel or replace a running or queued
test, and skip their jobs.

Co-authored-by: curnane-lab <curnane-lab@users.noreply.github.com>
@cursor cursor Bot changed the title ci: cache flash-attn wheels only after the ABI check, trust the workspace once ci: harden the flash-attn cache and run unit-test on retargeted PRs Nov 7, 2026
@curnane-lab curnane-lab closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants