Flaky CI failure: classify, then one experiment
Turns an intermittent CI failure into a diagnosis: classification with log evidence, ONE confirming experiment, and only then a fix candidate.
by @kourosh · Aug 4, 2026
Fill the placeholders
A CI job fails intermittently. Diagnose it with discipline: no fix suggestions until a classification is confirmed by an experiment.
Step 1 — CLASSIFY. Based on the log and test source below, classify the failure as exactly one of:
- flaky-timing (races, sleeps, timeouts, ordering assumptions)
- shared-state pollution (test order dependence, leaked globals/fixtures/db rows)
- external dependency (network, third-party service, rate limits, clock)
- genuine regression (the code is actually broken)
Cite the specific log lines that support the classification. If evidence is insufficient, say what is missing instead of guessing.
Step 2 — ONE EXPERIMENT. Propose the single cheapest experiment that would confirm or refute the classification (e.g. "run this one test 50x in isolation", "reorder these two tests", "pin the container clock"). State the expected outcome under your hypothesis AND under the strongest rival hypothesis.
Step 3 — FIX CANDIDATE (conditional). Only after the experiment is defined, give one concrete fix candidate for the confirmed class — smallest change first.
CI log:
{{ci_log}}
Test source:
{{test_source}}