1. Collect the evidence
Gilfoyle reads the CI history and logs for the test, comparing failed and passing runs of the same commit. Timing, test order, shared state and external calls are the usual suspects; the evidence decides which one applies.
Fix flaky tests
Gilfoyle finds why a test passes and fails on the same code, fixes the cause and shows repeated runs, instead of adding another retry.
flaky-tests
Example brief · Hypothetical workflow
Your brief
checkout.spec fails now and then and we just rerun it. Find out why and fix it properly, without adding retries.
GilfoyleAPP
I’ll compare the failed and passing runs, reproduce the intermittent failure and fix what causes it.
See example output ↓How Gilfoyle handles it.
Starts: When a check fails and then passes on rerun, or you name tests your team keeps retrying.
2. Reproduce and fix the cause
With connected Claude Code or Codex and GitHub, Gilfoyle repeats the test under the conditions that trigger it, such as parallel runs or a fixed random seed, and fixes the race, leaked state or unmocked dependency it finds.
3. Show that it holds
The draft PR includes the cause, the change and repeated runs before and after. If the flake cannot be reproduced, you get the evidence gathered and the next diagnostic step, not a quietly added retry. Your reviewers decide what merges.
Stable for a reason you can read.
Use this for intermittent failures. A check that fails every time belongs in the failing-CI workflow.
Hypothetical finished result
Example output
Checkout flake diagnosis
Hypothetical: a browser test fails intermittently when the suite runs in parallel.
- Cause
- Every failed run shares one pattern: another test resets the shared cart fixture while checkout.spec is mid-flow. Alone, the test passes; beside cart.spec, it fails.
- Fix
- Each test now creates its own cart through a per-test fixture, and the shared reset helper is removed from the checkout suite. No retries or longer timeouts were added.
- Evidence
- Before the change, 50 parallel repetitions produced 6 failures; after it, all 50 passed. The draft PR lists both commands and notes that repetition lowers, but does not rule out, the chance of an unseen flake.
The apps this job needs.
Connect the accounts and actions for your workflow. Start in Artie or add your Artisan to Slack; access stays within your account permissions.
Explore apps- GitHub
- Connect the repository with access to Actions run history and logs, and permission to open draft PRs.
- Claude Code or Codex
- Connect the coding tool and provide an environment that can run the test repeatedly, including any browsers or services it uses.
Questions about this workflow
Why not just add a retry?
A retry hides the failure and keeps the cause. Gilfoyle fixes the cause where the evidence supports one; adding a retry stays an explicit decision for your reviewer.
What if it only fails in CI?
Gilfoyle compares CI with local runs, including parallelism, resources and ordering, and reproduces those conditions where the environment allows.
How do we know it’s fixed?
The draft PR shows repeated runs before and after. Repetition is strong evidence rather than proof, so the note says what was and was not exercised.