How every case scored, and why.
- Case 01Automate now
- Case 02Needs one thing first
- Case 03Needs several things first
- Case 04Blocked
A QA practice that combines a custom AI skill, GitHub's Spec Kit and Playwright. It reads a feature's design document and the application's real code, then writes and runs end-to-end tests in the team's own conventions. Automation no longer waits on a few Playwright specialists.
Started as a proof of concept · now QA Engineering's default
Case Study v2.0, QA Engineering × Test Automation Practice. Onboarding and review-time figures as reported by the team.
AI-native development ships features faster, and QA has to keep up. Test automation is where the pressure lands, and it needs real fluency in frameworks like Playwright or Cypress.
Good tests when the author is skilled, but the skill doesn't transfer. A new engineer has to learn Playwright, the app and the team's conventions all at once. Coverage grows only as fast as the people who can write it.
Faster, but the result depends on how the feature was described and what code was pasted in. Two engineers prompting about the same feature get two differently built tests, and no shared record of what either checked.
BothThe process lives in one person's head and one model session, not in a standard the team runs. Neither is grounded in the application's real code.
The instinct is to point a generator at a design document and take whatever tests come back. That only moves the ungoverned part from the prompt to the tool.
Writes tests for behaviour the design document describes but the application never shipped.
Turns every scenario green, even where a real defect should keep a test red.
“Fixes” a broken shared helper on its own, quietly changing what every other test depends on.
Removes the moments that need a person: scoping the run, settling an unclear requirement, deciding what a stuck case needs.
The fixNot removing the person. One governed process, grounded in the real application, bounded in what it does on its own, and clear about the three points where it needs a person.
One custom skill replaces both hand-written and ad-hoc automation. It asks for four things, and every run follows the same rules, in the same order, whoever starts it.
Where the tests will live, with the suite's own conventions.
Already checked out. It never creates, switches or updates one; that stays the engineer's call.
The feature's intended behaviour, read for what it says. Not assumed to follow a company-wide format.
A knowledge graph built from the application's actual code, so the agent never reasons blind about what shipped.
Design documents describe intent, not always current reality. So the practice checks reality twice before it writes a test.
The feature map. An agent reasoning from the design document alone will assume behaviour that was never built.
Observe once. Before writing anything, it drives the product by hand once and records how the screens actually behave.
WhyMost stuck cases are stuck on a fact about the product, not a hard technical problem. One look at the screen answers what several more blind attempts cannot. This rule came from watching early runs go badly: the rule changed, not the tooling.
One run moves through the same six stages every time. It stops for a person three times: to scope the run, to answer the chain’s own clarifying questions, and at a checkpoint every ten cases.
Every case in scope is scored against the feature map, along with the helpers the repo already has for it.
Orange marks the three points where a person decides
Every case in scope is scored against the feature map before anything is built, along with the helpers the repo already has for it. Then a person scopes the run in one round of questions. After that, it runs.
The four triage outcomes
A person picks
Only after triage, scoping and one look at the product does it run GitHub’s Spec Kit chain in full, against rules the team wrote down and versioned, not a prompt improvised for each session.
The nine steps, in order
Orange marks where a person answers
All cases in scope run, and failures are grouped by cause. Each case gets up to four fix attempts. A case that is still stuck is parked with one specific question, instead of quietly using up the rest of the session.
Nothing is committedIt prepares the work and shows the diff. Committing stays with the person running it.
Run · every case in scope
Every run hands back the same six things, readable by QA and product, not just engineers.
How every case scored, and why.
What each case proves, reviewable by QA and product.
Written in the suite's own conventions.
Passed first time, fixed in this run, still failing or skipped.
So the next run doesn't rediscover what this one learned.
One page: what got built, what it cost, what still needs a person.
How every case scored, and why.
What each case proves.
In the suite's own conventions.
Not a wall of green.
What this run learned.
Next run starts here
One page for the owner.
Onboarding time for a new automation contributor. A person defines what to automate, not how to write it in Playwright. A new QA hire can scope and review a run on day one, instead of spending weeks building Playwright fluency.
Less code review time on automation pull requests. Common pitfalls and review feedback are written into the constitution step and enforced before a test reaches review. Reviewers no longer need to be Playwright specialists.
One QA functionNo split between “automation QA” and “manual QA”. Backlogged automation runs in parallel across several cases, and engineers spend their time on exploratory testing, triage and judgement calls, not on writing selectors.
SourceCase Study v2.0, QA Engineering × Test Automation Practice. Onboarding and review-time figures as reported by the team.
It started as a proof of concept: four runs against one production automation suite. Before writing a single new test, it found broken shared helpers, including an assertion that checked nothing, which every existing test had been silently relying on.
It has since been adopted across several new projects.
Where the time, the attention and the setup go.
Test execution dominates the budget, not the reasoning. Six cases on a slower suite run in hours, not minutes.
About three points of attention per run: scoping at the start, clarify questions mid-run, a checkpoint at the end.
Nothing beyond Python. The workflow installs its own tooling and removes it afterwards.
Worth saying plainly. It sets the right expectations and explains the guardrails.
What it costs
Where it fits
For a team choosing between hand-written tests that don’t scale past the people who write them, and ad-hoc prompting that changes with who is asking.
For a team that wants a hands-off generator.
Start with one real design document →
and the application surface it describes, from your own area. The same starting point this practice was proven on.