Test automation
Technology OpsSpec-Driven QATest automation case study
Technology Ops
Spec-Driven QA
TEST AUTOMATION

From design document to running tests, the same way every time.

A QA practice that combines a custom AI skill, GitHub's Spec Kit and Playwright. It reads a feature's design document and the application's real code, then writes and runs end-to-end tests in the team's own conventions. Automation no longer waits on a few Playwright specialists.

Started as a proof of concept · now QA Engineering's default

Day one
a new QA hire can scope and review a run, instead of spending weeks learning Playwright
Over50%
less reviewer time on automation pull requests
9steps
GitHub's Spec Kit chain, the same on every run, whoever starts it
Sources

Case Study v2.0, QA Engineering × Test Automation Practice. Onboarding and review-time figures as reported by the team.

4 inputs · 6 stages · 9 Spec Kit stepsA power tool with a person on it
THE PRESSURE

Development got faster. Automation still waits on specialists.

AI-native development ships features faster, and QA has to keep up. Test automation is where the pressure lands, and it needs real fluency in frameworks like Playwright or Cypress.

  • Experienced Playwright engineers are hard to find.
  • There is no standard way to automate, so every author does it differently.
  • Reviewing tests to catch flaky ones takes slow and growing effort.
  • Vibe-coded tests are cheap to produce but brittle, and they don't fit existing frameworks.
Features arrive faster than tests Grab and throw a tile · hover the gate
HOW TEAMS AUTOMATE TODAY

Two ways to automate today. Neither scales.

Hand-written

Good tests when the author is skilled, but the skill doesn't transfer. A new engineer has to learn Playwright, the app and the team's conventions all at once. Coverage grows only as fast as the people who can write it.

Ad-hoc prompting

Faster, but the result depends on how the feature was described and what code was pasted in. Two engineers prompting about the same feature get two differently built tests, and no shared record of what either checked.

BothThe process lives in one person's head and one model session, not in a standard the team runs. Neither is grounded in the application's real code.

Feature TestIllustrative
WHY A GENERATOR DOESN'T FIX IT

Removing the person doesn't fix it. It moves the problem.

The instinct is to point a generator at a design document and take whatever tests come back. That only moves the ungoverned part from the prompt to the tool.

Not grounded

Writes tests for behaviour the design document describes but the application never shipped.

Built to look good

Turns every scenario green, even where a real defect should keep a test red.

No guardrail

“Fixes” a broken shared helper on its own, quietly changing what every other test depends on.

Unattended

Removes the moments that need a person: scoping the run, settling an unclear requirement, deciding what a stuck case needs.

The fixNot removing the person. One governed process, grounded in the real application, bounded in what it does on its own, and clear about the three points where it needs a person.

Illustrative · hover a row or the stage
THE APPROACH

Four inputs. Nothing else up front.

One custom skill replaces both hand-written and ad-hoc automation. It asks for four things, and every run follows the same rules, in the same order, whoever starts it.

Automation repo

Where the tests will live, with the suite's own conventions.

Branch

Already checked out. It never creates, switches or updates one; that stays the engineer's call.

Design or spec document

The feature's intended behaviour, read for what it says. Not assumed to follow a company-wide format.

Feature map

A knowledge graph built from the application's actual code, so the agent never reasons blind about what shipped.

Same rules · same order · whoever starts it
GROUNDED IN WHAT SHIPPED

Look before you guess.

Design documents describe intent, not always current reality. So the practice checks reality twice before it writes a test.

On the code side

The feature map. An agent reasoning from the design document alone will assume behaviour that was never built.

On the product side

Observe once. Before writing anything, it drives the product by hand once and records how the screens actually behave.

WhyMost stuck cases are stuck on a fact about the product, not a hard technical problem. One look at the screen answers what several more blind attempts cannot. This rule came from watching early runs go badly: the rule changed, not the tooling.

Design document · example claims
  1. 1Sign in with a work account
  2. 2Filter the list by status
  3. 3Edit a record in place
  4. 4Export the list as a file
  5. 5Archive many records at once
  6. 6Keep a history of every change
Feature map · from the real code
Drag a node
THE WORKFLOW

Six stages. A person at three points.

One run moves through the same six stages every time. It stops for a person three times: to scope the run, to answer the chain’s own clarifying questions, and at a checkpoint every ten cases.

01
Stage Triage

Every case in scope is scored against the feature map, along with the helpers the repo already has for it.

Orange marks the three points where a person decides

One run, every time
STAGES 1 AND 2

Score every case. Then a person chooses.

Every case in scope is scored against the feature map before anything is built, along with the helpers the repo already has for it. Then a person scopes the run in one round of questions. After that, it runs.

The four triage outcomes

01Automate now
02Needs one thing first
03Needs several things first
04Blocked

A person picks

  • Which cases this run takes
  • Who the test user is
  • How each dependency gets satisfied
Triage · score every case
Grab a case · in Choose, select cases for this run
STAGE 4 · GITHUB SPEC KIT

The same nine steps a person would take. Every run.

Only after triage, scoping and one look at the product does it run GitHub’s Spec Kit chain in full, against rules the team wrote down and versioned, not a prompt improvised for each session.

The nine steps, in order

Orange marks where a person answers

GitHub Spec Kit · the full chain
01 · ConstitutionThe team’s rules, including common pitfalls and past review feedback, written once and enforced before a test reaches review.
03 · ClarifyThe chain’s own questions. A person answers them.
STAGE 5

Fix within a budget. Then ask one clear question.

All cases in scope run, and failures are grouped by cause. Each case gets up to four fix attempts. A case that is still stuck is parked with one specific question, instead of quietly using up the rest of the session.

4
fix attempts per case, at most
10
cases per run, at most

Nothing is committedIt prepares the work and shows the diff. Committing stays with the person running it.

One run · up to ten cases · up to four fixes each

Run · every case in scope

STAGE 6

An honest account, not a wall of green.

Every run hands back the same six things, readable by QA and product, not just engineers.

Triage report

How every case scored, and why.

Specs

What each case proves, reviewable by QA and product.

Test code

Written in the suite's own conventions.

Status table

Passed first time, fixed in this run, still failing or skipped.

Session tracker

So the next run doesn't rediscover what this one learned.

Owner dashboard

One page: what got built, what it cost, what still needs a person.

Six outputs · the same every run 04 · Status table
Triage report01

How every case scored, and why.

  • Case 01Automate now
  • Case 02Needs one thing first
  • Case 03Needs several things first
  • Case 04Blocked
Scored against the feature map
Specs02

What each case proves.

Case 01 · proves
Case 02 · proves
Case 03 · proves
Reviewable by QA and product
Test code03

In the suite's own conventions.

Playwright · suite conventions
Status table04

Not a wall of green.

Passed first time
01040910
Fixed this run
02030508
Still failing or skipped
0607
A real failure stays redParked with one question
Illustrative rows
Session tracker05

What this run learned.

Next run starts here

Nothing rediscovered
Owner dashboard06

One page for the owner.

Built
Cost
Needs a person
Figures not shown here
THREE APPROACHES

Same team, three ways to automate. One standard.

Five dimensions Hand-written Playwright Ad-hoc AI prompting Custom skill + Spec Kit
Full Playwright fluency plus app knowledge, per engineer
Prompting skill plus app knowledge, per engineer, per session
Same four inputs, same rules, for every engineer
No, conventions vary by author
No, output depends on the prompt and what was pasted in
Same Spec Kit chain and repo conventions, every run
Only as well as the author remembers the codebase
Only as well as what got pasted into that session
The feature map keeps the agent from reasoning blind about what shipped
Depends on the reviewer catching drift
No guardrail: a session can change a shared helper unnoticed
Never touched automatically; a broken one stops the run and asks
Depends on the author’s diligence
Depends on the session; nothing stops a claimed pass that isn’t one
Bounded fix attempts; a real failure stays red and comes back as a question
THE RETURN

The pay-off is in people and cycle time, not pass rates.

Near zero

Onboarding time for a new automation contributor. A person defines what to automate, not how to write it in Playwright. A new QA hire can scope and review a run on day one, instead of spending weeks building Playwright fluency.

Over50%

Less code review time on automation pull requests. Common pitfalls and review feedback are written into the constitution step and enforced before a test reaches review. Reviewers no longer need to be Playwright specialists.

One QA functionNo split between “automation QA” and “manual QA”. Backlogged automation runs in parallel across several cases, and engineers spend their time on exploratory testing, triage and judgement calls, not on writing selectors.

SourceCase Study v2.0, QA Engineering × Test Automation Practice. Onboarding and review-time figures as reported by the team.

Illustrative · same arrivals in both views
Coverage gated on what a run can carry, not on a few specialists Grab and throw a tile · hover the run
FROM PROOF OF CONCEPT TO DEFAULT

Before it wrote a single test, it found a bug in the suite.

It started as a proof of concept: four runs against one production automation suite. Before writing a single new test, it found broken shared helpers, including an assertion that checked nothing, which every existing test had been silently relying on.

4runs
the proof of concept, on one production suite
1no-op
assertion under every existing test, checking nothing
Default
QA Engineering's approach to test automation today

It has since been adopted across several new projects.

Scanning shared helpers before writing anything new
4 runs · 1 production suite · illustrativeFound before a single new test
THE RESULTS · WHAT IT COSTS

Lightweight to run. Three check‑ins, Python only, no extra tooling.

Where the time, the attention and the setup go.

Hover or select a card to replay it
01Hours, not minutes

Time

Test execution dominates the budget, not the reasoning. Six cases on a slower suite run in hours, not minutes.

02Three check‑ins

People

About three points of attention per run: scoping at the start, clarify questions mid-run, a checkpoint at the end.

03Python only

Setup

Nothing beyond Python. The workflow installs its own tooling and removes it afterwards.

WHAT IT WILL NOT DO

What it will not do, on purpose.

Worth saying plainly. It sets the right expectations and explains the guardrails.

Hover or select a tile for the reason
WHAT IT COSTS · WHERE IT FITS

A power tool with a person on it. Not a replacement for one.

What it costs

  • Running the tests dominates the budget, not the reasoning. A batch of six cases on a slower suite runs in hours, not minutes.
  • A person’s attention at roughly three points per run.
  • Nothing beyond Python to install. It installs its own tooling and removes it afterwards.

Where it fits

Worth it

For a team choosing between hand-written tests that don’t scale past the people who write them, and ad-hoc prompting that changes with who is asking.

Not yet

For a team that wants a hands-off generator.

One run · where the budget goes
Next step

Start with one real design document →

and the application surface it describes, from your own area. The same starting point this practice was proven on.

Move