Introduction
Data OpsAI-Native ScrapingThe standard web scraping process
Data Ops
AI-Native Scraping
WEB SCRAPING · THE STANDARD PROCESS

One standard process for every scraping request.

How the Web Scraping team develops, deploys and supports website scrapers: one scraping ticket per URL, the same five lifecycle stages every time, AI agents for the legwork and engineers for the technical decisions.

Scraping ticket · four AI agents · PR review · UAT · production · on-call

1
scraping ticket per website URL, tracked from request to closure
4
AI agents for research, implementation, execution and quality review, with engineers responsible for technical decisions
5
lifecycle stages from ticket to production, the same for every URL
Sources

Web Scraping Team AI Native Workflow: 1 Overview and delivery approach, 1.1 Delivery model, 1.2 AI-assisted development workflow, 2 AI-assisted development lifecycle.

Every request · 5 stages · 1 way of workingFrom client request to ongoing maintenance
WHY A STANDARD PROCESS

Every website is different. The process is not.

Data can sit in static or dynamic HTML, behind APIs, in Excel files, PDFs or downloads, spread across many pages and guarded by rate limits. The team answers that variety with one end-to-end process that combines engineering practice with AI-assisted workflows.

Formats

Static and dynamic HTML, APIs, Excel files, PDFs and downloadable documents.

Mechanics

Pagination, dynamically loaded content, authentication, chained requests and download mechanisms.

Constraints

Request volumes, rate limits, access restrictions, blocking and edge cases.

The aimImprove development efficiency, maintain consistency and reduce manual effort, from the client requirement through to on-call support.

SourceWeb Scraping Team AI Native Workflow: 1, Overview and delivery approach; Step 1, Ticket assessment and requirements exploration.

Sources of every kind in, one consistent shape of data out
0sources in0consistent rows out
Grab a source · hover for its challenges
THE DELIVERY MODEL

One URL, one ticket.

The BA team talks to clients, gathers requirements and raises a scraping ticket for each website URL. Requests flow into a backlog continuously, and the development team plans the upcoming week from it.

BA team

Communicates with clients, gathers requirements and creates scraping tickets for incoming requests.

Backlog

Each website URL is tracked through its own scraping ticket, and new requests continuously enter the backlog.

Weekly planning

The development team plans the upcoming week: review the available tickets, assess complexity, select the work, assign each ticket to a developer.

Build skill

Once assigned, each ticket follows the development lifecycle, started with /build-scrape-skill and using AI agents where applicable.

SourceWeb Scraping Team AI Native Workflow: 1.1 Delivery model; 2.1 AI impact by development ticket complexity; 5 Roles and responsibilities.

An illustrative week · ticket names are examples
THE AGENT TEAM

Four agents do the legwork. Engineers make the calls.

The /build-scrape-skill starts the AI-assisted process, with four AI agents working on each ticket where applicable. They may iterate between implementation, execution and analysis until the outcome is achieved or a person needs to step in.

Research Specialist

Investigates the target website, identifies data sources and extraction approaches, and prepares the implementation plan.

Implementation Specialist

Generates the scraper code and its extraction, transformation and validation logic.

Execution Specialist

Runs implementations and tests, identifies failures and iterates on corrections.

Quality Reviewer

Reviews implementation quality, extracted data, test results and production readiness.

People stay accountableThe development team retains responsibility for technical decisions, quality, production delivery and closure.

SourceWeb Scraping Team AI Native Workflow: 1.2 AI-assisted development workflow; 5 Roles and responsibilities.

One ticket’s path through the agents · illustrative
Research Specialist prepares the planCurrent step
STEP 1 · ASSESS AND EXPLORE

Look before you build.

The goal is a feasible, reliable extraction strategy. The developer reviews the scraping ticket, the expected output, the acceptance criteria and the business context, and the Research Specialist investigates the website.

Requirements

Find missing or ambiguous requirements, dependencies and risks; duplicate requests, existing scrapers or reusable components.

The website

Technologies, structure and data sources: static or dynamic HTML, APIs, Excel files, PDFs or downloads.

Mechanics and limits

Pagination, dynamic content, authentication, chained requests; request volumes, rate limits, access restrictions and blocking.

Output

A structured execution plan for the Implementation and Execution Specialists.

Where people decideThe developer assesses technical feasibility and flags where business clarification is needed; the BA team takes it to the client. Significant technical decisions stay with engineers.

SourceWeb Scraping Team AI Native Workflow: Step 1, Ticket assessment and requirements exploration.

The Research Specialist’s scan, layer by layer

Every websiteThe Research Specialist works through every layer and hands a structured execution plan to the Implementation and Execution Specialists.

STEP 2 · DEVELOP AND EXECUTE

Build it, run it, fix it, repeat.

The Implementation Specialist turns the Research Specialist’s plan into code that follows project conventions and the internal libraries, with logging, error handling and request management built in. The Execution Specialist runs it, tests extraction across the relevant formats and feeds failures back for correction.

  • Static and dynamic HTML parsing with CSS selectors, XPath and HTML tables; API integrations, authentication and chained requests.
  • Basic, deep and dynamic pagination; Excel across sheets and complex layouts; PDF tables, with OCR where feasible.
  • Request management, rate limiting, retries, exception handling and edge cases.
  • Transformation, cleaning, normalisation, validation and quality checks.

Where people step inComplex business logic, extensive R&D, architectural decisions, or issues the agents cannot resolve reliably.

SourcesWeb Scraping Team AI Native Workflow: Step 2, Development and execution; 1.2 AI-assisted development workflow.

Implementation and execution iterating · illustrative test cases
0runs0corrections0passing

Until the outcome is achievedThe agents may iterate between implementation, execution and analysis until the required outcome is achieved or human intervention is needed.

STEP 3 · PR REVIEW AND UAT

Checked by an agent, proven in UAT.

Every implementation goes through a pull request and the required PR review checks, a review, and then UAT, where the extracted data is compared with the expected output.

Standards

Correct use of the standard internal libraries and file-processing and loading logic.

Defects

Bugs, implementation issues and deviations from established project conventions.

Secrets

The code must not contain hardcoded credentials, API information or PII.

UAT

Missing records, duplicates and incorrect values; pagination, API responses, file formats and transformations verified.

Where people decideIf a check fails, the PR review agent adds review comments and the developer resubmits. Engineers resolve complex or unresolved findings and make significant technical decisions.

SourceWeb Scraping Team AI Native Workflow: Step 3, Pull request review and UAT; PR review checks.

A pull request on its way to production · illustrative
Pull requestScraper for ticket #1046Opened

PR review checks

  • Internal libraries and file loading
  • Bugs and convention drift
  • No credentials, API information or PII
  • Suggestions and improvements
Review commentRemove the hardcoded credential and load it from configuration. The developer fixes it and resubmits.
Approved
Deploy to UAT

UAT · extracted vs expected

UAT passed
STEP 4 · DEPLOYMENT AND CLOSURE

Closed only when production proves it.

The objective is to deploy the approved implementation, validate production behaviour and complete the ticket. The Execution Specialist supports the deployment workflow by running it, checking production execution and output, and flagging failures or unexpected behaviour.

Configure

Create the production dataset entries and fill in the production config with recipients.

Merge and deploy

Complete the required pull request and merge process, then run the production deployment.

Validate

Confirm the production run succeeded and verify the output and its data quality.

Close

Complete the ticket only after deployment and the required monitoring are satisfactory.

SourceWeb Scraping Team AI Native Workflow: 2, AI-assisted development lifecycle; Step 4, Production deployment and closure.

A production run · illustrative

Who owns itThe development team collectively owns production delivery and ticket closure.

AI IMPACT BY TICKET COMPLEXITY

The harder the ticket, the more the engineer leads.

Complexity is assessed during planning, and it sets how much of the work the agents can carry: the higher the score, the more the engineers lead.

25

Very high AI impact. Agents prepare, implement and validate with minimal review.

50

High. Largely automated, with occasional adjustments and clarification.

100

Significant productivity improvement; agents automate much of implementation and testing.

250

AI accelerates research, boilerplate and execution, but gains are more limited.

Human oversight on complex ticketsFor highly complex tickets, human oversight is essential. Without it, agents may make incorrect assumptions, produce overcomplicated implementations or require extensive iterations.

SourceWeb Scraping Team AI Native Workflow: 2.1 AI impact by development ticket complexity.

Who carries the work · illustrative balance, not a measurement
Typical engineering work

Expected AI impact

Human involvement

MAINTENANCE AND HANDOVER

About two weeks with the developer, then on-call once agreed.

Once a ticket moves to production, maintenance work starts on it. The Web Scraping team owns each deployment and its initial monitoring; ongoing support moves to on-call only once the BA team agrees a handover.

About two weeks

The developer who built and deployed it monitors it, investigates failures and fixes them.

Handover

For older deployments, the BA team decides whether support moves to on-call.

On-call

Once the handover is agreed, on-call owns the assigned operational issues.

Maintenance

The team monitors the scraper and, if it fails, fixes it with the on-call skill, /fix-incident-skill.

Why the handoverIt separates ownership of recent deployments from ongoing operational support, so every issue is handled by the appropriate team.

SourceWeb Scraping Team AI Native Workflow: 3 Production monitoring and operational handover; 4 On-call support.

In production · who handles an incident · illustrative
ON-CALL AND BUG RESOLUTION

The workflow proposes. The developer decides.

A developer picks up an incident and starts the /fix-incident-skill. It retrieves the context, investigates, finds the likely root cause and presents a proposed fix. Nothing is implemented until the developer approves it.

P1 · Escalated

Data problems first raised as P3 that need urgent attention; the Analyst team escalates them and developers start at once.

P2 · Critical

Known critical cases, highlighted before or flagged by clients as important. Trading data issues are an example.

P3 · All

Every other bug: generally not urgent, fixed through the normal process according to workload and planning.

Closed only when it is fixedThe developer validates the fix against the original incident. Code fixes follow review, deployment and production validation; operational issues with no code change follow operational recovery.

SourceWeb Scraping Team AI Native Workflow: 4 On-call support and bug resolution, 4.1 to 4.4.

Select a priority, then decide at the gate
ROLES AND RESPONSIBILITIES

Every step is someone’s job.

Four human roles and five AI agents, each with a defined job. The agents do the repeatable work; people own the requirements, the decisions, delivery and support.

People

AI agents

Select a roleEach one lights up the steps it is responsible for.

SourceWeb Scraping Team AI Native Workflow: 5 Roles and responsibilities, with sections 1 to 4.

Roles by step · mapped from the role descriptions · people AI agents
THE WHOLE PROCESS

The next website goes through the same steps.

From client request to production and ongoing maintenance: one scraping ticket, five lifecycle stages, AI agents for the legwork and engineers for the key decisions. On-call steps in only when a scrape fails, then it goes back to production.

  1. Client requestThe BA team gathers the requirement.
  2. Scraping ticketOne ticket per website URL, into the backlog.
  3. Weekly planningComplexity, selection and a developer for the upcoming week.
  4. /build-scrape-skillStarts the AI-assisted lifecycle.
  5. Research SpecialistAssess and explore; an execution plan.
  6. Implementation and Execution SpecialistsDevelop, run, correct and run again.
  7. Quality ReviewerQuality, extracted data, tests and readiness.
  8. PR review agentChecks, review and resubmission.
  9. UATExtracted data compared with the expected output.
  10. ProductionDeploy and validate; close the ticket once monitoring is satisfactory.
  11. MonitoringOngoing maintenance: about two weeks with the developer who built it, then continued monitoring.
  12. On-call, only if it failsFixed with /fix-incident-skill and a developer's approval, then back to production.

SourceWeb Scraping Team AI Native Workflow, sections 1 to 5.

Move