One standard process for every scraping request.
How the Web Scraping team develops, deploys and supports website scrapers: one scraping ticket per URL, the same five lifecycle stages every time, AI agents for the legwork and engineers for the technical decisions.
Scraping ticket · four AI agents · PR review · UAT · production · on-call
Sources
Web Scraping Team AI Native Workflow: 1 Overview and delivery approach, 1.1 Delivery model, 1.2 AI-assisted development workflow, 2 AI-assisted development lifecycle.
Every website is different. The process is not.
Data can sit in static or dynamic HTML, behind APIs, in Excel files, PDFs or downloads, spread across many pages and guarded by rate limits. The team answers that variety with one end-to-end process that combines engineering practice with AI-assisted workflows.
Static and dynamic HTML, APIs, Excel files, PDFs and downloadable documents.
Pagination, dynamically loaded content, authentication, chained requests and download mechanisms.
Request volumes, rate limits, access restrictions, blocking and edge cases.
The aimImprove development efficiency, maintain consistency and reduce manual effort, from the client requirement through to on-call support.
SourceWeb Scraping Team AI Native Workflow: 1, Overview and delivery approach; Step 1, Ticket assessment and requirements exploration.
One URL, one ticket.
The BA team talks to clients, gathers requirements and raises a scraping ticket for each website URL. Requests flow into a backlog continuously, and the development team plans the upcoming week from it.
Communicates with clients, gathers requirements and creates scraping tickets for incoming requests.
Each website URL is tracked through its own scraping ticket, and new requests continuously enter the backlog.
The development team plans the upcoming week: review the available tickets, assess complexity, select the work, assign each ticket to a developer.
Once assigned, each ticket follows the development lifecycle, started with /build-scrape-skill and using AI agents where applicable.
SourceWeb Scraping Team AI Native Workflow: 1.1 Delivery model; 2.1 AI impact by development ticket complexity; 5 Roles and responsibilities.
Client requests
BA team · one scraping ticket per URLBacklog
New requests keep arrivingUpcoming week
Complexity assessed, developer assignedFour agents do the legwork. Engineers make the calls.
The /build-scrape-skill starts the AI-assisted process, with four AI agents working on each ticket where applicable. They may iterate between implementation, execution and analysis until the outcome is achieved or a person needs to step in.
Investigates the target website, identifies data sources and extraction approaches, and prepares the implementation plan.
Generates the scraper code and its extraction, transformation and validation logic.
Runs implementations and tests, identifies failures and iterates on corrections.
Reviews implementation quality, extracted data, test results and production readiness.
People stay accountableThe development team retains responsibility for technical decisions, quality, production delivery and closure.
SourceWeb Scraping Team AI Native Workflow: 1.2 AI-assisted development workflow; 5 Roles and responsibilities.
Look before you build.
The goal is a feasible, reliable extraction strategy. The developer reviews the scraping ticket, the expected output, the acceptance criteria and the business context, and the Research Specialist investigates the website.
Find missing or ambiguous requirements, dependencies and risks; duplicate requests, existing scrapers or reusable components.
Technologies, structure and data sources: static or dynamic HTML, APIs, Excel files, PDFs or downloads.
Pagination, dynamic content, authentication, chained requests; request volumes, rate limits, access restrictions and blocking.
A structured execution plan for the Implementation and Execution Specialists.
Where people decideThe developer assesses technical feasibility and flags where business clarification is needed; the BA team takes it to the client. Significant technical decisions stay with engineers.
SourceWeb Scraping Team AI Native Workflow: Step 1, Ticket assessment and requirements exploration.
Every websiteThe Research Specialist works through every layer and hands a structured execution plan to the Implementation and Execution Specialists.
Build it, run it, fix it, repeat.
The Implementation Specialist turns the Research Specialist’s plan into code that follows project conventions and the internal libraries, with logging, error handling and request management built in. The Execution Specialist runs it, tests extraction across the relevant formats and feeds failures back for correction.
- Static and dynamic HTML parsing with CSS selectors, XPath and HTML tables; API integrations, authentication and chained requests.
- Basic, deep and dynamic pagination; Excel across sheets and complex layouts; PDF tables, with OCR where feasible.
- Request management, rate limiting, retries, exception handling and edge cases.
- Transformation, cleaning, normalisation, validation and quality checks.
Where people step inComplex business logic, extensive R&D, architectural decisions, or issues the agents cannot resolve reliably.
SourcesWeb Scraping Team AI Native Workflow: Step 2, Development and execution; 1.2 AI-assisted development workflow.
Until the outcome is achievedThe agents may iterate between implementation, execution and analysis until the required outcome is achieved or human intervention is needed.
Checked by an agent, proven in UAT.
Every implementation goes through a pull request and the required PR review checks, a review, and then UAT, where the extracted data is compared with the expected output.
Correct use of the standard internal libraries and file-processing and loading logic.
Bugs, implementation issues and deviations from established project conventions.
The code must not contain hardcoded credentials, API information or PII.
Missing records, duplicates and incorrect values; pagination, API responses, file formats and transformations verified.
Where people decideIf a check fails, the PR review agent adds review comments and the developer resubmits. Engineers resolve complex or unresolved findings and make significant technical decisions.
SourceWeb Scraping Team AI Native Workflow: Step 3, Pull request review and UAT; PR review checks.
PR review checks
- Internal libraries and file loading
- Bugs and convention drift
- No credentials, API information or PII
- Suggestions and improvements
UAT · extracted vs expected
Closed only when production proves it.
The objective is to deploy the approved implementation, validate production behaviour and complete the ticket. The Execution Specialist supports the deployment workflow by running it, checking production execution and output, and flagging failures or unexpected behaviour.
Create the production dataset entries and fill in the production config with recipients.
Complete the required pull request and merge process, then run the production deployment.
Confirm the production run succeeded and verify the output and its data quality.
Complete the ticket only after deployment and the required monitoring are satisfactory.
SourceWeb Scraping Team AI Native Workflow: 2, AI-assisted development lifecycle; Step 4, Production deployment and closure.
Who owns itThe development team collectively owns production delivery and ticket closure.
The harder the ticket, the more the engineer leads.
Complexity is assessed during planning, and it sets how much of the work the agents can carry: the higher the score, the more the engineers lead.
Very high AI impact. Agents prepare, implement and validate with minimal review.
High. Largely automated, with occasional adjustments and clarification.
Significant productivity improvement; agents automate much of implementation and testing.
AI accelerates research, boilerplate and execution, but gains are more limited.
Human oversight on complex ticketsFor highly complex tickets, human oversight is essential. Without it, agents may make incorrect assumptions, produce overcomplicated implementations or require extensive iterations.
SourceWeb Scraping Team AI Native Workflow: 2.1 AI impact by development ticket complexity.
About two weeks with the developer, then on-call once agreed.
Once a ticket moves to production, maintenance work starts on it. The Web Scraping team owns each deployment and its initial monitoring; ongoing support moves to on-call only once the BA team agrees a handover.
The developer who built and deployed it monitors it, investigates failures and fixes them.
For older deployments, the BA team decides whether support moves to on-call.
Once the handover is agreed, on-call owns the assigned operational issues.
The team monitors the scraper and, if it fails, fixes it with the on-call skill, /fix-incident-skill.
Why the handoverIt separates ownership of recent deployments from ongoing operational support, so every issue is handled by the appropriate team.
SourceWeb Scraping Team AI Native Workflow: 3 Production monitoring and operational handover; 4 On-call support.
The workflow proposes. The developer decides.
A developer picks up an incident and starts the /fix-incident-skill. It retrieves the context, investigates, finds the likely root cause and presents a proposed fix. Nothing is implemented until the developer approves it.
Data problems first raised as P3 that need urgent attention; the Analyst team escalates them and developers start at once.
Known critical cases, highlighted before or flagged by clients as important. Trading data issues are an example.
Every other bug: generally not urgent, fixed through the normal process according to workload and planning.
Closed only when it is fixedThe developer validates the fix against the original incident. Code fixes follow review, deployment and production validation; operational issues with no code change follow operational recovery.
SourceWeb Scraping Team AI Native Workflow: 4 On-call support and bug resolution, 4.1 to 4.4.
Every step is someone’s job.
Four human roles and five AI agents, each with a defined job. The agents do the repeatable work; people own the requirements, the decisions, delivery and support.
People
AI agents
Select a roleEach one lights up the steps it is responsible for.
SourceWeb Scraping Team AI Native Workflow: 5 Roles and responsibilities, with sections 1 to 4.
The next website goes through the same steps.
From client request to production and ongoing maintenance: one scraping ticket, five lifecycle stages, AI agents for the legwork and engineers for the key decisions. On-call steps in only when a scrape fails, then it goes back to production.
- Client requestThe BA team gathers the requirement.
- Scraping ticketOne ticket per website URL, into the backlog.
- Weekly planningComplexity, selection and a developer for the upcoming week.
- /build-scrape-skillStarts the AI-assisted lifecycle.
- Research SpecialistAssess and explore; an execution plan.
- Implementation and Execution SpecialistsDevelop, run, correct and run again.
- Quality ReviewerQuality, extracted data, tests and readiness.
- PR review agentChecks, review and resubmission.
- UATExtracted data compared with the expected output.
- ProductionDeploy and validate; close the ticket once monitoring is satisfactory.
- MonitoringOngoing maintenance: about two weeks with the developer who built it, then continued monitoring.
- On-call, only if it failsFixed with /fix-incident-skill and a developer's approval, then back to production.
SourceWeb Scraping Team AI Native Workflow, sections 1 to 5.