A collection of Agent Skills for web scraping and structured data extraction. Each skill teaches an agent how to think about a scraping problem — API-first, anti-bot reality, what fields matter, build-vs-buy — and routes it to the fastest working path.
These skills are maintained by Thirdwatch, which runs 70+ production web scrapers on the Apify Store. Where a maintained scraper already exists for a target site, the relevant skill points to it so you don't have to fight Cloudflare/DataDome yourself. Where one doesn't, the engineering skills show you how to build it.
| Skill | Use it when |
|---|---|
web-scraping-playbook |
You're starting any scraping task and want the build-vs-buy decision tree, the cost-first technique ladder (HTTP → TLS spoof → stealth browser), and compliance basics. |
anti-bot-scraping |
A site is blocking you — Cloudflare Turnstile, DataDome, Akamai, PerimeterX, 403s, JS challenges, CAPTCHAs. |
apify-actor-builder |
You want to package a scraper as a deployable, monetizable Apify Actor (Python), with input/output schemas and pay-per-event pricing. |
| Skill | Covers |
|---|---|
job-market-scraping |
Jobs, salaries, candidates: LinkedIn, Indeed, Glassdoor, Naukri, Google Jobs, RemoteOK, Wellfound, Monster, ZipRecruiter, Reed, Adzuna, Upwork, CutShort, AmbitionBox. |
ecommerce-product-scraping |
Products, prices, variants: Amazon, Flipkart, AliExpress, Myntra, Nykaa, Meesho, Snapdeal, Noon, AJIO, FirstCry, Tata Cliq, Shopify. |
review-reputation-scraping |
Reviews & ratings: Trustpilot, G2, Capterra, Yelp, Google Maps, Shopify review widgets. |
social-media-content-scraping |
Social & content: Twitter/X, Instagram, TikTok, Reddit, LinkedIn posts, Pinterest, YouTube (+ transcripts), Facebook Ad Library, IMDb. |
business-lead-data-scraping |
B2B leads & company data: Google Maps, IndiaMart, JustDial, LinkedIn employees/candidates, GST verification, UN Comtrade trade data, Product Hunt. |
real-estate-scraping |
Property listings: Rightmove (UK), 99acres, MagicBricks, NoBroker, CommonFloor (India). |
travel-hotel-scraping |
Hotels & rentals: Booking.com, Trip.com/Ctrip, TripAdvisor, Turo. |
food-delivery-scraping |
Restaurants & menus: Swiggy, Zomato, Talabat, Deliveroo, Noon Food. |
seo-serp-scraping |
Search & SEO: Google Search, Google News, bulk SEO audits, sitemap/subdomain discovery. |
Each skill is a folder with a SKILL.md. To use one with Claude Code, copy the folder into your project's .claude/skills/ (or ~/.claude/skills/ for all projects):
git clone https://github.com/thirdwatch-dev/scraping-skills
cp -r scraping-skills/skills/web-scraping-playbook ~/.claude/skills/The skill activates automatically when your request matches its description. The same SKILL.md files work with any agent runtime that supports the Agent Skills format.
The scrapers run on Apify with pay-per-result pricing (you're billed per row returned, with a free tier). Run one from the command line:
curl -X POST "https://api.apify.com/v2/acts/thirdwatch~google-maps-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"search": "coffee shops in Austin", "maxResults": 25}'Get a free token at console.apify.com. Each actor's exact input fields are on its Store page (linked from every skill).
These skills target publicly accessible data. Respect each site's Terms of Service and robots.txt, rate-limit your requests, and handle personal data in line with applicable law (GDPR, CCPA, India DPDP, etc.). Don't use them to scrape behind logins you're not authorized for, or to evade access controls.
MIT — see LICENSE. Built by Thirdwatch · Apify Store profile.