Web scraping

#How web scraping works

A scraper can parse the HTML a server returns, or call the internal requests a page makes to load its data. Some scrapers use a headless browser to render the page before parsing it. Either way, the output is structured data that a program can use.

#Why it matters for AI agents

Agents need web data, and scraping is the default route to it. Scraping suits agents poorly for two reasons. Parsers break when a page changes, and many sites run bot protection that blocks automated access.

#Wield and web scraping

A Wield recipe is not a scraper. It is a small public, signed JSON file that describes the requests a site's own pages make. The Wield extension runs it in your browser, as you, with the session you already have.

Recipes run under fixed limits: 12 steps, 40 requests, 5 MB per response and 60 seconds per run. Requests go only to the origins a recipe lists. Any write needs your Allow. Wield does not try to get past bot protection. When a site shows a check, the run stops and you solve it in your own tab.

#Related terms

• Internal API
• Recipe
• Rate limiting
• Site challenge
• Headless browser

#FAQ

#Is web scraping legal?

This page does not give legal advice. Rules depend on the country and on the site's terms. Check the site's terms before you automate anything. Wield's recipes note known restrictions for each site.

#How is a recipe different from a scraper?

A scraper fetches pages and parses them with its own logic. A recipe is data: it lists the requests, the inputs and the extraction. It runs only against the origins it lists, in your browser, under the limits above. Wield checks each recipe's signature, and wield-mcp refuses a recipe that is unsigned or has changed.