How we benchmark recipes against browser agents

#Setup of both sides

The browser agent sees the page through screenshots and clicks, as stock Browser Use does. Browser Use's own LLM judge is off, because the machine check decides success. Its answer must also fit a structured schema for each task.

#Tasks and queries

The task file has one task for each read recipe. Each task has three queries. A query has two forms: a natural-language instruction for the agent, and the exact recipe parameters for the same request. One train query, for example, asks for a connection from Berlin Hbf to München Hbf after 08:00 on a fixed Friday. Dates are fixed and fall in the week after the run. Parameters are validated against each recipe's schema before a run starts.

Runs go round-robin. Run 1 of every query goes first, then run 2, then run 3. A budget stop therefore still leaves every query with a result.

#The machine check

The same check decides both sides:

1. The browser agent must return output that fits a per-task schema, for example a list of journeys with departure, arrival and price.
2. The recipe's output is mapped into the same shape with a JMESPath expression.
3. The task's rule runs on both. Examples: at least one connection departing on the requested date, with departure, arrival and price, or at least five listings with title and price.

Prices must be numbers above zero, or text that contains a digit. Departures must start with the query date. The check tests that a result is plausible and complete. It does not compare values between the two sides.

#Results as of 7 November 2026

At the median, the recipe is about 85 times faster (908 ms against 77,577 ms). Two summaries appear in the results. The pooled figure adds up every run, so tasks with more runs weigh more. The per-task figure gives each task equal weight, and it is the headline. The time figure is the median of the per-task medians.

#Caveats: preliminary, paced and blocked runs

• Preliminary. The benchmark was still running on 7 November 2026. A task with no runs on a side shows no number for that side.
• Paced. Runs are paced at about 30 site requests per hour. The browser-agent side moves slowly, so the run counts grow over time.
• Blocked runs. A run counts as blocked when it fails with a 403 or 429 on a document or API call, or when the agent reports a bot check or an access denied page. In our runs, bot protection blocked the cloud browser on bahn.de, Lieferando, Zalando, ImmoScout24, Idealo, Eventim, mobile.de, Walmart, Zillow and OpenTable. We say "in our runs" because the data comes from one setup on one set of dates.
• Shared runs. The Zalando HTML-fallback recipe reuses the browser-agent runs of the main Zalando task, so those runs count in both Zalando rows.
• Wield's misses. The data shows where the recipe failed. The Zalando recipe passed 0 of 9 runs. ImmoScout24 passed 0 of 2. mobile.de passed 5 of 9. Airbnb passed 8 of 9.

Incidents and exclusions are listed in the method notes in the project repository.

#Reproduce it

Everything needed to check the numbers lives in the project repository:

• bench/tasks.yaml holds the tasks, the queries and the check rules.
• bench/results.json holds the aggregated table shown on the benchmark page.
• bench/runs/ holds the raw logs, one per run. They record destinations, timing, tokens and check results, and they hold no page contents.
• bench/README.md describes the full method.

To regenerate the table from the raw runs, run uv run wield-bench aggregate.

#FAQ

#Why is the benchmark marked preliminary?

It was still running on 7 November 2026. Each task still has planned runs to complete, and pacing keeps the browser-agent side slow. The numbers can change as runs finish, so read them with the run counts in the table.

#Why does the recipe side run in a real browser?

Because that is how Wield runs. A recipe runs in the user's browser, with the user's own session. The browser-agent side runs the way cloud agents run, in a fresh headless Chromium with an empty profile. Each side is tested the way it is actually used.

#Is the browser agent set up to fail?

No. It uses stock Browser Use settings with three changes: navigation is limited to the site's domains, output must fit a structured schema, and the LLM judge is off because the machine check decides success.

#What does the cost figure mean?

It is the model's list price per run, not a bill. A Wield run uses no model, so its cost is $0.