#Benchmarks

reins do hands a whole browsing goal to TypeSafe's Jev model. The other way to get the same thing done is the agent itself driving reins one command at a time: snapshot, click, type, read, repeat. This page measures the two against each other on 38 tasks in the maintainer's real Chrome, on 2026-09-28. Both are scored by the same independent check on the page the run ended on. Every number is recomputed from the raw JSON, and nothing is rounded in a flattering direction. The full report is on GitHub: 2026-09-reins-do.md.

Read the limitations before quoting a number. The dev figure is from a set the fixes were tuned on; the unseen figure is 8 tasks; and each arm fails tasks the other passes.

##Headline

• Dev set (30 tasks, tuned on for five rounds): reins do passed 124 of 150 runs (82.6%). Claude step by step passed 81 of 90 (90%); 81 of 87 (93.1%) without fx-newtab, a task its rules made impossible.
• Unseen set (holdout3, 8 tasks, never looked at while fixing): reins do passed 35 of 40 runs (87.5%) on its first and only run. Claude step by step passed 21 of 24 (87.5%).
• Speed and cost, on the 30 tasks both arms passed at least once: the median task was 4.6x faster with reins do (range 2.4x to 22.6x) and 111x cheaper in model spend (range 44x to 535x). Pooled over those tasks' passing runs, the median reins do run took 5.7 s and the median Claude run 24.5 s. The cost counts only Jev; the calling agent's own turn is not in it.
• The self-check: of the 166 reins do runs that ended done, 17 were wrong, and all 17 carried a self-check under 0.5 ("unsure"); 20 of the 149 right ones (13%) were marked unsure too.

##Results

reins do columns: runs passed, then the median time, Jev input tokens and Jev cost of the passing runs. Claude columns: runs passed, then the median time, cost and turns of the passing runs. Speed-up and cost ratio are Claude's median over reins do's, shown only where both arms passed at least once. Times are wall-clock from spawn to exit, with the tab's initial 2 s load outside the clock. Numbers in brackets point at the notes below the tables.

###Dev set, 30 tasks

###Holdout3, 8 unseen tasks

1. fx-newtab is not a fair comparison. Its link opens the docs in a new tab, and the step-by-step prompt forbids touching any tab but the one it was given, so Claude stopped every time and said so. It is in the Claude totals, which are also given without it.
2. fx-risky's correct outcome is a stop. reins do ended risky_action 5 of 5 times without clicking; Claude stopped before the delete 3 of 3 times. The runner first scored those Claude runs as failures by a rule meant for reins do's status; they are counted here by the page check, as the runner now does.
3. One passing run only, so that median is a single sample: mdn's Claude figures, huggingface's reins do figures.
4. Every debian run ended left_site, not done: the search form on www.debian.org submits to packages.debian.org, and on the holdout's code reins do stopped there as a site change. The page it stopped on was the goal, so the check passes. Round 5 treats a leading www. as the same site; a later confirmation run ended done 5 of 5 but is not counted, because a holdout is measured once.

Per set, the median task speed-up is 4.2x on dev (24 tasks) and 5.4x on holdout3 (6 tasks); the median cost ratio is 111x and 105x. On the same code as the holdout3 run (f1af855), the dev set scored 122/150 (81.3%).

• reins do, dev: 901 Jev calls, 6,256,057 input tokens, $0.263.
• reins do, holdout3: 205 Jev calls, 1,177,962 input tokens, $0.050.
• Claude, all 38 tasks: 114 runs, 1,728 turns, $21.83. The dearest run was openlibrary #1 (52 turns, $0.91, failed); the slowest was openlibrary #3 (318.9 s, failed).

###The self-check

Every done is put back to Jev once as a yes/no question (is every requirement in the goal visibly satisfied on this page?), and the answer's probability comes back as doneConfidence. Below 0.5 the output reads done (unsure: self-check 0.43) and the next: line says to verify. Recomputed from every done run of the dev and holdout3 runs:

• The 17 wrong ones: fx-filters ×5 (0.32 to 0.49), fx-datepicker ×5 (0.12 to 0.14), arxiv ×5 (0.38 to 0.45), github ×1 (0.44), huggingface ×1 (0.08). The highest, 0.49, sits one hundredth under the line.
• The 20 right-but-unsure ones are four whole tasks: cookies, fx-newtab, wolframalpha and gutenberg, 5 runs each (0.11 to 0.49). Unsure means "check the page", not "failed".

###Where step by step failed and reins do passed

Read from each failed run's final reply. Several point at reins' own step commands rather than at Claude's reasoning: reins snapshot, which the step-by-step arm reads pages with, did not look inside shadow roots at the time, while reins do's observation did, and stale refs from an earlier snapshot could point at hidden elements ("zero size"). Both are fixed in 0.6.1. The fx-wizard page also opened with its modal already showing, for both arms, because of a CSS bug in the fixture (fixed since).

• fx-newtab, 0/3. The link opens a new tab and the prompt forbids other tabs. Claude stopped and asked, in 3 to 4 turns.
• openlibrary, 0/3. Run 1 (52 turns, $0.91) searched but never opened the "Sort by" menu: its options are inside a web component and never appeared in a snapshot. Run 2 went through Advanced Search, landed on a "Verify you are human" page and stopped rather than click it. Run 3 never found the search field, which sits in a web component's modal.
• fx-wizard, 0/3. Runs 1 and 2: clicking the "Team" card came back "element has zero size", Next still advanced with the default Private, and Back and Close could not be used, so Claude stopped rather than create a Private project. Run 3 pressed Enter and then Space on "Create project", both worked, and it created the project twice; Claude noticed and said so.
• mdn, 1/3. Runs 2 and 3 never found MDN's search box (a web component): the search button reported zero size and reins press / was rejected as an unknown key. Run 1 got there in 34 turns.
• fx-settings, 2/3. Run 2 saw the switches without names, a click on the right one reported zero size, and a selector probe toggled a different switch; Claude stopped without saving and said which switch might have changed.

No step-by-step run hit its $2 cap or the 600 s limit; each failure is a stop Claude chose, with a reply saying why.

##Method

• Fixture tier (11 tasks): local pages served by the runner, each built around one widget that trips browser agents: an autocomplete that only commits on a picked suggestion, a calendar popover with a Done button, a consent modal over a search, a search with a filter and a sort menu, a link to a new tab, a three-step form, a "Delete draft" button (the right outcome is a stop), a settings panel with switches and Save, a paginated table, an FAQ accordion with a vote, and a modal wizard with a custom radio group.
• Live tier (27 tasks): public sites, no logins, no purchases: Google Flights, Wikipedia, GitHub, BBC Weather, arXiv, npm, MDN, Cambridge Dictionary, Hugging Face, Amazon, Hacker News, Wolfram|Alpha, the Python docs, PyPI, a date calculator, lit.dev, MusicBrainz, Open Library, crates.io, IANA, OpenStreetMap, Project Gutenberg, HN Search, Debian packages, the Rust std docs, pkg.go.dev and xe.com.
• Checks: a JavaScript check per task, run in the tab the run ended on. Each was validated without reins do: false on the start page and on near-miss states (searched but not sorted, the wrong edition, a date picked but not confirmed), true on a goal state reached another way. They are in tasks.mjs.
• A pass: the check returns true and the arm finished on its own inside its limit. A timeout never counts (none happened).
• reins do: reins do '<goal>' --fill … --timeout 60 --tab <id> --json, 90 s for four slow sites, at most 30 steps. 5 runs per task.
• Step by step: claude -p with the same goal and fill values, allowed only the reins step commands (snapshot, click, type, fill, select, press, hover, scroll, wait, text, screenshot), one per Bash call, --restricted with an allowlist, no reins do, no navigation by URL, no other tabs, stop before anything irreversible, $2 cap. 3 runs per task, because each costs about a hundred times more. Its cost is Claude Code's reported total_cost_usd.
• Dev and holdout: a holdout set is used once. After its first run it has been seen and its failures discussed, so it folds into dev: holdout v1 (7 tasks, 25/35) and holdout2 (8 tasks, 16/40) did. Holdout3 was frozen at f1af855 before any reins do run on it; its one run is the unseen number here, measured before round 5.
• Machine and network: the maintainer's Mac, reins 0.5.0, Node 24, on a residential network in India. Jev runs on the US West Coast, so every call crossed the Pacific, and the reins do times include that.
• Money: Jev input tokens at $0.042 per million, output free (TypeSafe's announcement, read 2026-09-27; TypeSafe says it cannot prove the price is not subsidised). Tokens are the API's own count.
• Arithmetic: medians over passing runs, lower-middle for even counts. Jev cost rounded up, Claude cost rounded down, ratios truncated, so no rounding favours reins do.

##How it got here

reins do was tuned in five rounds on the dev set. Each change was tried alone and A/B'd on the whole dev set, and kept when the total rose by at least 3 runs, no task at 4/5 or better fell below 3/5, and fx-risky still stopped every time. Three were kept below that bar on judgment, because their target moved from 0/5 and the drops elsewhere traced to unrelated coin flips. Tasks were only replaced or corrected when the task itself was broken, never because reins do failed them. The dev set grew as holdouts were folded in, so these rates are not one series on one set.

Round 5's total rose from 122 to 124, which the traces put down to noise: neither round-5 mechanism fired on a dev task.

Tried and reverted, each on its own A/B:

• Observe settle without the re-attach: fx-form 3/3 to 1/3 (a password-manager frame drops the debugger session).
• Offer the "Open field" click only on fields that open something: flights 2/3 to 0/3.
• Stale-element guard scoped to row containers: no change.
• Page text joined per block, tried twice: arxiv's failure changed shape but did not pass.
• Key-by-key typing without the resume: fx-form 5/5 to 1/5.
• A 300 ms grace for a click's navigation to start: no gain.
• Stable select option ids, and refusing to re-select the current option: datecalc 4/5 to 2/5 and 1/5.
• A stuck self-check that credits the run's recorded actions: no outcome changed.
• The refocus rule alone, and the slotted-click fix alone: no gain each; kept together.

##Limitations

Tasks reins do still fails, and why (from the traces):

• fx-filters, 0/5. Jev types the query, clicks the TypeScript filter before submitting it (the page drops unsubmitted text, as GitHub does), sorts, retypes and submits. The final URL has the query and the sort but no language. Every run ends done, self-check 0.32 to 0.49.
• fx-datepicker, 0/5. Jev opens the calendar, goes forward two months, picks November 18, then answers DONE without pressing the calendar's Done button, which is in its observation. Every run ends done, self-check 0.12 to 0.14.
• arxiv, 0/5. Jev searches, then opens a different paper whose title starts with the same words. The wanted paper is not on the newest-first results page, and the sort control is offered and never chosen. Every run ends done, self-check 0.38 to 0.45.
• fx-orders, 0/5. The order is on page 2. The pager is in the viewport and among Jev's candidates, and Jev answers BLOCKED (0.94 to 0.95) at step 0 every run. No self-check runs on a blocked stop.
• huggingface, 1/5. The page has two search boxes. In 4 runs Jev typed into the site-wide one, whose Enter opens the top model, then was stuck (×3, self-check 0.04) or said done there (×1, 0.08). The pass used the list's own "Filter by name" field.
• github, 4/5. The miss followed the "advanced search" link, which drops the typed query. After typing, Jev's choice between that link and giving up (which reins turns into a submit) is a near tie, so it flips from run to run.
• datecalc, 4/5. The miss clicked Calculate before setting the end day, then flipped the end-day select between 1 and 2 until the 30-step budget ran out.
• xe, 0/5 (holdout3). Stops needs_text at step 0: xe labels its amount input "Receiving amount", the same as the converted output, so no field matches the amount fill. A re-run with the currency fills the task first lacked was still 0/5. Unsolved.

• done is Jev's opinion. On this suite every wrong done was marked unsure, but that is 166 runs on 38 tasks, and the 17 wrong ones come from 5 tasks. Verify the page before reporting success.
• Every text value comes from --fill. reins do never makes up text; a field the fills do not cover stops the run with needs_text, and a site that mislabels its fields (xe) stops it even when the value was given.
• What it can see and do. It reads open shadow roots, never closed ones. A page that opens a tab from script (window.open) can still bring Chrome to the front. The risky-click stop is a heuristic on English words (buy, pay, send, delete…) and unlabelled buttons; other languages and odd labels can slip past it.
• A small benchmark. 38 tasks, one machine, one network, one day; n=5 for reins do, n=3 for Claude. A live site can change tomorrow. The fixture pages, the checks and every fix were written by Claude agents working for the maintainer.
• Tuned on dev. Five rounds of fixes were tuned on the dev set, so 82.6% is optimistic. The unseen 35/40 is the number that says how it generalises, and it was measured before round 5. It is 8 tasks: xe alone is 5 of its 40 runs, and its 5 debian passes ended left_site (note 4).
• What the cost leaves out. The Jev figure is only Jev: the calling agent still spends its own turn to issue the command, read the result and verify the page. The Claude figure is the whole session, including Claude Code's own system prompt and tool definitions (17 of 114 replies mention the maintainer's claude.ai connectors, so those were in context too).
• What the step-by-step arm measures. Claude plus reins' step commands, and some of its failures are the step commands' (reins snapshot does not read shadow roots). A better step toolkit, URL navigation or other tabs would score higher and change the speed and cost.
• Privacy. reins do sends the goal, your fill values, the URL and title, the visible text and the page's control labels to TypeSafe. The full list is under reins do and TypeSafe on the security page. Step by step sends page content to Anthropic. Either way it is opt-in.

##Reproduce

$ REINS_JEV_TRACE=/tmp/jev-trace.jsonl reins restart$ node packages/cli/scripts/bench-do.mjs --arm do --set dev --runs 5 --trace /tmp/jev-trace.jsonl --out r5.json$ node packages/cli/scripts/bench-do.mjs --arm do --set holdout3 --runs 5 --out h3.json$ node packages/cli/scripts/bench-do.mjs --arm manual --set all --runs 3 --out manual-v2.json

Other flags: --tier fixture|live, --tasks id1,id2, --claude-budget-usd, --out-dir and --dry (a self-test with no browser and no spend). It needs a TypeSafe key (reins key set typesafe) and claude on PATH. It costs money: about $0.32 of Jev for the 190 runs here and $21.83 of Claude for 114. It opens tabs in your real browser, in the foreground, for the whole run: about 25 minutes for dev, 5 for holdout3 and 90 for the Claude arm. Holdout3 is no longer unseen. The script is bench-do.mjs. Raw run files aren't kept in the repo; these commands regenerate them.

##Earlier version of this benchmark

The first version (2026-09-27) ran 4 live tasks (flights, wikipedia, github, cookies) 5 times per arm. On the build that shipped then, reins do passed 14/20 and Claude step by step 19/20; github was 1/5 for reins do, and on the tasks it passed reins do was 4.6x to 7.9x faster. It had no holdout and no fixture tier, which is why this version exists.