reins

← reins.tech

Benchmarks

reins do hands a whole browsing goal to TypeSafe's Jev model. The alternative is your agent driving reins itself: snapshot, click, type, repeat. We ran both on 38 tasks in a real Chrome on 2026-09-28, and scored both with the same check on the page each run ended on.

Results

About as accurate on tasks it had never seen, 4.6x faster and about 100x cheaper at the median. Neither side wins everywhere: each one fails tasks the other passes.

Runs passed

reins doClaude, step by step

8 unseen tasks

87.5% 35/40
87.5% 21/24

30 tuned tasks

82.6% 124/150
90% 81/90
Runs passed. 8 unseen tasks: reins do 35 of 40, Claude 21 of 24, both 87.5%. 30 tuned tasks: reins do 124 of 150 (82.6%), Claude 81 of 90 (90%).

The 30 tuned tasks are the ones reins do was fixed against over five rounds, so its score there flatters it. The 8 unseen tasks were run once, before anyone looked at them.

Time per task

Median run on each of the 30 tasks both passed. reins do was faster on all 30, from 2.4x to 22.6x.

reins doClaude, step by step

  • fx-form: reins do 8.1 s, Claude 141.5 s.
  • datecalc: reins do 10.1 s, Claude 115.7 s.
  • mdn: reins do 4.8 s, Claude 108.1 s.
  • flights: reins do 15.9 s, Claude 64.4 s.
  • rustdocs: reins do 5.9 s, Claude 46.8 s.
  • fx-settings: reins do 3.4 s, Claude 45.8 s.
  • lit: reins do 7.7 s, Claude 42.3 s.
  • fx-autocomplete: reins do 3.9 s, Claude 39.5 s.
  • github: reins do 11 s, Claude 34.6 s.
  • npm: reins do 3.5 s, Claude 31.7 s.
  • fx-consent: reins do 5.3 s, Claude 31.3 s.
  • pypi: reins do 7.4 s, Claude 30.1 s.
  • hnsearch: reins do 8.9 s, Claude 29.1 s.
  • cambridge: reins do 6.5 s, Claude 27.8 s.
  • crates: reins do 6.7 s, Claude 25.1 s.
  • debian: reins do 4.5 s, Claude 24.5 s.
  • pydocs: reins do 5.9 s, Claude 23.5 s.
  • amazon: reins do 9.3 s, Claude 23.2 s.
  • cookies: reins do 8.5 s, Claude 22.6 s.
  • gopkg: reins do 9.1 s, Claude 22.6 s.
  • osm: reins do 6.4 s, Claude 21.9 s.
  • fx-faq: reins do 2.5 s, Claude 21.4 s.
  • huggingface: reins do 5.1 s, Claude 21.3 s.
  • hackernews: reins do 3.8 s, Claude 20 s.
  • wikipedia: reins do 4.1 s, Claude 19.3 s.
  • gutenberg: reins do 2.8 s, Claude 19 s.
  • wolframalpha: reins do 6.8 s, Claude 18.9 s.
  • musicbrainz: reins do 4.9 s, Claude 17.4 s.
  • iana: reins do 1.8 s, Claude 13.5 s.
  • fx-risky: reins do 0.7 s, Claude 8 s.
Median seconds per task for the 30 tasks both passed. reins do was faster on every one.

Cost

reins do$0.32 of Jev for 190 runs
Claude$21.83 for 114 runs
Median task111x cheaper with reins do

Jev's spend covers Jev only. Your agent still spends a turn sending the command and checking the result.

Check every done

reins do ends done when Jev thinks the goal is met, then asks Jev once more to confirm. 17 of 166 done runs were wrong, and all 17 came back marked unsure. So an unsure done is the one to look at before you report success. It is not proof of failure: 20 correct runs were marked unsure too.

Where each one fails

reins do never passed these:

  • A date picker. It picks the date and never presses the calendar's Done button.
  • A search with filters. It sets the language filter before submitting, and loses it.
  • arXiv. It opens a different paper whose title starts with the same words.
  • A paginated table. The order is on page 2, and it gives up on page 1.
  • xe.com. The site labels its input and its output the same, so the amount never lands.

Claude driving reins step by step never passed these:

  • Open Library. The sort menu and the search field sit inside web components.
  • A modal wizard. Clicks on a custom radio card reported "zero size". This and Open Library were mostly gaps in reins snapshot, fixed in 0.6.1.
  • A link that opens a new tab, because its rules forbade other tabs.

It is a small benchmark: 38 tasks, one machine, one day, from India to Jev's US servers. Live sites change. Jev input costs $0.042 per million tokens, TypeSafe's published price. reins do sends page text to TypeSafe; here is exactly what.

Run it yourself

$ node packages/cli/scripts/bench-do.mjs --arm do --set dev --runs 5 --out do.json$ node packages/cli/scripts/bench-do.mjs --arm manual --set all --runs 3 --out claude.json

Needs a TypeSafe key (reins key set typesafe) and claude on your PATH. It opens tabs in your real browser for about two hours and spends about $22, nearly all of it on Claude. --dry checks the setup with no browser and no spend. The script is bench-do.mjs.

Full report

Per-task numbers, how each task is checked, every fix and what it moved, the ones we reverted, and the traces behind each failure: 2026-09-reins-do.md on GitHub.

Other pages