Appearance

Benchmarks ​

lessloc is measured on real changes to real open-source projects. Each task is a change a human merged. A fresh agent session gets the project's code from just before that change and the text of its issue or pull request. It is judged by the tests the human's change added, which the agent never sees. The results come from this benchmark. The other tools on this page replay finished code without a model.

Results ​

lessloc 0.10.0 against no tool, with Haiku (claude-haiku-5-5) in Claude Code 2.1.303. There are 117 tasks from three Python projects, two sessions per task for each setup, 468 sessions in all.

lessloc wrote less code on every project and cost 23% less overall, with the same results: tasks passed show no measurable difference.

  • Same results. lessloc's sessions passed 157 times out of 234, and no tool's 161: a difference of 2 tasks, with a 95% confidence interval from 5 fewer to 0.5 more, which includes zero.
  • Less code. It wrote 27% fewer production lines per task, with an interval of 23% to 32% fewer.
  • Lower cost. It cost 23% less per task, with an interval of 10% to 31% less, its own hooks included.
Pass rate: no toolPass rate: lesslocProduction codeCost per task
starlette, 43 tasks76.7%75.6%26% less26% lower
pydantic-settings, 45 tasks60.0%57.8%23% less22% lower
w3lib, 29 tasks70.7%69.0%36% less15% lower
All 117 tasks68.8%67.1%27% less23% lower

Dollar figures are Claude Code's own estimates at API prices. The run measured the lessloc 0.10.0 release candidate; the release adds the precision and command fixes made after the run.

In full, as bench/real/stats.py prints it from the run's results. Each cell reads no tool → lessloc, with the 95% interval of the difference in parentheses:

TasksPassed: no tool → lessloc, difference in tasksProduction lines per taskCost per taskEarlier changes broken per taskTest lines per task
starlette4376.7% → 75.6%, −0.5 (−1.5 to 0)14.0 → 10.4, −26% (−30% to −19%)$0.0201 → $0.0148, −26% (−39% to +2%)0.30 → 0.1630.5 → 16.8
pydantic-settings4560.0% → 57.8%, −1.0 (−3.0 to 0)17.1 → 13.1, −23% (−33% to −16%)$0.0250 → $0.0195, −22% (−33% to −9%)0.67 → 0.6729.6 → 19.1
w3lib2970.7% → 69.0%, −0.5 (−2.0 to +1.0)14.0 → 8.9, −36% (−49% to −28%)$0.0129 → $0.0109, −15% (−22% to −7%)0.00 → 0.0017.1 → 8.6
All11768.8% → 67.1%, −2.0 (−5.0 to +0.5)15.2 → 11.1, −27% (−32% to −23%)$0.0202 → $0.0156, −23% (−31% to −10%)0.37 → 0.3226.8 → 15.6

Every task ​

The tasks of each project, in the order their changes were merged. Each setup ran every task twice: passed is how many of the two sessions passed, and lines and cost are the mean of the two.

starlette ​

TaskMergedPassed: no toollesslocProduction lines: no toollesslocCost: no toollessloc
pr34102026-08-052/22/24840.5$0.016$0.022
pr34182026-08-062/22/237.528.5$0.011$0.010
pr34192026-08-072/22/221.5$0.009$0.008
pr34202026-08-072/22/275$0.011$0.007
pr34212026-08-072/22/216.515.5$0.008$0.007
pr33892026-08-082/22/211$0.005$0.005
pr34302026-08-082/22/2144$0.012$0.011
pr34312026-08-080/20/297$0.011$0.011
pr31302026-08-080/20/2137$0.013$0.008
pr34712026-08-250/20/26.55$0.007$0.007
pr34722026-08-262/22/25036$0.037$0.025
pr34362026-08-262/22/27.56.5$0.017$0.013
pr27672026-08-300/20/226$0.009$0.010
pr34892026-08-300/20/233$0.016$0.014
pr34922026-08-300/20/29.56.5$0.022$0.024
pr35102026-09-052/22/233$0.008$0.008
pr33762026-09-052/22/222$0.008$0.007
pr35162026-09-052/21/216.511$0.014$0.011
pr35172026-09-052/22/2136$0.014$0.013
pr33722026-09-062/22/233$0.006$0.008
pr32012026-09-062/22/232.5$0.006$0.006
pr34762026-09-062/22/265.5$0.010$0.009
pr32712026-09-072/22/266$0.006$0.006
pr35322026-09-122/22/264.5$0.008$0.008
pr35452026-09-122/22/222$0.006$0.007
pr35632026-09-200/20/22620.5$0.020$0.019
pr35682026-09-232/22/265$0.007$0.008
pr35832026-09-260/20/222$0.009$0.008
pr35672026-09-272/22/232.5$0.010$0.007
pr35652026-10-032/22/22011.5$0.021$0.015
pr36182026-10-042/22/25.56$0.007$0.007
pr36172026-10-042/22/288$0.007$0.006
pr31862026-10-042/22/222$0.005$0.004
pr36402026-10-062/22/211$0.004$0.005
pr36512026-10-092/22/213$0.005$0.011
pr36312026-10-132/22/296.5$0.011$0.017
pr36712026-10-170/20/243$0.009$0.007
pr36222026-10-172/22/212.56.5$0.014$0.012
pr34912026-10-180/20/2202.5140$0.418$0.214
pr35852026-10-182/22/22.52$0.008$0.009
pr35422026-10-182/22/264$0.008$0.010
pr36752026-10-182/22/221$0.005$0.006
pr36792026-10-182/22/233$0.006$0.008

pydantic-settings ​

TaskMergedPassed: no toollesslocProduction lines: no toollesslocCost: no toollessloc
pr8782026-06-030/20/250.520.5$0.033$0.023
pr8762026-06-052/22/22423$0.029$0.023
pr8822026-06-070/20/248$0.011$0.008
pr8892026-06-192/22/217.51$0.013$0.012
pr8602026-06-290/20/27165$0.218$0.105
pr8982026-07-062/22/2187.5$0.018$0.017
pr9002026-07-070/20/21810.5$0.017$0.014
pr9022026-07-082/22/23939$0.030$0.029
pr9052026-07-082/22/28.52$0.017$0.010
pr9102026-07-142/20/213.56.5$0.010$0.009
pr9122026-07-152/22/222$0.007$0.008
pr9132026-07-160/20/265.5$0.009$0.009
pr9142026-07-220/20/232$0.007$0.007
pr9202026-08-032/22/221$0.009$0.010
pr9262026-08-062/22/211$0.007$0.007
pr9402026-08-210/20/23828.5$0.026$0.021
pr9452026-08-232/22/266$0.004$0.006
pr9472026-08-282/22/24.53$0.008$0.008
pr9582026-09-042/22/264$0.012$0.012
pr9592026-09-112/22/212$0.008$0.009
pr9682026-09-112/22/211$0.005$0.006
pr9752026-09-180/20/213.58$0.018$0.014
pr9732026-09-182/22/223$0.012$0.011
pr9722026-09-190/20/215.513$0.024$0.025
pr9772026-09-212/22/222.5$0.012$0.011
pr9762026-09-212/22/28.58$0.011$0.009
pr9792026-09-220/20/2105$0.042$0.030
pr9802026-09-232/22/212.512.5$0.009$0.011
pr9812026-09-251/21/21313.5$0.022$0.018
pr9882026-09-252/22/266$0.015$0.016
pr9892026-09-252/22/233$0.012$0.016
pr9922026-10-020/20/24.54$0.012$0.012
pr10012026-10-022/22/216.513$0.017$0.018
pr9942026-10-022/22/23.53$0.012$0.011
pr9972026-10-090/20/231.525.5$0.018$0.022
pr10092026-10-092/22/232.5$0.010$0.007
pr10102026-10-090/20/276.5$0.013$0.012
pr10062026-10-090/20/21718.5$0.021$0.019
pr10042026-10-120/20/23627.5$0.028$0.038
pr10082026-10-120/20/22210.5$0.037$0.014
pr10192026-10-132/22/266$0.019$0.015
pr10252026-10-130/20/23016$0.017$0.016
pr10292026-10-132/22/225.518.5$0.037$0.019
pr10222026-10-132/22/23027$0.021$0.016
pr10332026-10-141/21/2116.598$0.188$0.144

w3lib ​

TaskMergedPassed: no toollesslocProduction lines: no toollesslocCost: no toollessloc
pr2642026-06-152/22/211$0.005$0.005
pr2662026-06-152/22/211$0.007$0.006
pr2682026-06-172/22/222$0.009$0.007
pr2712026-06-222/22/222$0.006$0.007
pr2732026-07-092/22/211$0.006$0.006
pr2862026-07-232/22/211$0.009$0.008
pr2882026-07-242/22/218.59$0.035$0.021
pr2892026-07-242/22/213.512.5$0.011$0.009
pr2902026-07-272/22/26.51.5$0.010$0.008
pr2912026-07-282/22/277$0.007$0.005
pr2922026-07-292/22/232$0.008$0.007
pr1672026-08-030/20/21812$0.011$0.007
pr3002026-08-131/21/214.510.5$0.010$0.011
pr3012026-08-132/22/281.5$0.009$0.009
pr3022026-08-132/22/22.51$0.009$0.007
pr2932026-08-131/21/241.513$0.021$0.016
pr3032026-08-130/20/27.51.5$0.006$0.005
pr2992026-08-130/20/225.512$0.013$0.012
pr3052026-08-172/22/277$0.009$0.009
pr3072026-08-302/22/28.53.5$0.008$0.010
pr3172026-09-082/21/28.53.5$0.014$0.012
pr3212026-09-092/22/24.55$0.005$0.006
pr3332026-09-170/20/2111.585$0.023$0.014
pr3322026-09-170/21/216.511$0.023$0.022
pr3342026-09-171/20/225.520$0.022$0.016
pr3372026-09-170/20/2169.5$0.013$0.014
pr3382026-09-180/20/210.55$0.016$0.018
pr3422026-09-212/22/23.51.5$0.008$0.008
pr3432026-09-222/22/21915.5$0.041$0.029

Measures ​

  • Passed: every hidden test of the task passed after the session. A task's pass rate is the share of its sessions that passed. The difference is counted in tasks: the sum, over tasks, of lessloc's pass rate minus no tool's.
  • Production lines: the non-blank lines a session's change added to the project's Python files outside its tests, comments and docstrings included. Test lines are the same count inside them: tests/, test/, testing/, test_*.py, *_test.py and conftest.py.
  • Cost per task: the mean cost of a task's sessions, lessloc's own additions included. A session that runs out of its 20 minutes is priced from the tokens it streamed; in this run, none did.
  • Earlier changes broken: how many of the project's earlier judged changes had hidden tests that failed after the session. In pydantic-settings, pr860 and pr979 show the same earlier changes broken in every session of both setups.
  • Intervals: 95% bootstrap intervals over tasks, paired by task, from 4,000 resamples. Each task counts once, as the mean of its sessions.

The projects ​

The tasks are the judged changes among each project's first 70 changes to its source after a release:

  • starlette from 1.3.1: 43 tasks, merged from August 5 to October 18, 2026. One more judged change, pr3463, is left out, as planned before the run: its hidden tests need packages the project added after 1.3.1, so in the environment the benchmark builds at the start, even the human's change fails them.
  • pydantic-settings from v2.14.1: 45 tasks, merged from June 3 to October 14, 2026.
  • w3lib from v2.4.1: 29 tasks, merged from June 15 to September 22, 2026.

lessloc's checks were designed on other projects, tomlkit and click. pydantic-settings and w3lib were the validation projects: the checks were measured on both, this version was chosen on their results, and the note at a turn's 40th tool call came from long sessions on pydantic-settings. starlette was the held-out project: no version of lessloc was scored or tuned on it before this run.

Their project.json files, with uv on the PATH:

json
{ "repo": "Kludex/starlette", "start": "1.3.1", "branch": "main", "source": ["starlette"], "tests": ["tests"], "max": 70,
  "setup": "UV_PYTHON=3.12 uv sync -q --frozen --no-default-groups --group dev",
  "python": ".venv/bin/python" }
json
{ "repo": "pydantic/pydantic-settings", "start": "v2.14.1", "branch": "main", "source": ["pydantic_settings"], "tests": ["tests"], "max": 70,
  "setup": "UV_PYTHON=3.12 uv sync -q --frozen --no-default-groups --group testing --all-extras && uv pip uninstall -q -p .venv/bin/python pytest-pretty",
  "python": ".venv/bin/python" }
json
{ "repo": "scrapy/w3lib", "start": "v2.4.1", "branch": "master", "source": ["w3lib"], "tests": ["tests"], "max": 70,
  "setup": "uv venv -q --allow-existing -p 3.12 .venv && uv pip install -q -p .venv/bin/python --exclude-newer 2026-10-18T00:00:00Z pytest-timeout -e . pytest && echo 'import os; os.environ.setdefault(\"PYTEST_TIMEOUT\", \"60\")' > .venv/lib/python3.12/site-packages/zz_bench_timeout.pth",
  "python": ".venv/bin/python" }

How the benchmark works (bench/real) ​

bench/real turns a real open-source project's history into tasks. The start is a released version. The tasks are the later changes to its source, in the order each reached the main branch, up to the project's max (80 by default); a change of more than 1,000 source lines is skipped. Each task's text is the linked issue, else the pull request.

The agents cannot look up the real change. Their copy of the project has no history past the task's start and no remote. In Claude Code they have no web tools. In their shell, the usual ways online fail: gh, curl and wget on the PATH, git remotes, and the pip and uv package indexes.

A task is judged by the tests its real change added, which the agent never sees. Those tests are kept only if they fail before the change and pass after it, both in the layout the grader uses. A test that uses a name the change introduced, where the task text never mentions that name, is dropped, as SWE-bench Verified's review demands. A hidden test that a later real change broke on purpose retires at that change. After each task, the hidden tests of every earlier judged task run again.

The harness runs tasks in two ways:

  • Per-task replays, which the results use. Each task starts from the project's own code just before its change landed, with no later commit in reach, so no task depends on another, as in SWE-bench.
  • Timelines. One setup's run through the whole history: every task goes to a fresh session, on the code exactly as earlier sessions left it, with no resets, so each setup builds on its own code. Tasks without hidden tests are done too, so the code later sessions build on never misses a step. Timelines also measure code growth: how much the source folder grew, in non-whitespace characters of non-comment lines, so packing code onto fewer lines does not shrink it.
sh
node bench/real/mine.mjs project.json clone > candidates.json      # changes and their texts, via the GitHub CLI
node bench/real/prepare.mjs candidates.json clone prepared/         # hidden tests, judged or not, retirement
node bench/real/run.mjs prepared/ clone --model haiku --dry-run     # the start fails every judged task; no commit after it
node bench/real/run.mjs prepared/ clone --model haiku --arms none,lessloc --runs 2 --replay list.json   # list.json: [{"task": "pr3410"}, ...]
python3 bench/real/stats.py <results.json> prepared/                # the tables above

The harness takes Python projects tested with pytest whose history is linear: every change on the main line is a squash commit or a pull-request merge. Pick projects without branch syncs, such as merges from a stable branch.

Reproducing the results ​

For each project, mine and prepare its tasks from its project.json above, in a full clone with its tags. Then list the project's task ids from Every task in a file, as [{"task": "pr3410"}, {"task": "pr3418"}, ...], and run:

sh
node bench/real/run.mjs <prepared dir> <clone> --model haiku --arms none,lessloc --runs 2 --replay <task list>
python3 bench/real/stats.py <starlette results.json> <starlette prepared dir> <pydantic-settings results.json> <pydantic-settings prepared dir> <w3lib results.json> <w3lib prepared dir>

Mining fetches each task's text from GitHub as it reads on that day, so an issue edited since this run gives a slightly different task. Keep your candidates.json to rerun the same texts later.

  • --model haiku must be given, because the harness defaults to sonnet.
  • --runs is sessions per task in a replay, and timelines per setup otherwise (default 3). --jobs is how many run at once (default 3).
  • --tasks <n> takes the first n tasks of a timeline; without it, every task runs.
  • --replay <file> runs single tasks. The file is a JSON list of {"task": <task id>}, each started from the project's own code just before the task's change landed. It can also list {"repo": <kept timeline repository>, "task": <task id>}, each started from the state that timeline reached just before the task.
  • Each arm in --arms is one setup: none, lessloc, lessloc-no-<component> (an ablation), or lessloc-old (another lessloc checkout, given with --old <dir>). The first arm is the baseline.
  • --host claude is the default, and --claude <command> picks the Claude Code command.
  • --host codex or --host opencode runs the same tasks in Codex or OpenCode. Each timeline, and each replayed task, gets a config home of its own for the tool, so the user's own configuration never leaks in. Codex runs with CODEX_HOME set to a fresh folder that holds lessloc's hooks only in lessloc arms; it signs in with OPENAI_API_KEY or a link to your ~/.codex/auth.json. Codex reports tokens but no cost, so --price <in>,<cached in>,<out> (dollars per million tokens) turns its tokens into one. OpenCode needs --model <provider>/<model>, runs with a fresh XDG_CONFIG_HOME, loads lessloc's bridge plugin only in lessloc arms, and reports its own cost estimate from tokens and the model's list price.
  • A session cut short by a usage limit, or one that returned no result, runs again. One that runs out of its 20 minutes counts as it ended, priced from the tokens it streamed at what the run's complete sessions paid per input-side token, so it never counts as free.
  • With bubblewrap installed (bwrap), each session sees only its own processes, so an agent's pkill -f <pattern> cannot end another session, its tests or the run. LESSLOC_BENCH_SHARED_PIDS=1 turns this off.
  • TMPDIR decides where the results JSON and each timeline's kept repository go, so point it at a disk with room. When uv's cache is on another filesystem, set UV_LINK_MODE=copy.
  • The results JSON holds every session and records the versions (Claude Code, model, lessloc) and which code ran: the harness's commit and a hash of its uncommitted diff, and each lessloc setup's commit, diff hash and LESSLOC_OFF switches. Run from a clean checkout, so the diff hash is empty.

Refusing unfinished work, without a model (bench/candidates.mjs) ​

For each judged task of a prepared real project, the human's change is a finished candidate. Variants of it leave work out: the tests, half of the fixed files, a block replaced by a stub, the final test run, or a test run that comes before the last edit. Each candidate is replayed through lessloc's hooks the way an agent would make it, with the request, one write per file, a real run of the project's tests and a final reply that claims they pass. The task's hidden tests say whether its code is finished. A hold on finished work is a false reject; a candidate with work left out that goes through is a false accept.

sh
node bench/candidates.mjs <prepared dir> <clone> [--tasks pr471,pr486]

Precision on code humans merged, without a model (bench/replay.mjs) ​

Commits humans already merged are the false-positive test. Each one is replayed through lessloc's review as check --base <parent> sees it, and through the turn end's check of requested names, with the commit's message as the request.

sh
node bench/replay.mjs <clone> [commits=30] [--show copy,untested] [--json <file>]

In full clones at ripgrep 3fce3b5, ky 3541888, cobra adbc881, gson e2292f9 and click 2247b35:

  • 30 commits each. copy fired 10 times: 8 on code repeated verbatim, and 2 on ripgrep code with the same shape but different fields or values (two builder setters, two flag definitions). The other kinds that stop a turn fired 23 times: weakened-tests 9, untested 9, suppression 3 and commented-out 2; these 23 were counted, not hand-labeled.
  • 200 commits each. The messages ask for 121 names (ripgrep 18, ky 59, cobra 7, gson 12, click 25), and unaddressed would have held none of the 1,000 commits.

What the hooks add to a session, without a model (bench/weight.mjs) ​

One scripted session (session start, a prompt, a search, an edit, a test run, the turn end) is replayed through a lessloc checkout's hooks on a throwaway clone of a repository. It prints each event's time and the tokens it adds to the agent's context.

sh
node bench/weight.mjs <clone> --file <path to edit> --prompt <text> [--lessloc <dir>] [--test <command>] [--runs 5] [--show]

Scope ​

The results cover three Python projects, all with Haiku in Claude Code. --host and --model run the same tasks in Codex or OpenCode and with other models. Correctness is judged by the hidden tests of each human change.

Last updated:

Released under the Apache-2.0 license. · llms.txt