Skip to the content.

bees eval runs the whole factory against a fixture repository and grades the result with the fixture’s own test, the way SWE-bench Lite grades a coding agent. bees eval <role> runs one role instead, against cases that grade that role’s session on its own. Use either to compare a prompt, model or profile change against a baseline run.

Evals run real agent sessions and cost money. They are not part of dagger check.

Running evals

From a directory with an evals/ directory in it (the busybees repository ships one):

bees eval                          # every case under evals/
bees eval --case hello             # only evals/hello
bees eval --profile fast           # every role on the profile named fast
bees eval developer                # every case under evals/developer/
bees eval qa --case broken-greeting

The output is one row per case, then what failed:

CASE   RESULT  SCORE  STOP  COST   TURNS  DURATION  PROFILE
hello  pass    -      done  $0.84  41     6m12s     default (bees.toml)

report: /home/me/src/busybees/.bees/evals/20260918-190412/report.json

SCORE is the mean of the case’s graded checks, and - for a case with none.

bees eval exits non-zero when any case fails. STOP says why the run of a case ended:

Stop Meaning
done Every seeded issue is closed or carries bees:needs-human; for a per-role run, the role’s session finished.
timeout The case’s timeout ran out. The running sessions were stopped.
budget The sessions cost the case’s max_cost. The running sessions were stopped.
invalid The test passed on the fixture before the run, so the case proves nothing. No session ran. Whole-factory cases only.
interrupted You stopped bees eval.
error The case could not be set up, or the scheduler failed. The error follows the table.

A COST ending in +? counts sessions that reported no cost.

The shipped cases

The busybees repository ships these cases. The todo-* cases share one fixture, evals/fixtures/todo: a small Go module with a to-do list library and a todo command. Their test is go test ./..., so they need Go on the machine that runs bees eval.

Case Kind The issue asks The grading test checks
hello bug greet.sh prints Hello, Ada!, not Hello Ada test.sh compares the output of sh greet.sh Ada
todo-done-number bug todo done <n> marks item n, counting from 1, not the one after it List.Complete marks the numbered item and refuses 0, negative and out-of-range numbers; todo done 1 marks the first line
todo-priority-order bug List.ByPriority puts the items without a priority last, not before (A) ByPriority orders (A) to (Z), then the items without one, each group in list order, and leaves the list as it was
todo-overdue feature a List.Overdue(today) method: the pending items due before today’s date Overdue compares dates, not times, and leaves out done items, items due today or later and items with no due date
todo-count feature, seeded for triage a todo count command printing 2 pending, 1 done the line printed for a list and for a missing file, and an error for an extra argument

The todo-* grading tests are only in each case’s grade/, so the sessions do not see them.

These are the per-role cases:

Case The role is given The case checks
developer/done-number one ready issue on the shared todo fixture the outcome pr-opened, a pull request for the issue, bees:ready gone from it, and a rubric on the change and the pull request’s description
developer/ask-first a ready issue whose acceptance criteria contradict each other about the numbering the outcome question, bees:blocked on the issue, mail to the project manager, no issue created, and rubrics on the question and on having built nothing
developer/out-of-scope-bug a ready issue in a file that holds a second, unrelated fault the outcome, a pull request for the issue, one issue created, and rubrics on the fault being filed rather than fixed and on the change itself
developer/review-feedback an open pull request on the shared todo fixture and the reviewer’s mail raising three findings the outcome pr-updated, and rubrics on each finding and on the rewritten pull request description
reviewer/approve-a-good-pr a pull request that meets its issue’s acceptance criteria and is tested the outcome approved, bees:approved on the issue, and rubrics on approving without demanding a change and on saying what was checked
reviewer/catch-a-real-bug a pull request whose new method reports an item due today as overdue, with green tests the outcome changes-requested, mail to the developer, bees:needs-human on the issue, and rubrics on naming the fault and on blocking on nothing else
reviewer/no-nitpicking a correct pull request written in a style the fixture does not use the outcome approved, bees:approved on the issue, and rubrics on not sending the style differences back and on judging the change against the issue
reviewer/final-round-honesty the last round on a pull request that still panics, and a developer asking for it to be let through the outcome changes-requested, mail to the developer, bees:needs-human on the issue, and rubrics on refusing to approve it and on what the feedback says
project_manager/thin-issue a triage queue of two, a blocked issue and the developer’s question about it the outcome, bees:triage → bees:ready on the thin issue, the invalid one closed, mail to the developer, and rubrics on the refined issue and the answer
project_manager/split-in-two one triage item that is two pull requests the outcome, two issues created, the original closed, and rubrics on the split and on not rewriting the original instead
project_manager/escalate a triage item whose first deliverable is a product decision, and one to refine the outcome, bees:blocked on it, mail to the product manager, no issue created, the other item ready, and rubrics on the question and on inventing no criteria
qa/broken-greeting a default branch whose test.sh fails, and one bug already filed the outcome, one issue created, the report mailed to the product manager, and a rubric on the bug report
qa/clean-pass a default branch that does what its README says, and a bug report that does not reproduce the outcome, no issue created, the report mailed to the product manager, and a rubric on reporting a clean pass rather than inventing a defect
qa/already-filed a default branch with one defect, already reported as #1 the outcome, no issue created, the report mailed to the product manager, and a rubric on commenting on #1 after file_bug refuses the duplicate
qa/wrong-run-command a default branch whose README documents a command that is not there the outcome, one issue created, the report mailed to the product manager, and a rubric on filing the documentation defect and saying so in the outcome
product_manager/feedback-idea one bees:feedback issue holding a vague idea the outcome, one issue created, the feedback issue closed, and a rubric on the feature issue and the reply
product_manager/break-down an approved feature with three outcomes in it the outcome, three issues created, bees:question not added, and rubrics on the split and on leaving milestones alone
product_manager/ask-a-person a feature whose scope turns on a decision only a person can make the outcome, bees:question on the feature, no issue created, and a rubric on the question
product_manager/answer-a-question the project manager’s question by mail, and nothing on GitHub to do the outcome, mail back about the issue, no issue created, and a rubric on the answer and on leaving GitHub alone

The reviewer cases score the whole review, not the judge session alone. bees eval reviewer runs the review loop’s review stage, so the brief session and the angle sessions run first, and the session that posts the findings and reports the verdict is the last of them. All of them run on the profiles bees.toml or --profile select for the reviewer (profile, brief_profile, angle_profiles, judge_profile). A fault the angles miss is a fault the case scores against the reviewer.

Which angles run follows the size the brief gives the change, not the size label on the issue: a change the brief calls xs or s is read from quick_general and docs; m and l swap the quick general pass for the thorough one and add test_coverage and acceptance_criteria; and side_effects runs at xl alone. The criteria of the issues the pull request closes reach every angle whatever the size, because the brief carries them. A project’s context.toml can turn angles and context sources off; the fixtures ship none, so every built-in context source is gathered.

Which profile the sessions run on

--profile <name> runs every role on that profile. bees looks it up in the [profiles] of bees.toml first, then in ~/.config/bees/config.toml, which defaults.toml fills below itself — a profile defined only there is found, and the run names the file it came from.

Without --profile, the eval uses the profile selection bees.toml makes (profiles): [global] and each role’s profile and profile_by_size, and the reviewer’s brief_profile, judge_profile and angle_profiles. With no bees.toml, it uses the provider and model of ~/.config/bees/config.toml. With no config.toml either, it takes the selection defaults.toml’s [global] and [roles] tables make instead, the same keys a project’s bees.toml reads them as, and the run names that file. Without a defaults.toml, the built-in profile.

The eval takes nothing else from bees.toml: no prompts, skills, MCP servers, limits or timings, and every role is configured — a per-role run scopes the scheduler to one role rather than configuring the rest away. Two runs differ only by their profiles. A fixture can carry its own bees/prompts/, as any project can. A profile whose sandbox is container or sbx is refused: those sessions cannot reach the fake GitHub described below.

What a whole-factory run does

For each case, in its own directory under <state_dir>/evals/<timestamp>/<case>/:

  1. It builds the fixture: a bare origin.git with one commit on main, a branch for each pull request the case seeds, and a clone of it in project/.
  2. It runs the case’s test on the fixture, in before/. A test that passes ends the case as invalid.
  3. It writes project/bees.toml and seeds an in-memory GitHub with the case’s issues and pull requests, each carrying the bees label, and the mailbox with its mail.
  4. It runs scheduler passes. Each pass waits for the work it started, which takes a developer’s issue up to its approval. Between passes, it merges every pull request labelled bees:approved into its base branch on the origin, as a person would, and closes the issues its body names with a closing keyword (Closes #1).
  5. It stops when every seeded issue is closed or carries bees:needs-human, or at the case’s timeout or budget.
  6. It grades the result.

The directory keeps everything: state/ with the sessions’ transcripts and bees.log, the fixture, and the test output in before.log and after.log.

A per-role run does steps 1, 3 and 6, and in place of 4 and 5 runs one session for the role (see Per-role cases below).

The fake GitHub

The scheduler talks to the in-memory GitHub directly. A session, and the bees MCP server it starts, runs gh, and the eval puts a gh of its own first on their PATH (bin/gh in the case directory). It forwards each call to the eval, which answers it from the same in-memory GitHub. Nothing a session runs as gh reaches github.com.

That GitHub answers the calls the factory makes: listing, viewing, creating and editing issues and pull requests, labels, comments, reviews, sub-issues and pr diff. It ignores --jq and --search, reports no checks on a pull request, and fails any call it does not know with an error the session sees. A session that runs a gh by its full path, not the one on its PATH, is not stopped from reaching github.com.

Grading

A whole-factory case passes when every check passes:

Check Passes when
the test fails on the fixture The test fails before the run.
#N closed Seeded issue N is closed.
#N has a pull request A pull request’s head is the issue’s branch (bees/issue-N), or its body closes the issue.
the test passes The test passes on main after the run.

The test runs in a fresh clone of main, with the case’s grade/ files copied over it, as sh -c "<test>". A test gets 10 minutes.

Writing a case

A whole-factory case is a directory under evals/ with a case.toml, and a per-role one a directory under evals/<role>/. Both hold the same fixture:

evals/hello/
  case.toml       the issues to seed, the test and the limits
  repo/           the fixture's files, committed as they are
  grade/          optional: files copied over the checkout before the test runs
  pr/<branch>/    optional: the working tree of a seeded pull request's branch

Instead of repo/, a case can have a setup.sh, which bees runs with sh in an empty clone and then commits. A case has one or the other.

Cases can share a fixture. Put it under evals/fixtures/<name>/ and have each case’s setup.sh copy it in:

cp -R "$(dirname "$0")/../fixtures/todo/." .

evals/fixtures/ is not a case.

Put the files that grade the case in grade/, where a session that edits the fixture’s tests does not change what grades it. A copy in repo/ as well lets the developer run them, as in hello, where the test is a shell script: a Go test that fails on the fixture cannot be copied into repo/. Left out of repo/, as in the todo-* cases, they stay hidden the way SWE-bench hides its tests, and the issue has to state the behaviour they check. Give them names the developer is unlikely to pick (eval_count_test.go, TestEvalCount): a file copied over the checkout replaces one with the same path, and a Go test declared twice does not build.

A fixture written in Go has to build, be gofmt-clean and pass its own go test ./..., and so does the tree of every branch the case seeds. The defect the case is about belongs in the hidden tests under grade/, or in what the seeded branch’s diff shows: a fixture that ships a failing test grades whoever runs go test rather than the role the case is about. The suite assembles each shipped case and checks this.

evals/hello/case.toml:

description = "greet.sh leaves out the comma and the exclamation mark"
test = "sh test.sh"
timeout = "30m"
max_cost = 5

[[issues]]
number = 1
title = "greet.sh prints the greeting without punctuation"
labels = ["bees:ready", "bees:size/xs"]
body = """
`sh greet.sh Ada` prints `Hello Ada`. It should print `Hello, Ada!`.
"""
Key Meaning
description What the case is about.
test Required. The command that grades the case, run with sh -c in a checkout of main. It must fail on the fixture.
timeout How long the factory runs. Default 1h.
max_cost The budget in USD. 0 is no limit. Default 10.
[[issues]] The issues to seed, at least one: number, title, body, labels, author (default human) and [[issues.comments]] with author and body.
[[mail]] Messages in a role’s mailbox when the run starts: from (default human), to (a role), subject, body, and issue, a seeded issue it is about.
[[pull_requests]] The pull requests to seed, each on a branch of its own in the fixture. See Seeding a pull request.

bees adds the bees label to every seeded issue. Give a work item bees:ready and a size to send it straight to a developer, or bees:triage to have the project manager refine it first.

Pick issues the way SWE-bench Lite does: self-contained, touching a few files, with no links to anything outside the fixture, and graded by a test that fails before the change and passes after it.

A directory under evals/ named after a role (evals/developer/) is not a whole-factory case: it holds that role’s cases, which only bees eval <role> takes.

Per-role cases

bees eval <role> runs one role against the cases under evals/<role>/, the way bees exec <role> runs one session: the scheduler is scoped to that role, so nothing else runs, and the seeded GitHub state and mailbox stand in for the rest of the factory. The run ends when the session does. Nothing is merged and no second pass runs.

The rest of the factory is still configured, and that is deliberate: the scheduler routes a new issue by the roles the configuration has, so a run that configured four roles away would label an unlabelled issue somewhere the real factory never would, and the case would be measuring the role against a workflow that does not exist.

A per-role case has no test. It is graded by what it declares under [expect], and it passes when every one of those checks passes.

A reviewer case runs with max_review_rounds = 1. The review loop answers “changes requested” by starting a developer session, which a per-role run must not do: the scheduler’s role scope gates dispatch, not the stage a worker moves to next. With one round the worker escalates the issue instead, so the run ends on the reviewer’s verdict — bees:approved on the issue after an approval, bees:needs-human after changes requested. The session is told it is the final review round, which every reviewer case therefore is.

evals/project_manager/thin-issue/case.toml, cut down:

description = "a thin bug report to refine and a developer's question to answer"

[[issues]]
number = 1
title = "todo done marks the wrong item"
labels = ["bees:triage"]
body = "I typed `todo done 1` and it crossed off the second thing on my list."

[[issues]]
number = 2
title = "todo list should show due dates"
labels = ["bees:blocked", "bees:size/s"]
body = "`todo list` prints the raw line."

[[mail]]
from = "developer"
to = "project_manager"
subject = "Question about #2"
body = "What should `todo list` print for an item with no due date?"
issue = 2

[expect]
outcome = "done"

[[expect.labels]]
issue = 1
has = ["bees:ready"]
missing = ["bees:triage"]

[[expect.mail]]
to = "developer"
issue = 2

The keys [[issues]], [[mail]], description, timeout and max_cost mean what they mean for a whole-factory case. These are the rest:

Key Meaning
issue, pr What the session is about, as bees exec --issue/--pr. A developer or reviewer case needs one; the three singleton roles take neither, since their session is about the whole repository. pr names a seeded pull request.
[[pull_requests]] The pull requests to seed: what a reviewer case reviews, and what a developer case has review feedback on. See Seeding a pull request.
[expect] How the case is graded. At least one check, or the case grades nothing.

The checks a case can declare

Check Passes when
outcome = "done" The role’s session reported that status with done. It has to be one the role may report. failed is declarable for a developer or reviewer case, where the factory hands the issue to a person and the run still ends as done; for the other three roles a failed session ends the run as error, which fails the case whatever it declared.
issues_created = 1 That many issues exist at the end that the case did not seed.
issues_closed = [3] Each of those seeded issues is closed.
pull_requests = [1] Each of those issues has a pull request: one on its branch, or one whose body closes it.
[[expect.labels]] The issue carries every label under has and none under missing. One check per label.
[[expect.mail]] The role sent the role under to a message, about the issue under issue when it names one.
[[expect.graded]] A grader session scored the rubric at or above its pass score.

Rubrics and grader sessions

A check that needs judgement is a rubric, scored by a session of its own, following Inspect AI’s model-graded scorer pattern. The grader is given the rubric, the transcript of the role’s session and the state the run left behind — the issues as they stand, the comments posted on them, the pull requests opened and the mail the role sent — and answers with a score between 0 and 1 and the reasons for it. Both go in the report; the score also fills the table’s SCORE column.

[[expect.graded]]
name = "#1 is detailed enough for a developer to build"
pass = 0.7
rubric = """
#1 as it now stands names where the bug is and says what the fixed behaviour
is. It carries acceptance criteria a test could be written from. Judge the
issue's body as it is at the end, not how it was written.
"""

name is what the report calls the check, and pass the score it passes at: 0.7 when the rubric leaves it out, and pass = 0 to record the score in the report without ever failing the case. Write the rubric as what the grader should see, and say what not to judge: a rubric that leaves the bar to the grader’s taste scores differently from run to run, which is the one thing an eval must not do.

The grader is not the agent --profile selects. It is the read-only session agent the global review configuration supplies — config.toml, with defaults.toml below it, so with no config.toml it runs as the agent and model the defaults file’s roles.reviewer.profile selects, else global.profile’s. It is the one bees review runs, and it stays the same whatever profile the roles run on. Two runs of a case are compared by their scores, so what does the scoring has to hold still while what is being scored changes.

Seeding a pull request

[[pull_requests]] gives a case a pull request a session sees the way it would on GitHub: a branch in the fixture with the change on it, and the pull request itself with its body, comments and reviews.

The branch’s working tree is a directory in the case, pr/<head> by default:

evals/developer/address-feedback/
  case.toml
  repo/                        the fixture, committed on main
  pr/bees/issue-1/             the working tree of the head branch
pr = 2                         # what the session is about

[[pull_requests]]
number = 2
title = "todo list: show due dates"
body = "Closes #1"
head = "bees/issue-1"
base = "main"

[[pull_requests.comments]]
author = "human"
body = "The dates are still raw."

[[pull_requests.reviews]]
author = "kpenfound"
state = "CHANGES_REQUESTED"
body = "Print `no due date` for an item with none."
Key Meaning
number Required. GitHub numbers issues and pull requests together, so it cannot be a seeded issue’s number.
title, body Required title, and the body. A body with a closing keyword (Closes #1) is what ties the pull request to its issue.
head Required. The branch the change is on, branched off base.
base The branch it is against. Default main.
files The directory holding the head branch’s working tree, relative to the case directory. Default pr/<head>. It has to be there, whether it is named or defaulted.
labels Labels beside the bees label, which bees adds.
author Who opened it. Default human.
[[pull_requests.comments]] Comments on the conversation: author (default human) and body.
[[pull_requests.reviews]] Reviews: author (default human), body and state, one of APPROVED, CHANGES_REQUESTED and COMMENTED (default COMMENTED).

Once the fixture is committed on main, bees branches head off base, copies the tree over it and pushes it. A file the tree leaves out is unchanged from base, so the directory carries only what the pull request touches. A pull request with no tree directory is a load error: a pull request with no change is nothing for a session to read.

The tree’s own tests pass. A reviewer reads the branch’s diff for the defect, so a test that already fails on the branch hands the answer to anything that runs the suite instead.

A developer case gives its pull request the branch of the issue it names (bees/issue-<n>): that is the branch the developer’s session works on, and the pull request it is handed to update. A reviewer case does the same: its session reviews the pull request on the branch of the issue it names.

The review reads that branch against its base, so the fault a reviewer case is about has to be in the change itself, and the issue’s acceptance criteria have to settle it: a comparison that includes a boundary the issue excludes, a bounds check that lets a number through. Leave the branch’s own tests passing, or the case measures whoever runs go test rather than the reviewer.