# BenchFlow BenchFlow is a frontier environment lab for AI agents. We ship SkillsBench, ClawsBench, PostTrain, and the BenchFlow runtime. Site: https://benchflow.ai Source: https://github.com/benchflow-ai ## Documentation index ### BenchFlow - /docs/benchflow/authentication - /docs/benchflow/concepts - /docs/benchflow/continue-runs - /docs/benchflow/environment-plane - /docs/benchflow/external-agents - /docs/benchflow/getting-started - /docs/benchflow/llm-judge - /docs/benchflow/progressive-disclosure - /docs/benchflow/reference/cli - /docs/benchflow/reference/python-api - /docs/benchflow/rubric-review - /docs/benchflow/running-any-benchmark - /docs/benchflow/running-evaluations - /docs/benchflow/sandbox-hardening - /docs/benchflow/sandboxes - /docs/benchflow/skill-eval - /docs/benchflow/start-in-5-minutes - /docs/benchflow/task-authoring - /docs/benchflow/task-authoring-task-md - /docs/benchflow/task-standard - /docs/benchflow/traj-upload - /docs/benchflow/use-cases ### SkillsBench - /docs/skillsbench/contributing - /docs/skillsbench/getting-started --- ## /docs/benchflow/authentication BenchFlow installs and runs an agent inside each sandbox. The agent therefore needs the same model credentials it would use when run directly. Choose one authentication method for the agent/model pair you plan to evaluate. ## Subscription login on your machine For an interactive local run, this is usually the shortest path. Sign in once with the agent's host CLI: | Agent | Host login | Credential BenchFlow detects | |---|---|---| | `codex` | `codex login` | `~/.codex/auth.json` | | `claude` | `claude auth login` | `~/.claude/.credentials.json` | | `gemini` | `gemini` interactive flow | `~/.gemini/oauth_creds.json` | BenchFlow copies the relevant credential into the sandbox for that run. You do not need a Daytona account or a model API key to use a supported subscription login with the local Docker sandbox. ## API keys Export the key for your provider before invoking `bench`: ```bash export OPENAI_API_KEY='...' export ANTHROPIC_API_KEY='...' export GEMINI_API_KEY='...' ``` Other common variables include: ```bash export CODEX_API_KEY='...' # Codex alias for OPENAI_API_KEY export LLM_API_KEY='...' # OpenHands / LiteLLM-compatible providers export AZURE_API_KEY='...' export AZURE_API_ENDPOINT='https://.openai.azure.com/' ``` Never put a key directly in a committed run config or command example. Shell variables must be exported to reach BenchFlow. For a local `.env` file: ```bash set -a source .env set +a bench eval run ... ``` BenchFlow also reads well-known credential keys from a `.env` in the current directory, but exporting works regardless of where the command is run. ## CI and headless machines Claude supports a long-lived OAuth token: ```bash claude setup-token export CLAUDE_CODE_OAUTH_TOKEN='' ``` Codex can use `CODEX_ACCESS_TOKEN` when a host or orchestrator provides one. Gemini currently uses either the saved host login or an API key. Store CI credentials in the CI system's secret manager, not in the repository. ## Provider-prefixed models Models routed through a user-supplied endpoint generally use `_API_KEY` plus `_BASE_URL`. For example: ```bash export DEEPSEEK_API_KEY='...' export DEEPSEEK_BASE_URL='https://api.deepseek.com' ``` Providers with a fixed endpoint may need only the API key. Azure Foundry uses `AZURE_API_KEY` and `AZURE_API_ENDPOINT` with model names such as `azure-foundry-openai/gpt-5.5`. ## Credential precedence Provider-specific credentials selected by the model prefix take precedence over the agent's native credentials. Claude's native order is: cloud provider credentials, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_API_KEY`, `apiKeyHelper`, `CLAUDE_CODE_OAUTH_TOKEN`, then the saved host subscription login. If you intended to use a subscription but an API key is set, unset the key before running: ```bash unset OPENAI_API_KEY CODEX_API_KEY # or unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN ``` To see the available agent names and their native auth requirements: ```bash bench agent list ``` --- ## /docs/benchflow/concepts The mental model for benchflow. Read once, then refer back from the how-tos. --- ## The five primitives | Primitive | What it is | |-----------|------------| | **Task** | A directory on disk: a `task.md` document (YAML frontmatter + prompt body) plus `environment/Dockerfile` for the sandbox, `verifier/` checks, and optional `oracle/` — or the legacy split layout (`task.toml` + `instruction.md` + `tests/` + `solution/`). Authored once, evaluated many times. | | **Agent** | A registered ACP-speaking program (Claude Code, Gemini CLI, OpenCode, etc.). Identified by name (`"gemini"`, `"opencode"`) plus an optional model ID. Use the `acpx/` prefix (e.g. `acpx/gemini`) to route through [ACPX](https://acpx.sh/), a headless ACP client with persistent sessions and crash recovery. | | **Environment** | The sandbox where the agent runs and the verifier checks the result. Docker is the local default; Daytona, Modal, and AgentCore are optional remote backends. See [Sandboxes](/docs/benchflow/sandboxes). | | **Verifier** | The test runner that scores the rollout. Its entry point is a `test.sh` script (native `verifier/test.sh`, legacy `tests/test.sh`) — which typically runs `pytest` against the workspace the agent left behind. For subjective tasks, use an [LLM-as-judge](/docs/benchflow/llm-judge) verifier scored against a rubric. Outputs `rewards: {reward: float}`. See the [verifier file map](#verifier-file-map) for which file lives where in native vs legacy packages. | | **Rollout** | One agent run on one task. Holds the lifecycle (setup → start → install → execute → verify → cleanup). All higher-level primitives below are built on Rollouts. | --- ## Rollout lifecycle A `Rollout` is decomposable: each phase is a callable method, you can either run them in sequence or invoke `Rollout.run()` to execute all six in order. Multi-agent flows reuse phases (e.g. `connect` + `execute` + `disconnect` repeats per role). ``` ┌──────────────────────────────────────────────────────────────┐ │ Rollout.run() │ │ │ │ setup() resolve config, create sandbox env handle │ │ ↓ │ │ start() start container, upload task files │ │ ↓ │ │ install_agent() install agent binary, write credentials, │ │ set up sandbox user │ │ ↓ │ │ ┌─ connect_as(role) ◄─── multi-agent loops here │ │ │ execute(prompts) each role's turn │ │ └─ disconnect() │ │ ↓ │ │ verify() harden sandbox, run pytest, score │ │ ↓ │ │ cleanup() kill agent procs, stop container │ └──────────────────────────────────────────────────────────────┘ ``` Each phase has a name, a clear contract, and is independently testable. `Rollout.run()` is the convenience that calls them in order. ```python import benchflow as bf from benchflow import RolloutConfig, Scene from pathlib import Path config = RolloutConfig( task_path=Path("tasks/edit-pdf"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-pro-preview")], environment="docker", ) result = await bf.run(config) # full lifecycle print(result.rewards) # {'reward': 1.0} ``` --- ## Scenes, Roles, Turns A **Scene** is authoring sugar for Step metadata. Inside a Scene: - **Roles** are the agents that participate (one or more). - **Turns** are the prompt sequence — which Role acts when, and what they're told. - All Roles share the same sandbox filesystem. Before rollout execution, BenchFlow desugars Scenes into explicit rollout Steps carrying role, prompt, and skill attribution. Scene has no runtime object, scheduler, message router, or lifecycle. ```python Scene( name="review-loop", roles=[ Role(name="coder", agent="opencode", model="anthropic/claude-sonnet-4-6"), Role(name="reviewer", agent="gemini", model="gemini-3.1-pro-preview"), ], turns=[ Turn(role="coder"), Turn(role="reviewer", prompt="Review the current workspace."), Turn(role="coder", prompt="Read the reviewer's feedback and revise."), ], ) ``` A Rollout may have multiple Scenes — used for staged flows like "skill generation → solve" (BYOS / Bring Your Own Skill). Same sandbox, sequential Scenes. --- ## The User abstraction (multi-round, single-agent) Sometimes you want the agent to take multiple turns guided not by another LLM but by a Python callback that watches what happened and decides what to say next. That's a **User**. A User is a `BaseUser` subclass (or `FunctionUser` wrapping a function) with two methods: - `setup(instruction, solution)` — once, before round 0 - `run(round, instruction, round_result) → str | None` — per round; return `None` to stop the loop Between rounds, BenchFlow executes `soft_verify()` (verifier without the destructive parts of full hardening), gives the user the round's `RoundResult` (trajectory, rewards, verifier output, tool count), and lets the user decide round N+1's prompt. Use `BaseUser` when the loop logic is rule-based (compress instruction → show test failures as hints → stop on pass). See [`progressive-disclosure.md`](/docs/benchflow/progressive-disclosure) for the full guide. --- ## Verifier, sandbox, hardening Once the agent stops, the verifier runs. Its entry point is the task's `test.sh` script — uploaded to `/verifier` for native packages (`/tests` for legacy ones) — executed against the workspace the agent left behind. benchflow runs `test.sh` **as a script** (it `chmod +x`'s the file and executes it directly; a native `script` strategy runs `cd /verifier && `). It never hands `test.sh` to `pytest` — pytest cannot collect a shell script as a test target. Most `test.sh` scripts *invoke* pytest internally. For those invocations, benchflow applies hardening through `PYTEST_ADDOPTS` in the verifier environment — every pytest run inside `test.sh` inherits roughly: ```text PYTEST_ADDOPTS="-c /dev/null --confcutdir= --rootdir= -p no:cacheprovider" ``` where `` is `/verifier` for native packages (`/tests` for legacy), and `` is the agent workspace (`/app` for Harbor/SWE-bench conventions, `/root` for SkillsBench — injected dynamically). `-c /dev/null` blocks `pyproject.toml`/`pytest.ini` discovery and `--confcutdir` blocks `conftest.py` walk-up beyond the verifier dir. Tasks that do not use pytest (e.g. a `test.sh` that diffs files and writes `reward.txt` directly) are scored the same way — pytest is just the most common tool, not a requirement. Between agent and verifier, benchflow **hardens** the sandbox to prevent the agent from gaming the score: - Kill any lingering agent processes - Restore build-config files (setup.py, pyproject.toml, …) to their pre-agent snapshots - Delete agent-injected `conftest.py`, `sitecustomize.py`, `.pth` files - Lock the workspace to root, set restrictive PYTHONPATH/PATH for the verifier process - Run pytest with plugin auto-discovery off, only allowing plugins declared in the task config (`[verifier] pytest_plugins` in the `task.md` front-matter, or `task.toml` for split-layout tasks, or auto-discovered root-owned plugins) This catches the BenchJack and Meerkat exploit families. When a task ships a legitimate `conftest.py` (e.g. qutebrowser uses one to break a real circular import), the task opts out in its task config (`task.md` front-matter, or `task.toml` for split-layout tasks): ```toml [verifier.hardening] cleanup_conftests = false ``` See [`progressive-disclosure.md`](/docs/benchflow/progressive-disclosure#per-task-hardening-opt-outs) for the full opt-out list. ### Verifier file map Native `task.md` packages and the legacy split layout name their verifier files differently. The runtime resolves native files first and falls back to the legacy names, so a task ships **one** of each row, not both: | What it is | Native (`task.md`) package | Legacy split layout | Sandbox path | |---|---|---|---| | Verifier directory | `verifier/` | `tests/` | `/verifier` (native), `/tests` (legacy) | | Script entry point | `verifier/test.sh` | `tests/test.sh` | executed as a script (chmod +x then run) inside the verifier dir | | Strategy declaration (how it's scored) | `verifier/verifier.md` | — (legacy uses `[verifier]` in `task.toml`) | not uploaded as a runtime target; selects the strategy | | LLM-judge rubric | `verifier/rubrics/verifier.md` + `verifier/rubrics/verifier.toml` | `tests/rubric.toml` (also `rubric.json`, Harvey-LAB style) | downloaded for the judge | A plain `test.sh` is a complete verifier on its own: with no `verifier.md` strategy declared, the runtime just executes it. `verifier/verifier.md` declares *how* a task is scored (script / llm-judge / reward-kit / agent-judge / ors-episode) and is the native equivalent of the legacy `[verifier]` section in `task.toml`. The native LLM-judge rubric lives under `verifier/rubrics/` (both a human-readable `verifier.md` and a machine-readable `verifier.toml`), not in a single top-level `rubric.toml`. For the native verifier document and its strategy table see [Native task.md authoring](/docs/benchflow/task-authoring-task-md); for the legacy `[verifier.judge]` rubric path see [LLM-as-judge](/docs/benchflow/llm-judge). --- ## Multi-turn vs multi-round vs multi-scene Three different axes — easy to confuse, worth pinning down: | Axis | What changes | Example | |------|--------------|---------| | **Multi-turn** | Same Role, multiple prompts within one Scene. The ACP session persists; the agent has continuous memory. | One coder gets prompted twice: "fix the bug", then "now write a test". | | **Multi-round** | Same Role, multiple `connect → execute → disconnect` cycles. New ACP session each round; sandbox state persists; a Python `User` callback decides each round's prompt. | Progressive disclosure on SWE-bench Pro: round 0 terse spec, round 1 hints with failing tests, round 2 full spec. | | **Multi-scene** | Multiple Scenes in one Rollout. Sandbox state persists; agent process and ACP session restart between Scenes. | BYOS: Scene 1 generates a skill, Scene 2 solves the task using it. | Single-agent simple runs use none of these. Pick the axis based on what state needs to persist (memory? sandbox? both?). --- ## Trajectories and rewards Every agent action is captured as an event in the **trajectory** — tool calls, agent messages, agent thoughts. A `RolloutResult` (aliased as `RunResult`) has the full trajectory plus tool count, plus rewards from the verifier and any error. `rewards` is a dict produced by the task's verifier. Convention: `{"reward": float}` where 1.0 = pass, 0.0 = fail. Tasks may add additional metrics (e.g. `exact_match`, `partial_credit`). Trajectories are written to `///trajectory/acp_trajectory.jsonl` (the `--jobs-dir` directory, default `jobs/`). Use them for replay, debugging, or training data. --- ## Where to go next - [Getting started](/docs/benchflow/getting-started) — install, run your first eval. - [Running evaluations](/docs/benchflow/running-evaluations) — single tasks, batches, skills, and results. - [Sandboxes](/docs/benchflow/sandboxes) — start with Docker and choose a cloud backend only when needed. - [Task authoring (native task.md)](/docs/benchflow/task-authoring-task-md) — write a task as a single `task.md` document plus `environment/` and `verifier/`. - [Migrating a legacy task](/docs/benchflow/task-authoring) — convert an existing `task.toml` + `instruction.md` split package to `task.md` (the split layout is no longer a first-class authoring path). - [LLM-as-judge](/docs/benchflow/llm-judge) — use an LLM to score subjective tasks against a rubric (see the [verifier file map](#verifier-file-map) for native vs legacy rubric paths). - [Progressive disclosure](/docs/benchflow/progressive-disclosure) — the User abstraction; SWE-bench Pro case study. - [Use cases](/docs/benchflow/use-cases) — multi-agent patterns (coder/reviewer, simulated user, BYOS, stateful environments). - [CLI reference](/docs/benchflow/reference/cli), [Python API reference](/docs/benchflow/reference/python-api). - [Skill evaluation](/docs/benchflow/skill-eval) — when the artifact is a skill, not a workspace. --- ## /docs/benchflow/continue-runs `bench eval continue` resumes a previous, **unfinished** (timed-out) agent run to completion. It is a standalone tool — it does **not** modify benchflow's normal `eval`/run path — and currently targets the **`openhands`** agent. The goal is a *transparent* resume: the continued run behaves as if the original timeout had simply been larger. The agent keeps its exact context and environment and continues its own loop with **no injected prompt**. > The command lives under the `eval` group (`bench eval continue`). The original > top-level `bench continue` still works as a hidden, deprecated alias. ## The problem it solves A finished run keeps nothing of the container — cleanup tears the sandbox down. What survives on disk is the run folder: `config.json`, `result.json`, `prompts.json`, and `trajectory/llm_trajectory.jsonl`. So a historical timeout has only its *trajectory* + the *task*; there is no saved container to restore. `bench eval continue` reconstructs the missing state from the trajectory. ## How it works — record-replay The recorded `llm_trajectory.jsonl` is the exact sequence of LLM request/response pairs from the original run. `bench eval continue`: 1. **Loads** the original run folder and the recorded exchanges. 2. **Boots a fresh, pristine sandbox** from the same base image. 3. Stands up a **replay proxy** that OpenHands talks to via `LLM_BASE_URL`. For the first *N* requests it returns the recorded responses **in order**, so the agent re-executes its own past decisions *for real* — rebuilding the byte-exact workspace and its exact internal conversation/event state. 4. When the recorded responses run out (the timeout cut-point), the proxy flips to the **live model** and the agent continues — no new prompt. 5. **Re-verifies** with the task verifier and writes a new HF-compatible folder, with a stitched `llm_trajectory.jsonl` (recorded prefix + live suffix) and `continued_from` provenance — a drop-in replacement for the timed-out entry. Because the agent rebuilds its own state by re-doing its own steps, no reverse-engineering of OpenHands internals is needed, and the result is a single continuous run rather than a fresh agent on a warm filesystem. ## Usage ```bash bench eval continue path/to/original/run-folder \ --tasks-dir path/to/tasks # where the task source (verifier) lives ``` The uploaded run folder does **not** ship the task's verifier, so point `--tasks-dir` at the directory containing the task (matched by name). If the `task_path` recorded in `config.json` still exists on disk, `--tasks-dir` is optional. ### Options | Flag | Default | Meaning | | --- | --- | --- | | `--tasks-dir DIR` | recorded `task_path` | Task source (instruction + verifier). | | `--model MODEL` | original run's model | Override the **live-continuation** model. | | `--timeout SEC` | original run's timeout | Wall-clock budget for the continuation. | | `--output DIR` | `/continued` | Output jobs dir for the new run. | | `--require-timeout` | off | Refuse runs whose recorded status isn't a timeout. | | `--strict-divergence` | off | Abort if replay leaves the original rails. | | `--replay-only` | off | Rebuild via replay and stop at the cut-point (no live model needed). | ### Models and credentials - The **live-continuation model** defaults to the original run's model so the continuation is a faithful continuation of the same brain. Tests use `--model gemini-3.1-flash-lite-preview` for a cheap path. - The **replay phase needs no API key** — responses are served from the recording. Only the **live continuation** calls the real provider, so the host needs that provider's credentials (e.g. `GEMINI_API_KEY`) in its environment. `--replay-only` skips the live leg entirely. ## Limitations and caveats - **`openhands` only** for now (the proxy seam relies on `LLM_BASE_URL`). - **Replay fidelity is best-effort.** Replay re-runs the original shell commands for real; if a command's output diverges from the original (network, timestamps, nondeterminism), the agent may see a different observation than recorded. A message-count check warns on divergence (`--strict-divergence` aborts instead). - **"Identical output" means a faithful continuation**, not a bit-identical result — the model samples, and no "original full run" exists past the timeout. The bar is: the stitched trajectory reads as one continuous run, as if the timeout had been larger. - Re-running the episode's commands costs wall-clock time (model latency is skipped, since recorded responses are served instantly). --- ## /docs/benchflow/environment-plane The **Environment plane** is the stateful world the agent acts in — Han's "S" in `E = {T, H, V, S, C}`. It is one of BenchFlow's four swappable planes (Sandbox, Agent, Environment, Reward). See [`architecture.md`](https://github.com/benchflow-ai/benchflow/blob/main/docs/architecture.md), "The Environment plane & the manifest". A benchmark author never subclasses the framework. They write one file — an **`environment.toml` manifest** — and the default adapter (`ManifestEnvironment`) runs it on any Sandbox provider. The manifest is the entire integration surface. ## The manifest schema The manifest's keys live under an `[environment]` table. ### `[environment]` | Field | Type | Default | Meaning | |---|---|---|---| | `name` | str | — (required) | Environment / benchmark name. | | `image` | str | `None` | A ready-to-run image. Set this **or** `base_image`. | | `base_image` | str | `None` | Image that per-task images build `FROM` (smolclaws-style). | | `ports` | list[int] | `[]` | Ports the environment exposes (in addition to service ports). | | `owns_lifecycle` | bool | `true` | `true` — the image entrypoint starts the services. `false` — the framework starts the `[[services]]`. | | `keep_alive` | bool | `true` | Keep the environment up for the whole rollout. | | `isolation` | `"per_task"` \| `"persistent"` | `"per_task"` | `per_task` — a fresh environment per episode. `persistent` — cross-episode state. | Exactly one of `image` / `base_image` must be set. When `owns_lifecycle` is `false` the manifest must declare `[[environment.services]]`; when it is `true` it must not. ### `[environment.task_selection]` | Field | Type | Default | Meaning | |---|---|---|---| | `mechanism` | `"image"` \| `"env_var"` | `"env_var"` | `image` — the task's seed data is baked into a per-task image. `env_var` — one image, the task id passed at runtime. | | `key` | str | `"BENCHFLOW_TASK_ID"` | Env var name (when `mechanism = "env_var"`). | | `inject_into` | `"entrypoint"` \| `"exec"` | `"entrypoint"` | `entrypoint` reaches PID 1; `exec` does not. | ### `[[environment.services]]` An array — one table per service the framework starts (only when `owns_lifecycle = false`). It is the declarative replacement for the hard-coded `SERVICES` dict in `benchflow/sandbox/services.py`. | Field | Type | Default | Meaning | |---|---|---|---| | `name` | str | — (required) | Service name. | | `command` | str | — (required) | Full start command. | | `port` | int | — (required) | Port the service listens on. | | `health_path` | str | `"/health"` | HTTP path probed for readiness. | ### `[environment.readiness]` | Field | Type | Default | Meaning | |---|---|---|---| | `http` | list[str] | `[]` | Explicit HTTP probes. When empty, derived from the services. | | `tcp` | list[int] | `[]` | TCP-connect probes. | | `timeout_sec` | int | `120` | How long to wait for readiness before failing the rollout. | ### `[environment.forward_env]` | Field | Type | Default | Meaning | |---|---|---|---| | `keys` | list[str] | `[]` | Host env vars forwarded into the environment container. | ### `[environment.state]` Present only for an environment that supports **roll-back** — `snapshot` / `restore`. Absent this table, the environment is treated as stateless and `snapshot`/`restore` raise `RuntimeError`. | Field | Type | Default | Meaning | |---|---|---|---| | `kind` | `"sqlite"` | `"sqlite"` | State backend. Only SQLite is supported today. | | `paths` | list[str] | `[]` | The database files to capture and restore (one snapshot covers all of them). | ## Worked example — ClawsBench `benchmarks/clawsbench/environment.toml` — the internal-dogfood stateful multi-service benchmark (mock Gmail / Slack / Calendar / Docs / Drive): ```toml [environment] name = "clawsbench" base_image = "kywch/smolclaws-base:latest" owns_lifecycle = false isolation = "per_task" [environment.task_selection] mechanism = "image" [environment.readiness] timeout_sec = 60 [environment.forward_env] keys = ["ANTHROPIC_API_KEY"] [[environment.services]] name = "gmail" command = "claw-gmail --db /data/gmail.db serve --host 0.0.0.0 --port 9001 --no-mcp" port = 9001 # ... slack (9002), gcal (9003), gdoc (9004), gdrive (9005) ``` One manifest serves the whole benchmark even though smolclaws builds a per-task image carrying only a subset of the services: `ManifestEnvironment` probes each service's entry point with `--help` and starts only the services whose package is actually installed in this per-task image. ## Worked example — chi-bench `benchmarks/chi-bench/environment.toml` — the *other* topology, and the external proof that a heavy environment onboards untouched. chi-bench is a ~25k-LOC healthcare simulator that ships **one** ready-to-run image whose entrypoint starts its own services, so the manifest declares no `[[services]]`: ```toml [environment] name = "chi-bench" image = "chi-bench:latest" owns_lifecycle = true isolation = "per_task" ports = [8020, 8023, 8100, 8200] [environment.task_selection] mechanism = "env_var" key = "CHI_BENCH_TASK_ID" inject_into = "entrypoint" [environment.readiness] http = ["http://localhost:8023/health"] timeout_sec = 120 [environment.forward_env] keys = ["ANTHROPIC_API_KEY"] ``` This ~25-line manifest is the *entire* framework-integration surface: chi-bench's image, Dockerfile, and entrypoint are unmodified, and the ~920 LOC of Harbor coupling it previously carried collapses into the manifest. ClawsBench (`base_image` + framework-started `[[services]]`) and chi-bench (`image` + `owns_lifecycle = true`) are the two topologies behind one contract. See [`benchmarks/chi-bench/README.md`](https://github.com/benchflow-ai/benchflow/blob/main/benchmarks/chi-bench/README.md) for the field-by-field mapping. ## How it runs `ManifestEnvironment` runs the **in-sandbox topology** (the architecture's core): the services run inside the rollout's own sandbox, so the agent reaches them on `localhost`. During a rollout: 1. `Rollout.start()` provisions the environment — starts the declared services inside the sandbox. 2. It gates on `readiness()` — the agent never runs before the environment is healthy. 3. `Rollout.cleanup()` tears the environment down. Run one task or a task directory against an environment manifest with `bench eval run --tasks-dir ...`. `--environment-manifest` applies the Environment-plane manifest to every rollout in the Job pipeline. ```bash # one task bench eval run --tasks-dir benchmarks/clawsbench/tasks/ \ --environment-manifest benchmarks/clawsbench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 # task directory bench eval run --tasks-dir benchmarks/clawsbench/tasks \ --environment-manifest benchmarks/clawsbench/environment.toml \ --agent claude-agent-acp --model claude-haiku-4-5 ``` YAML configs may declare the same seam with ``environment_manifest: `` at the top level so the batch run is reproducible from disk. A task can also pin its own manifest in `task.md` frontmatter (`benchflow.environment.manifest: `, resolved relative to the task directory). An explicit per-run binding always wins: `--state` beats `--environment-manifest`, and either flag beats the frontmatter pin, which applies only when no flag is given. `--environment-manifest` is distinct from `--sandbox`: the sandbox is *where* it runs (the Sandbox plane); the environment manifest is *the world* (the Environment plane). ## Registry (`name@version`) `--environment-manifest` (and `--state`) also accept a **registry spec** instead of a file path: `name@version`, or a bare `name`, resolved by [`_utils/env_registry.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/_utils/env_registry.py). The registry is just a directory of manifest files: ``` /@.toml # a pinned environment version /.toml # optional default for a bare ``` `.yaml` / `.yml` work too. A bare `name` resolves to `.toml` when present, else the highest-sorting pinned version. Every resolution is content-addressed (`env_hash = sha256(manifest bytes)`) so a run records exactly which environment it bound. Which directory is the registry: 1. **`$BENCHFLOW_ENV_REGISTRY`**, when set — any local directory of `name@version.toml` files. It wins entirely: names it does not contain do not fall back to the built-in registry. 2. **The built-in registry** otherwise — the pinned manifests shipped inside the `benchflow` wheel ([`src/benchflow/environment/_registry/`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/environment/_registry)): [`env0@prod.toml`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/environment/_registry/env0@prod.toml) and [`env0@outage.toml`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/environment/_registry/env0@outage.toml). A bare `pip install benchflow` resolves these with no checkout and no env vars. ```bash # works on a bare pip install — no $BENCHFLOW_ENV_REGISTRY needed bench eval run --tasks-dir --environment-manifest env0@prod ... # override with your own registry directory export BENCHFLOW_ENV_REGISTRY=$PWD/env-registry bench eval run --tasks-dir --environment-manifest myenv@v1 ... ``` ## Exporting for training A scored rollout's trajectory exports to the Verifiers / ORS dataset format that prime-rl ingests — `benchflow.trajectories.export`: ```python from benchflow.trajectories.export import ( trajectory_to_verifiers_record, export_trajectories_to_jsonl, ) record = trajectory_to_verifiers_record( task_id="clawsbench/archive-alice", messages=trajectory_messages, verify_result=verify_result, model="claude-haiku-4-5", environment="clawsbench", ) export_trajectories_to_jsonl([record], "dataset.jsonl") ``` Each line is one record: `prompt`, `completion`, `reward`, `metrics`, `is_completed`, `is_truncated`, `example_id`, `info` — the shape pinned against the Verifiers `RolloutOutput` type. ## Roll-back — `snapshot` / `restore` `snapshot` / `restore` are **real**. For an environment that declares an `[environment.state]` table, `snapshot()` copies each declared SQLite file with `sqlite3 .backup` (a consistent online backup) into a per-snapshot directory inside the sandbox, and `restore(snap)` copies the captured files back over the live paths. This is the substrate `Rollout.branch()` runs on: a branch quiesces the agent and services, restores a snapshot, and explores an alternative continuation. An environment with no `[environment.state]` table is stateless — `snapshot`/`restore` raise `RuntimeError`. ## Reset — `reset` `reset` returns the environment to the per-task baseline so it can be reused for a fresh episode without tearing down the sandbox (distinct from `restore`, which rolls back to an arbitrary snapshot). For an environment that declares an `[environment.state]` table, `provision` captures a baseline; `reset` then stops the framework-started services, restores the baseline, and restarts the services. For an `owns_lifecycle = true` manifest the framework cannot restart entrypoint-owned services; `reset` is then a no-op (and the host must recycle the container for a hard reset). ## Not yet implemented `ManifestEnvironment` does not exercise: - **Sidecar / shared-fleet topology** — host-exposed ports, `AccountBroker`. --- ## /docs/benchflow/external-agents BenchFlow's built-in registry covers a handful of agents (`bench agent list`). Everything else — goose, qwen-code, prime-agent, the omnigent harnesses, … — lives in the public **[benchflow-ai/agents](https://github.com/benchflow-ai/agents)** repo and loads into BenchFlow through one of four paths. For most users the first one is all there is to know. ## 1. Zero-config remote autoload (the default) Using an agent name BenchFlow doesn't recognize triggers a one-shot fetch of the declarative manifests from `benchflow-ai/agents@main`. There is no separate agent package to install for manifest-based agents: ```bash # Install the normal local CLI; no cloud-sandbox extra is needed. uv tool install --python 3.12 --upgrade benchflow export DEEPSEEK_API_KEY=sk-... # credentials for your --model provider # Get a task (sparse checkout of one SkillsBench task — same recipe as # getting-started; any task.md/Harbor-format task directory works): git clone --depth 1 --filter=blob:none --sparse \ https://github.com/benchflow-ai/skillsbench cd skillsbench && git sparse-checkout set tasks/edit-pdf bench eval run --tasks-dir tasks/edit-pdf --agent prime-agent \ --model deepseek/deepseek-v4-flash --sandbox docker ``` Use another sandbox only if the run needs remote scale; see [Sandboxes](/docs/benchflow/sandboxes). While the agent works, a terminal (TTY) shows the live Rich dashboard — progress bar, pass/fail counts, and a per-task activity column that tracks tool calls/tokens and labels the non-agent stretches (`creating sandbox…`, `installing agent…`, `verifying…`). Plain output (CI, pipes) keeps the throttled progress heartbeat instead: single-concurrency runs print a line about every 45 seconds (`… 6.2min, 12 tool calls (last: …)`), and multi-concurrency jobs gate it off by default. `--quiet` silences both. The full event stream lands in `trajectory/acp_trajectory.jsonl` in the rollout dir; the run ends with the `✓ Score` line either way. The first rollout runs the agent's `install_cmd` inside the sandbox (a few minutes for agents that bootstrap toolchains); artifacts land in the jobs dir exactly as for built-in agents, including the gateway's raw LLM trace. Details worth knowing: - `DEEPSEEK_BASE_URL` is optional: unset, `deepseek/*` models default to the public OpenAI-compatible endpoint (`https://api.deepseek.com/v1`); set it only to route to a different OpenAI-compatible deployment. - The fetch happens **at most once per process**, only on a resolution miss, and only fills gaps — it never shadows a built-in or already-registered agent name. - Availability of a name depends on it being merged to the agents repo's `main`. What exists is listed in that repo's `acp/` directory and, for the ACP-registry tier, its generated `acp-registry/AGENTS.md`. ## 2. Pin the source: `BENCHFLOW_AGENTS_SOURCE` Override where the autoload fetches from — a branch/ref, another repo, a local directory, or off entirely: ```bash export BENCHFLOW_AGENTS_SOURCE="benchflow-ai/agents@my-branch" # owner/repo[@ref] export BENCHFLOW_AGENTS_SOURCE="/path/to/agents-checkout" # local dir export BENCHFLOW_AGENTS_SOURCE="off" # disable autoload ``` Accepted off-values: `off`, `0`, `none`, `disabled`, `false`. Pinning a ref is the standard way to try an agent from an open PR — e.g. verified live on BenchFlow 0.6.6: `BENCHFLOW_AGENTS_SOURCE="benchflow-ai/agents@add-prime-agent"` resolved and ran the `prime-agent` manifest with zero local setup. ## 3. Local checkout at import: `BENCHFLOW_AGENTS_DIR` For agents-repo development: point at a checkout and every `/manifest.toml` under it merges into the registry when `benchflow` imports (not lazily on miss): ```bash export BENCHFLOW_AGENTS_DIR=/path/to/agents-checkout ``` Unlike the miss-driven autoload, this path loads even for names that would never miss, and it is the loop used while editing a manifest. It is additive and compatible-merge only: colliding with an existing agent's aliases is a hard error rather than a silent shadow. Unset, the import is byte-for-byte identical to core — the mechanism is strictly opt-in. ## 4. Plugin packages (entry points) Agents that need host-side Python (session-factory adapters like the omnigent harnesses, or packaged code agents like `mini-swe-acp`) ship as pip packages that register through the `benchflow.agents` entry-point group — installing the package is all it takes: ```bash pip install "mini-swe-acp @ git+https://github.com/benchflow-ai/agents#subdirectory=acp/mini-swe-acp" bench eval run --tasks-dir ./tasks --agent mini-swe --model openai/gpt-4o-mini ``` These load at `benchflow` import time. A plugin that fails to import never blocks the run — the failure is recorded and surfaced in the "Unknown agent" error message if its name is later requested. ## Precedence 1. Built-in registry (core `AGENTS`). 2. `BENCHFLOW_AGENTS_DIR` manifests — merged at import; a collision with an existing agent name or alias is a hard error, so manifests never shadow built-ins. 3. Entry-point plugin packages — loaded at import, after the manifest merge. These register through plain `register_agent`, which overwrites by name: a plugin **can** replace a built-in (or manifest-registered) agent that shares its name. Well-behaved plugins skip names the registry already owns (as the acp-registry package does). 4. Remote autoload (`BENCHFLOW_AGENTS_SOURCE`, default `benchflow-ai/agents@main`) — consulted last, once, only for names still unknown at resolution time; it fills gaps and never overwrites. Manifest capabilities are deliberately bounded: a `manifest.toml` is data-only (install/launch commands, env mapping, model-routing hints — the [agents-repo contract](https://github.com/benchflow-ai/agents/tree/main/contract)). Anything needing host-side logic (credential files, session factories, native MCP config) must come in as a plugin package or a core agent instead. --- ## /docs/benchflow/getting-started Run one scored BenchFlow evaluation on your own machine. This path uses Docker and an existing Codex, Claude, or Gemini login; it does not require Daytona or another cloud sandbox account. Want the shortest copy-paste path? [Start a real local task in five minutes](/docs/benchflow/start-in-5-minutes). This page explains each part of that run and the available alternatives. ## What you need - [`uv`](https://docs.astral.sh/uv/) — the install command below can provision the required Python 3.12 runtime for you - Docker Desktop or another working Docker daemon - One supported agent login or API key Start Docker, then confirm it is reachable: ```bash docker info >/dev/null ``` If this command fails, start Docker before continuing. BenchFlow builds a task image and runs the agent inside it; the agent itself does not run directly on your host. ## 1. Install BenchFlow ```bash uv tool install --python 3.12 --upgrade benchflow bench --version ``` `benchflow` and `bench` are aliases for the same CLI. If `uv` reports `Executables already exist: bench, benchflow`, repeat the install with `--force` to replace an older entrypoint. Working from a source checkout instead? Use the repository environment: ```bash git clone https://github.com/benchflow-ai/benchflow cd benchflow uv sync --extra dev --locked uv run bench --version ``` ## 2. Sign in to one agent An existing subscription login is enough; an API key is not required for the Codex or Claude examples. ```bash # ChatGPT subscription through Codex CLI codex login # Or Claude subscription through Claude Code claude auth login # Or Gemini CLI's interactive login gemini ``` BenchFlow detects the saved host credential and makes it available inside the sandbox. If you prefer API keys, CI tokens, or a provider-hosted model, see [Authentication](/docs/benchflow/authentication). ## 3. Run one local evaluation This example downloads one public SkillsBench task, runs Codex inside a local Docker sandbox, executes the task's verifier, and saves the full trajectory: ```bash bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/3d-scan-calc \ --agent codex \ --model gpt-5.5 \ --sandbox docker \ --concurrency 1 ``` Use one of these agent/model pairs if you signed in somewhere else: | Host login | Replace the two flags with | |---|---| | Claude Code | `--agent claude --model claude-sonnet-4-6` | | Gemini CLI | `--agent gemini --model gemini-3.1-pro-preview` | The first run may take several minutes while Docker downloads and builds the task image. The benchmarked agent may pass or fail the task; either outcome is a valid completed evaluation. ## 4. Read the result The console ends with `[PASS]`, `[FAIL]`, or an execution error. `[FAIL]` means the agent completed but did not reach the verifier's pass threshold; it does not mean BenchFlow itself failed. Results are written under `jobs/`: ```text jobs/ / summary.json __/ result.json timing.json prompts.json trajectory/ acp_trajectory.jsonl llm_trajectory.jsonl # optional provider-level trace verifier/ reward.txt test-stdout.txt ``` Use the CLI to summarize them: ```bash bench eval list jobs/ bench eval metrics jobs/ ``` `result.json` is the quickest place to check the raw reward, error status, tool-call count, and token usage. `trajectory/acp_trajectory.jsonl` contains the agent/tool interaction trace. ## Run your own local task Point `--tasks-dir` at either one task package or a directory of task packages: ```bash bench eval run \ --tasks-dir tasks/my-task \ --agent codex \ --model gpt-5.5 \ --sandbox docker ``` Docker is BenchFlow's default, so `--sandbox docker` may be omitted. Keeping it in your first commands makes the execution location explicit. ## Where to go next | Goal | Read | |---|---| | Use a different login, API key, or provider | [Authentication](/docs/benchflow/authentication) | | Run local batches, YAML configs, or skill comparisons | [Running evaluations](/docs/benchflow/running-evaluations) | | Decide between Docker, Apple Container, Daytona, Modal, and AgentCore | [Sandboxes](/docs/benchflow/sandboxes) | | Understand task, agent, rollout, and verifier terminology | [Concepts](/docs/benchflow/concepts) | | Create a task | [Task authoring](/docs/benchflow/task-authoring) | | Look up every flag | [CLI reference](/docs/benchflow/reference/cli) | For local development and small evaluation sets, you can stop here: no Daytona setup is necessary. --- ## /docs/benchflow/llm-judge Use an LLM to evaluate agent outputs against a rubric instead of deterministic tests. --- ## When to use LLM-as-judge Use LLM-as-judge when the task output is subjective, open-ended, or hard to verify with unit tests — legal analysis, code review quality, document drafting, research summaries. For tasks with a clear right answer (e.g. "write fizzbuzz"), stick with deterministic `test.sh` verifiers. BenchFlow's LLM judge supports: - **First-class verifier strategy** — `type: llm-judge` in `verifier/verifier.md`, no `test.sh` needed - **Multi-criterion rubrics** with binary, likert, and numeric scoring - **Per-criterion weights** for non-uniform importance - **Dense reward events** emitted per criterion during evaluation - **Multi-provider routing** across Anthropic, OpenAI, and Google models - **Configurable aggregation** (weighted mean, all-pass, any-pass, threshold) The judge is a **first-class verification method** alongside the deterministic `test.sh` verifier. A task selects it with one line of config — the framework handles deliverable collection, prompting, provider routing, retries, and reward aggregation. --- ## Quick start ### 0. Install the judge provider SDKs The judge calls the Anthropic, OpenAI, and Google SDKs — these are **not** installed by default. Install the `judge` extra (you only need at least one provider's SDK for the model you use, but the extra ships all three): ```bash # in a checkout uv sync --extra judge # or as an installed tool uv tool install --python 3.12 --upgrade 'benchflow[judge]' # or with pip python3.12 -m pip install --upgrade 'benchflow[judge]' ``` If no provider SDK is installed, the judge cannot run: the verifier raises a **verifier error** (the rollout is marked errored) rather than silently recording a reward of `0.0` — a missing dependency is an environment failure, not a score. ### 1. Select the judge verifier in `verifier/verifier.md` ```md --- document_version: "0.3" verifier: name: draft-quality-verifier default_strategy: judge strategies: judge: type: llm-judge model: claude-sonnet-4-6 rubric: rubrics/verifier.toml input_dir: /app outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json --- ## verifier intent Score the submitted deliverables in `/app` against the rubric. ``` That's the entire native verifier. There is **no `verifier/test.sh`** to write — the `Verifier` downloads the agent's deliverables from `input_dir`, scores them against the rubric, and writes `reward.json` itself. Judge credentials are resolved from the host environment or `.env` and are scoped to the verifier runtime. ### 2. Write `verifier/rubrics/verifier.toml` Place it where the strategy's `rubric` field points: ```toml [[criterion]] name = "accuracy" description = "The response accurately addresses the question with correct facts" type = "binary" weight = 3.0 [[criterion]] name = "clarity" description = "The response is well-organized and easy to understand" type = "likert" points = 5 weight = 1.0 [scoring] aggregation = "weighted_mean" ``` A Harvey LAB style `rubric.json` works too — set `rubric: rubrics/verifier.json`: ```json { "title": "Task Title", "criteria": [ {"id": "criterion-1", "title": "...", "match_criteria": "What constitutes a pass"} ] } ``` That's it. Run the task as usual — the reward is the proportion of criteria passed (or the configured aggregation), a partial float in `[0, 1]`. --- ## `[verifier]` reference | Field | Type | Default | Description | |-------|------|---------|-------------| | `type` | string | `"script"` | `"script"` (run `verifier/test.sh`) or `"llm-judge"` | | `timeout_sec` | float | `600` | Overall verifier timeout | | `env` | table | `{}` | Env vars for the verifier — judge API keys go here | ### Native `llm-judge` strategy fields | Field | Type | Default | Description | |-------|------|---------|-------------| | `model` | string | `"claude-sonnet-4-6"` | Judge model; provider routed from prefix | | `rubric` | string | `"rubrics/verifier.toml"` | Rubric file relative to `verifier/` (`.toml` or `.json`) | | `input_dir` | string | `"/app"` | Sandbox dir whose contents are graded | | `input_type` | string | `"deliverables"` | Only `"deliverables"` is supported — trajectory judging is not available at verify time | | `context` | string | `""` | Extra judge context (defaults to the task instruction) | Legacy split-layout packages still project these fields through `[verifier]` and `[verifier.judge]` in `task.toml`; native `task.md` packages should use `verifier/verifier.md`. --- ## Library use — `LLMJudgeRewardFunc` The judge is also a composable `RewardFunc`, usable directly or from a custom `test.sh` verifier: ```python import asyncio from pathlib import Path from benchflow.rewards import LLMJudgeRewardFunc func = LLMJudgeRewardFunc(rubric_path=Path("rubric.toml")) score = asyncio.run(func.score(Path("/app"))) print(f"Score: {score:.2f}") ``` Auto-discovery — if `rubric.toml`/`rubric.json` is in the rollout directory or its parent, it's found automatically: ```python func = LLMJudgeRewardFunc() score = asyncio.run(func.score(Path("/app"))) ``` --- ## rubric.toml reference ### `[judge]` section | Field | Type | Default | Description | |-------|------|---------|-------------| | `model` | string | `"claude-sonnet-4-6"` | LLM model for judging. Prefix with `anthropic/`, `openai/`, or `google/` to force a provider | | `mode` | string | `"individual"` | `"individual"` scores each criterion separately; `"batched"` is reserved for future use | | `files` | string[] | `[]` | Default files to evaluate (fallback when a criterion doesn't specify its own) | | `timeout` | int | `120` | Timeout in seconds per judge call | ### `[[criterion]]` entries | Field | Type | Default | Description | |-------|------|---------|-------------| | `name` | string | — | Criterion identifier (falls back to first 40 chars of description) | | `description` | string | **required** | What the judge should evaluate | | `type` | string | `"binary"` | `"binary"`, `"likert"`, or `"numeric"` | | `weight` | float | `1.0` | Relative importance in aggregation | | `points` | int | `5` | Scale for likert type (1 to N) | | `min` | float | `0.0` | Minimum for numeric type | | `max` | float | `100.0` | Maximum for numeric type | | `files` | string[] | `[]` | Specific files this criterion should evaluate | ### `[scoring]` section | Field | Type | Default | Description | |-------|------|---------|-------------| | `aggregation` | string | `"weighted_mean"` | How to combine criterion scores | | `threshold` | float | `0.7` | Pass threshold (only used with `"threshold"` aggregation) | ### Score normalization Each criterion type normalizes its raw score to `[0, 1]`: | Type | Raw | Normalized | |------|-----|------------| | `binary` | pass/fail | `1.0` or `0.0` | | `likert` | 1–N integer | `(raw - 1) / (points - 1)` | | `numeric` | min–max float | `(raw - min) / (max - min)`, clamped to `[0, 1]` | ### Aggregation strategies | Strategy | Behavior | |----------|----------| | `weighted_mean` | `sum(score × weight) / sum(weight)` — continuous reward | | `all_pass` | `1.0` if every criterion scores ≥ 0.5, else `0.0` | | `any_pass` | `1.0` if any criterion scores ≥ 0.5, else `0.0` | | `threshold` | `1.0` if weighted mean ≥ threshold, else `0.0` | --- ## Criterion types ### Binary (pass/fail) The judge decides whether the criterion is satisfied. The LLM returns `{"verdict": "pass", "reasoning": "..."}`. ```toml [[criterion]] name = "has-executive-summary" description = "The document includes an executive summary in the first section" type = "binary" ``` ### Likert (scaled) The judge rates on a 1-to-N scale. The LLM returns `{"score": 4, "reasoning": "..."}`. ```toml [[criterion]] name = "writing-quality" description = "Overall quality of prose — grammar, flow, and precision" type = "likert" points = 5 ``` A score of 3 on a 5-point scale normalizes to `(3-1)/(5-1) = 0.5`. ### Numeric (range) The judge assigns a value within a continuous range. The LLM returns `{"score": 75.0, "reasoning": "..."}`. ```toml [[criterion]] name = "coverage-pct" description = "Percentage of key topics from the source material covered in the summary" type = "numeric" min = 0.0 max = 100.0 ``` --- ## Inline criteria (no TOML file) For programmatic use or Harvey LAB-style criteria, pass criteria directly: ```python func = LLMJudgeRewardFunc( criteria=[ { "description": "The response is factually accurate", "type": "binary", "weight": 2.0, }, { "description": "The response addresses all parts of the question", "type": "binary", "weight": 1.0, }, ], judge_model="claude-sonnet-4-6", ) ``` Harvey LAB `match_criteria` keys are also supported: ```python func = LLMJudgeRewardFunc( criteria=[ {"match_criteria": "Identifies the key risk factors", "type": "binary"}, {"match_criteria": "Provides supporting evidence", "type": "binary"}, ], ) ``` --- ## Dense reward events Each criterion emits a `RewardEvent` during evaluation, enabling per-criterion observability and training signal: ```python func = LLMJudgeRewardFunc(rubric_path=Path("rubric.toml")) score = await func.score(rollout_dir) for event in func.events: print(f" {event.source}: {event.reward:.2f} (step {event.step})") ``` Output: ``` criterion:accuracy: 1.00 (step 0) criterion:clarity: 0.50 (step 1) criterion:completeness: 0.75 (step 2) ``` Events have type `"dense"`, a `reward` in `[0, 1]`, a `source` of `"criterion:{name}"`, and a `step` index. Events are cleared between `score()` calls. --- ## Multi-provider routing The judge model string determines which provider SDK is used: | Prefix | Provider | Auth env var | |--------|----------|--------------| | `claude-*`, `anthropic/*` | Anthropic | `ANTHROPIC_API_KEY` | | `gpt-*`, `o1*`, `o3*`, `o4*`, `openai/*` | OpenAI | `OPENAI_API_KEY` | | `gemini*`, `google/*` | Google | `GOOGLE_API_KEY` or `GEMINI_API_KEY` | If the primary provider fails, the judge falls back through the other providers with retries and exponential backoff. The provider SDKs ship in the `judge` extra (`uv sync --extra judge`). If *none* are installed, the judge raises a verifier error instead of recording a reward — see [step 0](#0-install-the-judge-provider-sdks). --- ## Evaluation output After scoring, an `evaluation_details.json` is written to the rollout directory: ```json { "score": 0.75, "n_passed": 2, "n_total": 3, "results": [ { "id": "accuracy", "description": "The response accurately addresses the question", "score": 1.0, "weight": 3.0, "verdict": {"verdict": "pass", "reasoning": "..."} }, { "id": "clarity", "description": "The response is well-organized", "score": 0.5, "weight": 1.0, "verdict": {"score": 3, "reasoning": "..."} } ] } ``` The `score` field is the actual aggregated score from the configured strategy, not `n_passed / n_total`. --- ## File discovery The judge automatically discovers deliverable files in the rollout directory. Supported formats: | Extension | Reader | Dependency | |-----------|--------|------------| | `.txt`, `.md`, `.json`, `.csv` | Built-in | None | | `.docx` | pandoc or python-docx | `pandoc` (preferred) or `pip install python-docx` | | `.xlsx` | openpyxl | `pip install openpyxl` | | `.pptx` | markitdown | `pip install markitdown` | | `.pdf` | pdfplumber | `pip install pdfplumber` | Files larger than 50 MB are skipped. Hidden files (starting with `.`) and internal metadata files (`rubric.json`) are excluded. File content is truncated at 15,000 characters per file when sent to the judge. To scope a criterion to specific files: ```toml [[criterion]] name = "memo-quality" description = "The legal memo follows IRAC structure" files = ["memo.docx", "analysis.md"] ``` --- ## Python API All rubric config types are importable from the top level: ```python from benchflow import ( Criterion, JudgeConfig, LLMJudgeRewardFunc, RubricConfig, ScoringConfig, load_rubric, # dispatches on extension (.toml / .json) load_rubric_json, load_rubric_toml, ) # Load and inspect a rubric (TOML or Harvey LAB style JSON) rubric = load_rubric(Path("rubric.json")) print(f"Model: {rubric.judge.model}") print(f"Criteria: {len(rubric.criteria)}") for c in rubric.criteria: print(f" {c.id} ({c.type}, weight={c.weight})") ``` --- ## Worked example — legal analysis task A legal document analysis task scored entirely by config — no `test.sh`: ```md # verifier/verifier.md --- document_version: "0.3" verifier: name: legal-analysis-judge default_strategy: judge strategies: judge: type: llm-judge model: claude-sonnet-4-6 rubric: rubrics/verifier.toml input_dir: /app outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json --- ## verifier intent Score `analysis.md` against the legal review rubric. ``` ```toml # verifier/rubrics/verifier.toml [judge] files = ["analysis.md"] [[criterion]] name = "key-terms-identified" description = "All material terms from the contract are identified and listed" type = "binary" weight = 2.0 [[criterion]] name = "risk-assessment" description = "Each identified risk includes severity rating and mitigation suggestion" type = "likert" points = 5 weight = 3.0 [[criterion]] name = "completeness" description = "Percentage of contract sections addressed in the analysis" type = "numeric" min = 0 max = 100 weight = 1.0 [scoring] aggregation = "weighted_mean" ``` The framework downloads the agent's deliverables from `/app`, grades each criterion, aggregates, and writes `reward.json` — no scripting required. --- ## Where to go next - [Concepts](/docs/benchflow/concepts) — the five primitives including Verifier - [Task authoring](/docs/benchflow/task-authoring-task-md) — `task.md` frontmatter, `verifier/`, verifier contract - [Running benchmarks](https://github.com/benchflow-ai/benchflow/blob/main/docs/running-benchmarks.md) — Harvey LAB uses LLM-as-judge - [Python API reference](/docs/benchflow/reference/python-api) — `LLMJudgeRewardFunc` and friends --- ## /docs/benchflow/progressive-disclosure ## TL;DR `BaseUser` is a Python callback that drives a benchflow rollout across multiple rounds. Each round: the callback sees the previous verifier result and decides what to tell the agent next, or stops the loop. No second LLM, no outbox protocol — just a function that knows how to grade and hint. It was built for the SWE-bench Pro progressive-disclosure use case: the dataset's instructions are long structured specs that overwhelm agents in a single turn. A `BaseUser` lets you compress the spec for round 0, watch which tests fail, then disclose hints from the spec on subsequent rounds — all driven by deterministic Python, not by another LLM acting as a "user." Other agent-eval frameworks model this with a "simulated user" — a second LLM running in a sidecar container that talks to the agent over a side channel. benchflow's `BaseUser` is just in-process Python: no second LLM, no sidecar, no outbox protocol. ```python import benchflow as bf from benchflow import FunctionUser, RoundResult from benchflow.rollout import RolloutConfig, Scene from benchflow._utils.benchmark_repos import resolve_source def progressive(round: int, instruction: str, rr: RoundResult | None) -> str | None: if round == 0: return instruction.split("\n")[0] # terse: first line only if rr and (rr.rewards or {}).get("reward", 0) >= 1.0: return None # passed, stop if round >= 3: return None # cap at 3 rounds return ( f"Tests failed:\n{rr.verifier_output}\n\n" # show failures + spec f"Full spec:\n{instruction}" ) config = RolloutConfig( task_path=resolve_source("benchflow-ai/swebenchpro", path="instance_flipt-io__flipt-..."), scenes=[Scene.single(agent="opencode", model="anthropic/claude-sonnet-4-6")], user=FunctionUser(progressive), max_user_rounds=3, environment="docker", ) result = await bf.run(config) ``` --- ## Case study: SWE-bench Pro SWE-bench Pro tasks ship long, structured `instruction.md` specs (typically 2-5KB) describing API requirements, test fixtures, and expected behaviors. Single-shot agents either drown in the spec or under-engineer because they bail before reading to the bottom. The SWE-bench Pro eval that motivated this feature wanted exactly this loop: ``` round 0 "Fix the bug described here: " agent attempts → tests fail round 1 "Tests failed. Here is the full requirements section: ." agent retries → tests still fail round 2 "Still failing. Here's the full original spec: " agent makes final attempt ``` Rule-based, deterministic, and the "user" never needs to think — the disclosure schedule is fixed. Spinning up a second LLM to play the user role would (a) cost double, (b) introduce nondeterminism, and (c) require an outbox protocol the agent has to learn. ### Validation (2026-04-25, 5 SWE-bench Pro tasks, Daytona, Gemini 3.1 Pro Preview) | Task | Oracle | Single-round baseline | 3-round progressive (final) | Per-round soft-verify | |------|--------|-----------------------|------------------------------|------------------------| | ansible | ✅ 1.0 | ✅ 1.0 (23 tools, 207s) | ✅ 1.0 (126 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | flipt | ✅ 1.0 | ❌ 0.0 (61 tools, 1444s) | ❌ 0.0 (195 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | openlibrary | ✅ 1.0 | ✅ 1.0 (32 tools, 340s) | ✅ 1.0 (82 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | navidrome | ✅ 1.0 | (not tested) | ❌ 0.0 (145 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | | qutebrowser | ✅ 1.0 (with `cleanup_conftests=false`) | ❌ 0.0 (verifier broken pre-fix) | ✅ 1.0 (183 tools, 3 rounds) | 0.0 / 0.0 / 0.0 | What this run shows and doesn't show: - **The infrastructure works on real SWE-bench Pro tasks.** All 5 tasks completed 3 rounds end-to-end (after one retry on ansible/qutebrowser to clear intermittent flake). Round trajectories captured, soft_verify runs between rounds, BaseUser callback drives the loop. - **3/5 hit the canonical reward** (ansible, openlibrary, qutebrowser). flipt and navidrome stayed at 0.0 across all three rounds — Gemini 3.1 Pro doesn't crack them with this hint schedule, and progressive disclosure didn't help. - **Per-round soft-verify scored 0.0 even on tasks where the final hardened verify scored 1.0.** Soft-verify runs between rounds without the full hardening sequence (no workspace restore, no process kill so the sandbox stays alive), so its scoring can diverge from the final verifier. The user's hint schedule reacts to soft-verify, not the canonical reward — something to keep in mind when designing the loop. - **First-run flake.** ansible's first run hit a transport EOF after 17min and qutebrowser timed out at 50min. Both succeeded on retry. v0.3.3 adds `agent_idle_timeout` (default 600s) and clearer EOF diagnostics so the next time a hang happens the failure is fast and actionable rather than silent. This is one model on one day, not a published comparison. The notebook at [`examples/swebench_pro_progressive_disclosure.ipynb`](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/swebench_pro_progressive_disclosure.ipynb) has the executable cells. --- ## Where it lives in the rollout lifecycle `BaseUser` plugs into the existing `Rollout` lifecycle ([concepts](/docs/benchflow/concepts#rollout-lifecycle)) without changing any of the existing phases. When `RolloutConfig.user` is set, `Rollout._run_user_loop()` replaces the single-pass `connect → execute → disconnect` block with a per-round version: ``` setup() → start() → install_agent() ↓ [oracle setup if oracle_access=True: read /solution, hide it from agent] ↓ user.setup(instruction, solution) ← once ↓ ┌─ user.run(round, instruction, rr) → str | None │ │ None: break │ ↓ │ connect_as(role) │ execute(prompts=[prompt]) │ disconnect() │ ↓ │ soft_verify() ← partial hardening, sandbox stays alive │ ↓ │ build RoundResult, log, repeat └─ │ ↓ (loop ends when user returns None or max_user_rounds reached) [oracle restore: mv /solution_oracle_backup → /solution for final verify] ↓ verify() ← full hardening, final reward ↓ cleanup() ``` Multi-scene / multi-role configs are not compatible with `User` — the loop assumes one Scene with one Role. Setting both raises `ValueError`. --- ## Soft-verify and full-verify: two different verifiers Between rounds, BenchFlow needs to score the agent's progress so the user can react. But the final, end-of-rollout verifier does destructive things (kills the agent, restores the workspace, chowns to root) that would prevent the next round from running. So BenchFlow executes **two** verifier passes: | | Soft-verify (between rounds) | Full-verify (end of rollout) | |---|---|---| | Kills agent processes | ❌ no | ✅ yes | | Restores workspace from snapshot | ❌ no | ✅ optional, task-driven | | Purges agent-injected `conftest.py`, `sitecustomize.py`, `.pth` | ✅ yes | ✅ yes | | Locks down PATH/PYTHONPATH | ✅ yes | ✅ yes | | `chmod 777 /logs/verifier` | ✅ yes (so non-root verifier can write) | n/a (root) | | Runs verifier | ✅ yes | ✅ yes | | Result | feeds `RoundResult.rewards` | the rollout's final score | Soft-verify is intentionally weaker than full-verify — losing some score-gaming protection in exchange for keeping the sandbox alive. The cleanup step still purges agent-injected hook files (`CLEANUP_CMD`), so an agent can't plant a `conftest.py` that flips the round score. --- ## API ### `BaseUser` ```python from benchflow import BaseUser, RoundResult class MyUser(BaseUser): async def setup(self, instruction: str, solution: str | None = None) -> None: """Called once before round 0. instruction — the original task instruction (from instruction.md) solution — gold answer if oracle_access=True, else None """ self.spec = instruction self.gold = solution async def run( self, round: int, instruction: str, round_result: RoundResult | None = None, ) -> str | None: """Return the next prompt, or None to stop. round — 0-indexed instruction — original task instruction (unchanged each round) round_result — None on round 0; previous round's outcome on subsequent rounds """ ... ``` ### `RoundResult` Dataclass passed to `run()` from round 1 onward. ```python @dataclass class RoundResult: round: int # 0-indexed trajectory: list[dict] # ACP events from this round only rewards: dict | None # verifier rewards (None if verifier crashed) verifier_output: str | None # raw verifier stdout/log verifier_error: str | None # exception message if verifier failed n_tool_calls: int # tool calls in this round ``` ### `PassthroughUser` Sends the instruction unchanged on round 0, stops on round 1. Use it as the explicit single-round-equivalent. ### `FunctionUser` Wraps a plain function as a `BaseUser`. Sync or async — uses `inspect.isawaitable` to detect. ```python def fn(round, instruction, rr): ... user = FunctionUser(fn) async def afn(round, instruction, rr): ... user = FunctionUser(afn) ``` ### `RolloutConfig` fields ```python user: BaseUser | None = None # the callback max_user_rounds: int = 5 # cap on rounds (loop also stops when user returns None) oracle_access: bool = False # expose gold solution to user.setup() ``` --- ## Oracle access When `oracle_access=True`: 1. Before round 0, the rollout reads `/solution/solve.sh` and passes its contents to `user.setup(instruction, solution=...)`. 2. The rollout moves `/solution` → `/solution_oracle_backup` so the agent can't read it during its rounds. 3. Between rounds, soft-verify temporarily restores `/solution` (some verifiers consult it) then re-hides it. 4. Before the final `verify()`, the rollout permanently restores `/solution`. Step 4 is wrapped in `try/finally` against the user loop: if a round throws, the restore still runs. > ⚠️ Setting `oracle_access=True` *without* a `User` is a misconfiguration — the solution stays exposed to the agent for the entire rollout. benchflow logs a `WARNING` at setup time when this happens. Use cases for oracle access: - **Dataset generation** — the user has the answer, generates an optimal prompt for the agent - **Curriculum learning** — progressively reveal pieces of the solution - **Research** — measure how much oracle information is required for an agent to succeed --- ## Per-task hardening opt-outs The verifier's pre-run cleanup deletes `conftest.py` outside `/tests/` to prevent reward-hacking. Some tasks (qutebrowser) ship legitimate `conftest.py` files that fix real circular imports — deleting them breaks pytest collection. Tasks opt out in `task.toml`: ```toml [verifier.hardening] cleanup_conftests = false ``` | Flag | Default | Effect when `false` | |------|---------|---------------------| | `cleanup_conftests` | `true` | Don't delete `conftest.py` outside `/tests/` before verify | `sitecustomize.py`, `.pth` files, and `*.py` in `/tmp` always get cleaned — they have no legitimate use in a test artifact and disabling them broadens the attack surface beyond what real-world tasks need. Unknown keys in `[verifier.hardening]` are warned and ignored. String values for boolean flags are rejected. --- ## Failure modes The user loop catches exceptions from `user.run()` and stops, with the exception message stored in `Rollout._error`: ``` [User] round 2: prompt='Try again, focusing on...' ERROR user.run() failed at round 2: KeyError: 'spec_section' ``` `soft_verify()` between rounds catches its own timeouts and crashes — they surface as `RoundResult.verifier_error`, not as a rollout-level failure. The next round still runs and the user can decide what to do. Trajectory and tool counts are sliced per round from `Rollout._trajectory`. The session counters reset on `disconnect()`, so each round's `RoundResult.trajectory` and `n_tool_calls` reflect only that round's events, not cumulative. --- ## Comparison with multi-agent simulated user benchflow has two patterns for multi-round agent runs. Neither requires a sidecar container. | Pattern | What "user" is | When to use | |---------|---------------|-------------| | **`BaseUser` callback (this doc)** | Python function in the scheduler process | Programmatic, deterministic, rule-based. No second LLM. Cheap. Best for progressive disclosure, curriculum, scripted hints. | | **Multi-role Scene with simulated-user role** ([use-cases §1](/docs/benchflow/use-cases#1-interactive-user-simulation)) | Another LLM with full tool access | Open-ended, conversational. The "user" can read files, check outputs, give nuanced feedback. Best when the user's behavior must itself be adaptive or LLM-quality. | The two coexist. Choose based on whether your "user" needs to think (Scene-based) or just decide (`BaseUser`). For the SWE-bench Pro use case, the disclosure schedule is fixed, the grading is the verifier, and there's nothing for a second LLM to add — `BaseUser` wins on cost and determinism. --- ## Worked examples - [`examples/swebench_pro_progressive_disclosure.ipynb`](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/swebench_pro_progressive_disclosure.ipynb) — the SWE-bench Pro case study, executable end-to-end with the latest oracle/baseline data. - [`examples/swebench_pro_user_dogfood.py`](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/swebench_pro_user_dogfood.py) — runnable script for any of the 5 SWE-bench Pro tasks. `--task flipt --max-rounds 3`. - [`examples/user_dogfood.py`](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/user_dogfood.py) — minimal edit-pdf task with `FunctionUser`, useful as a starting template. --- ## /docs/benchflow/reference/cli BenchFlow uses a resource-verb pattern: `bench `. ```bash bench --version ``` --- ## bench agent > **`bench agent` is agent management only.** `bench agent list` and `bench > agent show` operate on **registered AI agents** (Claude Code, Gemini CLI, > Codex, OpenHands, …) — the programs that solve tasks. Onboarding a third-party > benchmark (scaffold → drive → parity-gate a `benchmarks//` adoption) is a > separate workflow under [`bench eval adopt`](#bench-eval-adopt). The legacy > `bench agent create|run|verify` still work as hidden deprecated aliases through > 0.6, printing a one-line notice; they are removed in 0.7. ### bench agent list List all registered agents with their protocol and native/default auth requirements. Provider-prefixed models may use provider-specific credentials; Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT`. ```bash bench agent list ``` ### bench agent show Show details for a specific agent, including native/default auth and a note about provider-specific credentials. ```bash bench agent show gemini ``` ## bench eval adopt Bring a third-party benchmark into the environment framework. `bench eval adopt` is a **single multi-mode command**: it scaffolds a `benchmarks//` package, drives the codex conversion, and parity-gates the result. The conversion guide is embedded in the command itself. It was previously a subgroup with `init`/`convert`/`verify` subcommands, and before that `bench agent create|run|verify`; both `bench adopt init|convert|verify` and `bench agent create|run|verify` still work as hidden deprecated aliases through 0.6 (they print a one-line notice and are removed in 0.7). The mode is selected by flags: - `bench eval adopt ` (default, **convert**) — scaffold `benchmarks//` if it is missing, then drive the codex conversion of the upstream benchmark at ``. Use `--dry-run` to preview the launch command without running it (and without writing any files). - `bench eval adopt --scaffold-only` — only scaffold the package, do not convert. - `bench eval adopt --verify` — run the parity gate for the named benchmark. In convert mode the argument is the SOURCE repo/path to adopt; in `--verify` / `--scaffold-only` mode it is the benchmark SLUG. `--verify` and `--scaffold-only` are mutually exclusive. **Convert (default).** The command resolves the slug (`--name`, else derived from the source basename), auto-scaffolds `benchmarks//` if it does not exist (a no-op if it already does), then launches the host `codex` CLI to drive the conversion toward a `benchmarks//` pull request. It assembles the adoption context — the source, the target path, the adoption skills, and the embedded conversion guide — and runs `codex exec` against the repo root. It is fail-closed on credentials: `codex` needs `OPENAI_API_KEY` (or `CODEX_API_KEY`) in the environment, or a `~/.codex/auth.json` from `codex login`, otherwise the command exits before assembling any context. `--dry-run` prints the exact launch command without running it (no credentials required) and writes no files. ```bash # Print the codex launch command without running it bench eval adopt https://github.com/org/some-benchmark --dry-run # Scaffold-if-missing, then launch the host codex driver against a local source bench eval adopt ./vendor/some-benchmark --name my-bench --model o3 ``` | Flag | Default | Description | |------|---------|-------------| | `--name` | derived from source | Benchmark slug (default: from source basename) | | `--model` | codex default | Model for the codex driver | | `--dry-run` | `false` | Print the launch command, do not run (writes no files) | | `--codex-bin` | `codex` | Host codex binary | | `-c`, `--codex-config` | — | Codex config override as `key=value`, passed through to codex as `-c key=value`; repeatable. Use it to work around host `~/.codex/config.toml` drift without editing the file — e.g. `-c service_tier=flex` when an installed codex version rejects a stale value. | | `--benchmarks-dir` | repo `benchmarks/` | Target benchmarks/ directory (used by the auto-scaffold) | **Scaffold only.** `bench eval adopt --scaffold-only` writes only the package layout, which mirrors the reference benchmark `benchmarks/programbench/`: `benchflow.py` (converter), `main.py`, `parity_test.py`, `run_.py`, `.yaml`, `benchmark.yaml`, `parity_experiment.json` (status `template`), `README.md`, and `__init__.py`. It is fail-closed: the slug is validated (lowercase, leading letter, single internal hyphens, max 64 chars) and the command refuses to overwrite an existing benchmark directory. ```bash bench eval adopt my-bench --scaffold-only bench eval adopt my-bench --scaffold-only --benchmarks-dir ./benchmarks ``` | Flag | Default | Description | |------|---------|-------------| | `--benchmarks-dir` | repo `benchmarks/` | Target benchmarks/ directory | **Verify.** `bench eval adopt --verify` runs the parity gate for an adopted benchmark and emits a confidence verdict. It reads `benchmarks//parity_experiment.json` and scores two layers: a deterministic conversion-faithfulness floor (every compared criterion's converted verdict must match the original's verdict on identical inputs) and a statistical reward-distribution layer (every legacy-vs-converted reward delta must sit within `--tolerance`). The gate is parity-only — a faithful conversion reproduces the original's behavior, including any reward-hackability the source has; it never "improves" or sanitizes the source. The verdict is one of `parity-confirmed`, `parity-divergent`, or `insufficient-evidence` (no recorded comparisons). On any non-confirmed verdict the command exits non-zero and emits a draft GitHub issue body for human support — printed to stdout, or written to `--issue-out`. The draft is never filed automatically. Pass `--roundtrip-task` to also run the structural round-trip conformance check on a concrete task directory. By default the gate **scores the recorded** `parity_experiment.json` — fast, but it trusts an artifact the conversion produced about itself. Pass `--rerun` to **independently re-execute** `parity_test.py --mode side-by-side` and score its fresh output instead. `--rerun` is fail-closed: a missing/failing `parity_test.py`, a timeout, or output that is not in the scoreable `parity_experiment.json` shape all exit non-zero (rather than silently reporting `insufficient-evidence`). ```bash bench eval adopt my-bench --verify bench eval adopt my-bench --verify --tolerance 0.05 --issue-out divergence.md bench eval adopt my-bench --verify --roundtrip-task benchmarks/my-bench/tasks/example bench eval adopt my-bench --verify --rerun # re-run parity_test.py, score fresh output ``` | Flag | Default | Description | |------|---------|-------------| | `--benchmarks-dir` | repo `benchmarks/` | Target benchmarks/ directory | | `--tolerance` | `0.02` | Max abs reward delta (statistical layer) | | `--issue-out` | — | Write the divergence issue draft to this path instead of stdout | | `--roundtrip-task` | — | Also run the structural round-trip check on this task dir | | `--rerun` | `false` | Re-execute `parity_test.py --mode side-by-side` and score its fresh output instead of the recorded `parity_experiment.json` | ## bench eval ### bench eval run Run an evaluation — single task or batch. Use it for YAML configs and batch runs; it also accepts a single task directory. > **Renamed from `bench eval create`.** The old name still works as a deprecated > alias and prints a deprecation notice; switch to `bench eval run`. ```bash # From YAML config bench eval run --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml # One task on the default local Docker sandbox bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/citation-check \ --agent codex \ --model gpt-5.5 # From remote repo (fast Daytona batch; token usage may be unavailable) bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona \ --concurrency 64 \ --sandbox-setup-timeout 300 # From remote repo with required token usage telemetry bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona \ --usage-tracking required \ --concurrency 16 \ --sandbox-setup-timeout 300 # From local directory bench eval run --tasks-dir ./tasks --agent gemini --model gemini-3.1-flash-lite-preview # Emit reproducible training/eval artifacts and publish them to Hugging Face bench eval run \ --tasks-dir ./tasks \ --agent openhands \ --model openai/gpt-5.4-mini \ --sandbox daytona \ --task-manifest-out task-manifest.json \ --health-summary-out health.json \ --canonicalize one-healthy-per-task \ --canonical-selection-out canonical-selection.json \ --publish-hf benchflow/env0-experiment-trajectories \ --hf-prefix experiments/my-run # From a hosted PrimeIntellect / Verifiers environment bench eval run \ --source-env primeintellect/general-agent \ --source-env-version 0.1.1 \ --source-env-arg task=calendar_scheduling_t0 \ --agent gemini \ --model google/gemini-2.5-flash-lite # Single task with mounted skills bench eval run \ --tasks-dir tasks/pdf-fix \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox daytona \ --skill-mode with-skill # Pinned registry dataset: resolves skillsbench@1.1, verifies task digests, # and stamps dataset identity into every result.json/config.json bench eval run -d skillsbench@1.1 --agent gemini --model gemini-3.1-flash-lite-preview # Matrix eval over multiple models/trials bench eval run --tasks-dir ./tasks --matrix matrix.yaml --trials 3 ``` | Flag | Default | Description | |------|---------|-------------| | `--config` | — | YAML config file | | `--run-config` | — | Explicit alias for the YAML run-config source file; equivalent to `--config` | | `--tasks-dir` | — | Local task dir (single native `task.md` package, compatibility split-layout task, or parent of many) | | `-d`, `--dataset` | — | Registry dataset to run as `@` (e.g. `skillsbench@1.1`). Resolves the pinned snapshot from the registry, clones tasks at their pinned commit, verifies each task's sha256 content digest, and checks the dataset's `bench_version` range against the installed benchflow. Each `result.json`/`config.json` is stamped with `dataset_name`, `dataset_version`, and the task's `task_digest`. | | `--registry` | skillsbench registry | Dataset registry JSON URL or local file. Only valid with `--dataset`. | | `--source-repo` | — | Remote repo as `org/repo` (e.g. `benchflow-ai/skillsbench`) | | `--source-path` | — | Subpath within the repo (e.g. `tasks`) | | `--source-ref` | — | Branch or tag to clone (e.g. `main`) | | `--source-env` | — | Hosted environment source (e.g. `primeintellect/general-agent`) | | `--source-env-version` | — | Hosted environment version | | `--source-env-arg` | — | Hosted environment argument as `KEY=VALUE`; repeatable | | `--source-env-num-examples` | `1` | Number of hosted environment examples | | `--source-env-rollouts-per-example` | `1` | Rollouts per hosted environment example | | `--source-env-max-tokens` | `1024` | Max tokens for hosted environment model calls | | `--source-env-temperature` | `0.0` | Temperature for hosted environment model calls | | `--source-env-sampling-arg` | — | Verifiers sampling argument as `KEY=VALUE`; repeatable (for example `reasoning_effort=minimal`) | | `--agent` | `claude-agent-acp` | Agent name | | `--model` | Agent default | Model ID | | `--reasoning-effort` | — | Agent reasoning/thinking effort when the agent exposes one (e.g. `max`) | | `--sandbox` | `docker` | Sandbox: docker, daytona, modal, apple-container, or agentcore | | `--usage-tracking` | `auto` | Token usage telemetry policy: `auto`, `required`, or `off` | | `--environment-manifest` | — | Environment-plane manifest applied to every rollout in the batch: a path to an `environment.toml`, or a `name@version` registry spec resolved via `$BENCHFLOW_ENV_REGISTRY` when set, else the built-in registry shipped with benchflow (`env0@prod`, `env0@outage`; see [Environment plane: Registry](/docs/benchflow/environment-plane#registry-nameversion)). Overrides a task.md `benchflow.environment.manifest` pin | | `--state` | — | S-axis environment binding; inline JSON, registry `name@version`, or manifest path. Takes precedence over `--environment-manifest` | | `--prompt` | task prompt | Prompt to send to the agent; repeatable for multi-prompt runs | | `--config-override` | — | C-axis task config overlay; inline JSON/YAML/TOML or `@file`, deep-merged into each task's resolved config | | `--concurrency` | `4` | Max concurrent tasks (batch mode only) | | `--build-concurrency` | `--concurrency` | Max concurrent docker image builds; set lower (e.g. `8`) when `--concurrency` is high to avoid overwhelming the docker daemon | | `--worker-concurrency` | — | Run batch eval through isolated worker subprocesses, each with at most this many concurrent tasks; `--concurrency` remains the aggregate target | | `--worker-retries` | `1` | Retry a crashed worker shard this many times, resuming its jobs dir | | `--worker-start-stagger-sec` | `1.0` | Seconds to stagger worker starts to avoid Daytona connection storms | | `--agent-idle-timeout` | (built-in default) | Abort ACP prompts after this many idle seconds; `0` disables idle detection | | `--quiet` | off | Suppress live progress output: the Rich dashboard on a TTY and the per-run console progress heartbeat during agent execution | | `--jobs-dir` | `jobs` | Output directory | | `--sandbox-user` | `agent` | Sandbox user (null for root) | | `--sandbox-setup-timeout` | `120` | Timeout in seconds for sandbox user setup | | `--context-root` | — | Repo/build-context root used to stage Dockerfile `COPY` sources for monorepo-authored local tasks | | `--base-image-override` | — | Rewrite task Dockerfile `FROM` images on the runtime task copy; use for reproducing runs whose base image moved namespaces | | `--skills-dir` | — | Advanced custom skills directory; valid only with `--skill-mode with-skill`. Omit it to use each task's `environment/skills`. | | `--skill-mode` | `no-skill` | Skill mode: `no-skill`, `with-skill`, or `self-gen` | | `--skill-creator-dir` | — | Path to a `skill-creator` directory (or a skills root containing it); used when `--skill-mode self-gen` | | `--self-gen-no-internet` | `false` | Disable web tools for the self-generated skill run | | `--agent-env` | — | Agent environment variable as `KEY=VALUE`; repeatable | | `--include` | — | Only run these task names; repeatable (e.g. `--include jax-computing-basics --include data-to-d3`) | | `--exclude` | — | Skip these task names; repeatable (e.g. `--exclude quantum-numerical-simulation`) | | `--loop-strategy` | — | Wrap each rollout in a loop, e.g. `verify-retry:k=3,feedback=names` or `self-review:k=3` (omit for single-shot) | | `--ignore-bench-version` | `false` | With `--dataset`, skip the dataset's `bench_version` compatibility gate | | `--task-manifest-out` | — | Write selected task-set manifest JSON with task ids, paths, digests, and source provenance | | `--run-config-out` | — | Write a redacted normalized run config JSON | | `--health-summary-out` | — | Write trajectory health summary JSON for the completed job | | `--expected-tasks` | — | Fail unless the selected task count, and canonical selected count when used, matches this value | | `--canonicalize` | `none` | Canonicalization policy: `none` or `one-healthy-per-task` | | `--canonical-selection-out` | — | Write canonical rollout-selection JSON | | `--canonical-jobs-dir` | — | Materialize selected rollout directories for trainer conversion | | `--retry-policy` | `default` | Retry policy label for reproducible eval artifacts: `default` or `unscored-only` | | `--retry-attempts` | — | Override retry attempts for the eval run | | `--retry-concurrency` | — | Reserved retry concurrency setting recorded in run config | | `--publish-hf` | — | Upload final eval artifacts to this Hugging Face dataset repo | | `--hf-prefix` | — | Path prefix inside the Hugging Face repo; requires `--publish-hf` | | `--hf-public-read-check` | `false` | Verify public Hugging Face reads after upload | | `--matrix` | — | YAML model matrix for repeated evals; currently requires `--tasks-dir` | | `--trials` | `1` | Number of trials for `--matrix` | See [Architecture: skill loading](https://github.com/benchflow-ai/benchflow/blob/main/docs/architecture.md#skill-loading) for how `with-skill` mode is registered with each agent. While the agent works, a terminal (TTY) shows the live Rich dashboard — progress bar, pass/fail counts, and a per-task activity column that tracks tool calls/tokens and labels the non-agent stretches (`creating sandbox…`, `installing agent…`, `verifying…`); `BENCHFLOW_NO_PROGRESS=1` disables it. Plain output (CI, pipes) prints a console progress heartbeat instead: about every 45 seconds on single-concurrency runs (`… 6.2min, 12 tool calls (last: …)`), auto-gated off for multi-concurrency jobs. Setting `BENCHFLOW_PROGRESS=on`/`off` overrides the heartbeat auto-gate; `--quiet` is shorthand for setting both `BENCHFLOW_PROGRESS=off` and `BENCHFLOW_NO_PROGRESS=1` for the run, silencing dashboard and heartbeat alike (so it also wins over an exported `on`). Note that on a TTY, `BENCHFLOW_PROGRESS=on` alone produces no heartbeat lines — the dashboard mutes INFO logging while it owns the screen; pair it with `BENCHFLOW_NO_PROGRESS=1` to get plain heartbeat lines on a TTY. The dashboard footer also carries a live token total: completed tasks' trusted telemetry plus every running rollout's live usage (ACP session counters reconciled with the sandbox gateway's live capture), so spend is visible mid-run. The live figure is a lower bound — it trails the gateway log by however much the capture has yet to read — and if that tail ever stops advancing altogether, the run logs one `Live token counter has stalled` warning so a stale number is never passed off as a current one. Cost stays completed-tasks-only — `$` comes from the gateway log imported at scoring time. After the run, each failed task gets one dim `✗ task: reason` line — verifier error first, else a compact reward/metric breakdown, else the scored reward, upgraded from small on-disk verifier artifacts (the CTRF report, `reward.json`, or a `test-stdout.txt` tail) when the in-memory reason is a bare reward. Multi-failure CTRF reports roll up as `(+N more failure(s); P/T checks passed)`, and a dim `(details: …/verifier)` pointer names the artifact directory whenever one exists on disk. The final `Score: P/T (…%)` line is pass-threshold aggregation — a task counts as passed only at reward 1.0 — while `mean reward` beside it is the average raw verifier reward, so `0/1 (0.0%)` next to `mean reward 0.80` means partial credit below the pass threshold, not a flat zero. Set `BENCHFLOW_ACP_HANDSHAKE_TIMEOUT` to a number of seconds (default 60) to give slow-starting agents more time to answer the pre-prompt ACP handshake (`initialize`/`session_new`) — heavyweight task images can push agent startup past the default. Daytona batch runs collect provider token/cost telemetry by default with a sandbox-local LiteLLM gateway. Use `--usage-tracking required` when missing telemetry should fail the rollout, or `--usage-tracking off` for recovery runs that should leave provider traffic untouched. For online-training rollouts against a chat-completions endpoint that supports sampled-token log probabilities, pass `--agent-env BENCHFLOW_CAPTURE_TOKEN_LOGPROBS=1`. The LiteLLM gateway adds `logprobs=true` to each chat request and preserves the provider's token logprobs in `trajectory/llm_trajectory.jsonl`. This is opt-in because providers that do not implement chat-completion logprobs may reject the request. `--source-env` is for external hosted environment hubs. The first supported runner is PrimeIntellect / Verifiers: BenchFlow preserves the hosted identity (`env_uid`, `hub_url`), installs the versioned package into an isolated local virtual environment, and runs `vf-eval`. `--sandbox` remains the BenchFlow task sandbox selector for local/repo task sources; Verifiers source environments own their own harness and sandbox behavior. `--model` is passed to the Verifiers model endpoint; use a model id available to that provider. Provider-specific sampling options are not inferred; pass them explicitly with `--source-env-sampling-arg`. ## bench review Grade finished rollouts against a rubric with a reviewer agent. Reviews run detached from the rollouts they grade: each review is an ordinary sandboxed rollout of a throwaway wrapper task built on a prebuilt image, evidence is a read-only copy, and results land in `review_report.json`. Reviewed rollouts' rewards and `result.json` are never modified. ```bash bench review jobs/2026-08-03__12-00-00 --sandbox docker -m gemini/gemini-2.5-flash bench review jobs// -r my-rubric.json --agent gemini bench review jobs/ --passing --sandbox daytona -n 8 -m gemini/gemini-2.5-flash ``` The default `opencode` reviewer has no registry default model, so `-m` is required with it (a run without one exits with an actionable error). | Flag | Default | Description | |---|---|---| | `--rubric`, `-r` | task / built-in | Rubric JSON file. Default: an admitted task copy's `verifier/rubric.json` (requires `--tasks-root` and a verified recorded digest), else the built-in default rubric | | `--prompt`, `-p` | built-in | Custom reviewer instruction template | | `--agent`, `-a` | `opencode` | Reviewer agent harness | | `--model`, `-m` | agent registry | Reviewer model (required for agents without a registry default; gateway ids such as `gemini/gemini-2.5-flash`) | | `--sandbox` | `docker` | Sandbox backend for reviewer rollouts | | `--concurrency`, `-n` | `4` | Max concurrent reviews | | `--passing` | `false` | Only review passing rollouts (reward 1.0) | | `--failing` | `false` | Only review failing rollouts | | `--timeout-sec` | `1800` | Reviewer agent timeout per rollout | | `--agent-env` | — | `KEY=VALUE` for the reviewer (repeatable) | | `--image` | digest-pinned `python` slim | Prebuilt sandbox image for reviewer rollouts (default is pinned by digest; a tag override is mutable) | | `--tasks-root` | — | Trusted directory holding reviewed tasks; required to include task definitions in evidence (a rollout-recorded path is untrusted and never read directly) | | `--allow-open-network` | `false` | Run reviewers without the no-internet declaration (required on backends that cannot enforce isolation, e.g. agentcore; recorded in the report) | | `--out-dir`, `-o` | `jobs/review-` | Review output directory | A rubric is a JSON object with one `criteria` list; each criterion is three strings — `name` (identifier; becomes a structured-output field), `description` (author-facing documentation, never shown to the reviewer), and `guidance` (the grading contract the reviewer follows). The reviewer answers each criterion with `pass` / `fail` / `not_applicable` plus an explanation. ### bench eval list List completed evaluations from a jobs directory. ```bash bench eval list jobs/ ``` ### bench eval metrics Collect and display metrics (pass/fail/score, memory score, tool calls, duration) from a jobs directory. Use `--json` for machine-readable output. ```bash bench eval metrics jobs/ bench eval metrics jobs/ --json ``` ### bench eval view Serve a trial trajectory viewer in the browser for a rollout or job directory. ```bash bench eval view jobs/run/task__abc123 bench eval view jobs/ --port 9000 ``` ## bench train Convert scored BenchFlow rollouts into trainer-ready datasets and validate trainer rows before handing them to a training framework. ### bench train convert Convert a rollout directory, jobs directory, canonical BenchFlow `results.jsonl`, or existing trainer JSONL into a trainer-specific dataset. The default `prime-sft` format writes OpenAI-compatible `messages` plus `tool_defs`. The `trl-sft` format writes conversational `prompt` and `completion` lists plus a `tools` column. ```bash bench train convert jobs/run-001 --out train.jsonl bench train convert jobs/run-001 --out train.jsonl --min-reward 1.0 bench train convert jobs/run-001 --out train.jsonl --canonical-selection canonical-selection.json bench train convert jobs/run-001 \ --format trl-sft \ --row-mode exchange \ --min-reward 1.0 \ --context-policy message-window \ --tokenizer Qwen/Qwen3-4B \ --tokenizer-revision \ --max-length 40960 \ --out train.trl.jsonl \ --manifest train.trl.manifest.json ``` `results.jsonl` remains the canonical scored-rollout artifact regardless of trainer. The selected format changes only the converted output. For TRL, `exchange` mode emits one supervised completion for every primary agent model call while excluding captured OpenCode title, summary, compaction, and helper calls. `rollout` mode emits only the final primary model call. TRL conversion never truncates implicitly. The default `full` context policy preserves every captured message. `message-window` first renders with the pinned tokenizer; when a row is too long it preserves all leading system messages, the original task user message, the target assistant completion, and the longest complete recent suffix of assistant/tool groups that fits. It records original/final token counts and every dropped-message count in both the row and conversion manifest. It fails if the required prefix and completion cannot fit. | Flag | Default | Description | |------|---------|-------------| | `--out`, `-o` | required | Output JSONL path | | `--format` | `prime-sft` | Trainer format: `prime-sft` or `trl-sft` | | `--min-reward` | — | Only include rows with reward greater than or equal to this value | | `--row-mode` | `rollout` | `rollout` writes one row per rollout; `exchange` writes one row per LLM exchange | | `--manifest` | — | Optional conversion stats JSON path | | `--expected-rows` | — | Fail before writing unless exactly this many rows would be exported | | `--canonical-selection` | — | Restrict conversion to rows selected by `canonical-selection.json` | | `--context-policy` | `full` | TRL context policy: exact `full` rows or tokenizer-aware `message-window` | | `--tokenizer` | — | Tokenizer/model ID required by `message-window` | | `--tokenizer-revision` | — | Immutable tokenizer revision for context windowing | | `--max-length` | — | Maximum rendered length required by `message-window` | ### bench train validate Validate Prime-RL or TRL SFT JSONL before upload or training. Both formats fail closed on malformed tool calls, undeclared tools, orphan tool outputs, and row count mismatches. TRL validation additionally requires object-valued tool-call arguments and exactly one assistant message in each completion. ```bash bench train validate train.jsonl bench train validate train.jsonl --expected-rows 4417 bench train validate train.jsonl \ --source-jobs jobs/run-001 \ --require-llm-trajectory \ --require-tool-calls bench train validate train.trl.jsonl \ --format trl-sft \ --source-jobs jobs/run-001 \ --require-llm-trajectory \ --require-tool-calls \ --tokenizer Qwen/Qwen3-4B \ --tokenizer-revision \ --max-length 40960 ``` When `--tokenizer` is set, TRL validation uses TRL's training chat template, checks that prompt tokenization remains a prefix of prompt-plus-completion, requires a non-empty assistant token mask after the prompt boundary, and fails instead of silently truncating a row beyond `--max-length`. The JSON report includes token-length distribution and minimum trainable assistant tokens. | Flag | Default | Description | |------|---------|-------------| | `--format` | `prime-sft` | Trainer format: `prime-sft` or `trl-sft` | | `--expected-rows` | — | Fail unless this many rows are present | | `--source-jobs` | — | Source BenchFlow jobs directory to audit alongside trainer JSONL | | `--source-canonical-selection` | — | Canonical selection JSON used for this trainer data | | `--task-manifest` | — | Task manifest for source rows | | `--require-llm-trajectory` | `false` | Fail unless source selected rows have valid `llm_trajectory.jsonl` | | `--require-tool-calls` | `false` | Fail unless trainer rows and source rows include tool calls | | `--tokenizer` | — | Tokenizer/model ID used to render and mask TRL rows | | `--tokenizer-revision` | — | Immutable tokenizer revision used for TRL validation | | `--max-length` | — | Fail when a rendered TRL row exceeds this token length | ### bench train run sft Launch a supervised fine-tuning job and record BenchFlow launch metadata. The first supported backend is `prime-rl`; BenchFlow wraps the native Prime-RL SFT entrypoint instead of re-modeling trainer internals. ```bash bench train run sft \ --backend prime-rl \ --config configs/qwen35-env0-sft.toml \ --data benchflow/env0-prime-sft \ --prime-rl-dir .local/prime-rl \ --work-dir train-runs/qwen35-env0-sft \ --publish-model benchflow/benchflow-qwen35-9b \ --publish-artifacts benchflow/env0-experiment-trajectories \ --hf-prefix experiments/env0-mobile-pr828/training \ --follow ``` The wrapper runs: ```bash uv run sft @ configs/qwen35-env0-sft.toml \ --data.name benchflow/env0-prime-sft \ --output-dir train-runs/qwen35-env0-sft/prime-rl-output ``` BenchFlow writes `/train-run.json`, `/command.txt`, and separate Prime-RL stdout/stderr logs under `/prime-rl/`. Secrets are not written to the manifest; only the names of recognized credential env vars that were present are recorded. For the Mobile300 PR828 reproduction, use `--compat-profile env0-mobile300-pr828`. That profile stages the historical custom-trainer pretokenized shifted-label rows, bypasses Prime-RL `stack`/`cat` packing for those staged rows so training sees one original trajectory per micro-batch, and enables `sample_mean` loss normalization through a run-local `sitecustomize.py` shim. The shim leaves Prime-RL package files untouched but fails closed if the Prime-RL SFT train loop or data module no longer exposes the expected hooks. | Flag | Default | Description | |------|---------|-------------| | `--backend` | `prime-rl` | Training backend. Currently only `prime-rl` is supported | | `--config` | required | Prime-RL SFT TOML config. Relative paths are resolved from the current directory first, then from `--prime-rl-dir` when set | | `--data` | — | Optional dataset override passed through as `--data.name` | | `--output-dir` | `/prime-rl-output` | Prime-RL trainer output directory | | `--compat-profile` | — | Named BenchFlow Prime-RL SFT compatibility profile. `env0-mobile300-pr828` expands to the Mobile300 PR828 reproduction settings | | `--work-dir` | `train-runs/sft` | BenchFlow training run directory | | `--prime-rl-dir` | current directory | Prime-RL checkout to run `uv run sft` from | | `--dry-run` | `false` | Pass `--dry-run` through to Prime-RL | | `--follow` | `false` | Stream trainer stdout while writing logs | | `--uv-no-sync` | `false` | Run Prime-RL as `uv run --no-sync sft ...`, useful after backend post-install steps such as `flash-attn` | | `--override` | — | Prime-RL override as `KEY=VALUE`; repeatable, emitted as `--KEY VALUE` | | `--target-examples` | — | Derive Prime-RL `max_steps` from target sample exposure and effective `data.batch_size`, rounding up | | `--target-micro-steps` | — | Derive Prime-RL `max_steps` from custom-trainer batch-size-1 microsteps, dropping the final partial accumulation | | `--sync-scheduler-to-max-steps` / `--no-sync-scheduler-to-max-steps` | `true` | When `--target-examples` or `--target-micro-steps` is set, also derive `scheduler.decay_steps` | | `--sync-ckpt-to-max-steps` / `--no-sync-ckpt-to-max-steps` | `false` | When deriving `max_steps`, also derive `ckpt.interval` and `ckpt.keep_interval` | | `--pack-function` | — | First-class Prime-RL `data.pack_function` override: `cat` or `stack` | | `--loss-mask` | — | First-class Prime-RL `data.loss_mask` override: `assistant`, `all`, or comma-separated roles from `system,user,assistant,tool` | | `--loss-normalization` | — | Prime-RL SFT loss normalization. `token_mean` keeps native Prime-RL behavior; `sample_mean` launches a run-local compatibility shim that matches the historical custom trainer's per-row mean loss and requires `data.pack_function=stack` | | `--model-attn` | — | First-class Prime-RL `model.attn` override, e.g. `sdpa` | | `--renderer-mode` | — | Prime-RL renderer override. `none` emits `--renderer None`, making Prime-RL use tokenizer `apply_chat_template` tokenization | | `--tool-defs-mode` | `preserve` | For local JSONL or local dataset dirs, keep tool schemas (`preserve`) or remove `tool_defs`/`tools` from the temporary training copy (`omit`) | | `--allow-unsafe-stack-flash-attn` | `false` | Allow Qwen3.5 `stack` packing with flash attention despite the known Prime-RL varlen-kernel risk | | `--force` | `false` | Overwrite an existing `/train-run.json` manifest | | `--publish-model` | — | Upload trainer output to this Hugging Face model repo | | `--model-tag` | — | Path prefix/tag for the model upload | | `--model-card` | — | Model card mode; currently accepts `auto` | | `--publish-artifacts` | — | Upload BenchFlow train run artifacts to this Hugging Face dataset repo | | `--hf-prefix` | — | Path prefix for `--publish-artifacts` | | `--hf-public-read-check` | `false` | Verify public Hugging Face reads after upload | Local JSONL files are packaged automatically into a temporary Hugging Face dataset directory under `/prime-rl-dataset`, with source validation metadata recorded in the manifest. If `--tool-defs-mode omit` is set, BenchFlow validates the source JSONL first and then strips tool schema columns only from the temporary training copy. ## bench skills ### bench skills list List skills discovered under the default skills roots (or `--dir`). ```bash bench skills list bench skills list --dir ./skills ``` ### bench skills eval Evaluate a skill against its evals.json test cases. ```bash bench skills eval skills/my-skill/ \ --agent gemini \ --model gemini-3.1-flash-lite-preview \ --sandbox docker ``` --- ## bench tasks ### bench tasks init Scaffold a new benchmark task. ```bash bench tasks init my-new-task bench tasks init my-new-task --dir tasks/ ``` | Flag | Default | Description | |------|---------|-------------| | `--format` | `task-md` | Task format. New tasks use `task-md`; the legacy scaffold path is retired. | ### bench tasks check Validate a task directory. Native packages use `task.md`, `environment/`, and `verifier/`; older split packages should be migrated with `bench tasks migrate`. ```bash bench tasks check tasks/my-task ``` With `--level`, validation runs at a chosen depth: `schema`, `structural`, `runtime-capability`, `publication-grade`, `acceptance`, or `acceptance-live`. Acceptance-level errors such as `acceptance validation requires benchflow.evidence mapping` refer to the `benchflow.evidence` schema documented in the "Assets, Provenance, And Evidence" section of `docs/task-standard.md`. ### bench tasks migrate Convert an older split task package into the unified `task.md` format. By default the old files are kept alongside the new `task.md`; for publication, use `--remove-legacy`. ```bash bench tasks migrate tasks/my-task bench tasks migrate tasks/my-task --overwrite --remove-legacy ``` | Flag | Default | Description | |------|---------|-------------| | `--overwrite` | `false` | Replace an existing task.md | | `--remove-legacy` | `false` | Delete split files and promote `tests/` to `verifier/` and `solution/` to `oracle/` after `task.md` is verified | ### bench tasks normalize Expand minimal `task.md` authoring profiles into the canonical `task.md` form. Prints the normalized document to stdout unless told otherwise. ```bash bench tasks normalize tasks/my-task bench tasks normalize tasks/my-task --write bench tasks normalize tasks/my-task -o normalized-task.md ``` | Flag | Default | Description | |------|---------|-------------| | `--output`, `-o` | — | Write normalized task.md to this path instead of stdout | | `--write` | `false` | Replace task.md in place with the normalized canonical form | ### bench tasks export Export a `task.md` task to a compatibility split package, with a compatibility loss report written to `compatibility/export-report.json` in the export directory. ```bash bench tasks export tasks/my-task out/my-task-split bench tasks export tasks/my-task --report-only bench tasks export tasks/my-task out/my-task-split --overwrite ``` Arguments: `TASK_DIR` (task directory to export) and optional `OUTPUT_DIR` (destination split-layout directory; may be omitted with `--report-only`). | Flag | Default | Description | |------|---------|-------------| | `--target` | `harbor` | Compatibility target: `harbor` | | `--overwrite` | `false` | Replace an existing export directory | | `--report-only` | `false` | Print the compatibility loss report without writing files | ### bench tasks snapshot-hf Materialize a Hugging Face dataset repo or subpath as a local BenchFlow task tree and write `.benchflow-source.json` provenance beside it. The resulting directory can be passed to `bench eval run --tasks-dir`; split-layout task snapshots under `tasks//` are discovered directly. ```bash bench tasks snapshot-hf benchflow/my-tasks .cache/hf-tasks/my-tasks bench tasks snapshot-hf benchflow/my-tasks .cache/hf-tasks/my-tasks --revision abc123 --path tasks --overwrite ``` Arguments: `REPO_ID` (Hugging Face dataset repo ID) and `OUTPUT_DIR`. | Flag | Default | Description | |------|---------|-------------| | `--revision`, `--ref` | — | Dataset revision, branch, tag, or commit | | `--path` | — | Optional subpath inside the dataset repo, e.g. `tasks` | | `--cache-dir` | HF default | Optional Hugging Face cache directory | | `--overwrite` | `false` | Replace an existing output directory | ### bench tasks digest Compute the content digest that pins a task's files, independent of git — the sha256 the dataset registry keys on (matches the digests `bench eval run -d` verifies and the `task_digest` stamped into every `result.json`). Recognizes both legacy `task.toml` tasks and native `task.md` tasks. Given a single task directory it prints the digest; given a directory of tasks it prints one ` ` line per task. Output goes to stdout via `echo` (not Rich), so it is safe to pipe into machine-readable tooling. ```bash bench tasks digest tasks/my-task # -> sha256: bench tasks digest tasks/ # one " sha256:" line per task ``` Arguments: `PATH` (a task directory, or a directory of task directories). ### bench tasks overlap Compare two task manifests, typically one emitted by a training-data collection run and one emitted by an evaluation run. ```bash bench tasks overlap train-task-manifest.json eval-task-manifest.json bench tasks overlap train-task-manifest.json eval-task-manifest.json --out overlap.json ``` The command reports exact task-id overlap and exact digest overlap. A zero overlap result means the task ids/digests are disjoint; it does not prove domain or generator-family disjointness. | Flag | Default | Description | |------|---------|-------------| | `--out`, `-o` | — | Optional JSON output path | ### bench tasks generate Generate benchmark task directories from real agent traces. ```bash bench tasks generate --from-local --project my-repo --limit 5 bench tasks generate --from-file session.jsonl --dry-run bench tasks generate --from-hf opentraces-test --limit 50 ``` | Flag | Default | Description | |------|---------|-------------| | `--from-local` | — | Generate from local Claude Code sessions | | `--from-file` | — | Generate from a JSONL trace file | | `--from-hf` | — | Generate from a HuggingFace dataset ID or alias | | `--output` | `tasks` | Output directory for generated tasks | | `--projects-dir` | `~/.claude/projects/` | Claude Code projects directory | | `--project` | — | Filter local sessions by project path substring | | `--format` | `auto` | Trace format override | | `--split` | `train` | HuggingFace dataset split | | `--max-rows` | `100` | Max rows to download from HuggingFace | | `--limit` | `20` | Max traces to process | | `--min-steps` | `2` | Minimum steps per trace | | `--outcome` | — | Filter by outcome: success, failure, unknown | | `--author` | `benchflow-traces` | Author name for generated task metadata | | `--task-format` | `task-md` | Generated task package format: `task-md` or `legacy` | | `--dry-run` | `false` | Preview traces without generating tasks | ### bench tasks list-sources List known HuggingFace trace datasets. The aliases listed here can be passed to `bench tasks generate --from-hf`. ```bash bench tasks list-sources ``` ## bench sandbox Local sandbox lifecycle: provision a task on a docker/daytona/modal backend, list active sandboxes, and reap stale ones. ### bench sandbox create Create an environment object from a task directory. This validates environment construction but does not start the sandbox. ```bash bench sandbox create tasks/my-task --sandbox daytona ``` ### bench sandbox list List active local (Daytona) sandboxes. ```bash bench sandbox list ``` ### bench sandbox cleanup Clean up orphaned Daytona sandboxes. By default this deletes sandboxes older than 24 hours; use `--dry-run` to preview what would be deleted. ```bash bench sandbox cleanup --dry-run --max-age 1440 ``` Daytona-backed evals also reap orphaned sandboxes automatically at run start (failure states such as `BUILD_FAILED` are reaped sooner than healthy ones, and an idle-activity guard means concurrent live runs are never reaped). Set `BENCHFLOW_DAYTONA_AUTO_REAP` to any of `0`/`false`/`no`/`off` (case-insensitive) to disable that automatic pass and rely on the manual command above. Every rollout attempt also runs under a host-side hard deadline computed from the task's own phase budgets — a backstop for awaits wedged below the phase-level timeouts (a tripped deadline abandons the sandbox to the provider's reaper). Set `BENCHFLOW_ROLLOUT_HARD_DEADLINE` to a number of seconds to override the computed value, or to `off`/`none`/`0` to disable the backstop. ## bench environment (deprecated) `bench environment` is a hidden **deprecated alias group**, removed in 0.7. The local lifecycle moved to [`bench sandbox`](#bench-sandbox) (`create`/`list`/`cleanup`) and hosted-provider browsing to [`bench hub list`](#bench-hub). The old `bench environment create|list|cleanup` and `show|inspect` (plus `list --provider`/`--hub`) still work, each printing a one-line stderr notice. ## bench traj upload Validate, redact, and contribute trajectory JSONL through BenchFlow's public broker. `PATH` can be one JSONL file, a directory of JSONL files, or a trial directory containing `trajectory/`. The command stages only JSONL artifacts, writes a content-addressed manifest last, and treats an already-ingested digest as a successful no-op. ```bash bench traj upload path/to/trial --github-id octocat --email octocat@example.com bench traj upload path/to/trajectory.jsonl --github-id octocat \ --email octocat@example.com --source-id my-project/run-42 bench traj upload path/to/trial --github-id octocat \ --email octocat@example.com --dry-run ``` | Flag | Default | Description | |------|---------|-------------| | `--github-id` | required | Self-asserted GitHub username stored in `manifest.json` | | `--email` | required | Contributor email stored in `manifest.json`; not printed by the CLI | | `--source-id` | derived from `PATH` | Stable contributor/run label stored in the manifest | | `--dry-run` | `false` | Validate, redact, hash, and list staged files without network traffic | | `--direct` | `false` | Use local Azure credentials instead of the public broker; requires the `azure` extra | | `--container-url` | — | Azure Blob container URL for `--direct`; alternatively set `BENCHFLOW_AZURE_CONTAINER_URL` | See [Trajectory upload](/docs/benchflow/traj-upload) for privacy and operator details. ## bench hub External environment hubs: browse a hub's environments (`list`/`show`/`inspect`) and check Harbor registry compatibility (`check`). ### bench hub list / show / inspect Read-only browsing of a hub's environments. `list` covers two hubs via `--provider`: `primeintellect` (hosted "Environments") and `harbor` (the benchmark registry). To *run* a hosted environment, use [`bench eval run --source-env`](#bench-eval-run). ```bash bench hub list --provider primeintellect --owner primeintellect --search general-agent --limit 5 bench hub list --provider harbor --search coding bench hub show primeintellect/general-agent --version 0.1.1 bench hub inspect primeintellect/general-agent --version 0.1.1 --path README.md ``` `bench hub env list|show|inspect` still resolves as a hidden back-compat alias. ### bench hub check Inventory or structurally check representative tasks from an environment hub's registry. Defaults to an inventory pass against the public Harbor registry JSON. ```bash # Inventory the public Harbor hub registry bench hub check # Structural check, two tasks per dataset, JSONL output bench hub check --level check --tasks-per-dataset 2 --out hub.jsonl ``` | Flag | Default | Description | |------|---------|-------------| | `--registry` | Harbor public registry URL | Harbor registry JSON URL or local file | | `--tasks-per-dataset` | `2` | Representative tasks selected per dataset | | `--level` | `inventory` | Compatibility level: `inventory` or `check` | | `--out` | — | Optional JSONL output path | | `--cache-dir` | `.cache/hub/harbor` | Cache directory for sparse clones | | `--limit` | — | Optional cap on selected task refs | ## YAML Config Format ### Batch config with skills ```yaml source: repo: benchflow-ai/skillsbench path: tasks environment: docker concurrency: 2 sandbox_setup_timeout: 300 agent: gemini model: gemini-3.1-flash-lite-preview skill_mode: with-skill skills_dir: shared-skills/ max_retries: 2 ``` ### Multi-scene (BYOS skill generation) Use the Python API for multi-scene experiments. `bench eval run --config` is for batch job configs; scene configs are loaded with `benchflow._utils.yaml_loader` or built directly in Python. ```yaml task_dir: tasks/my-task environment: docker sandbox_setup_timeout: 300 scenes: - name: skill-gen roles: - name: creator agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: creator prompt: "Analyze the task and write a skill document to /app/generated-skill.md" - name: solve roles: - name: solver agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: solver ``` --- ## bench eval continue Resume a previous, unfinished (timed-out) `openhands` run to completion via record-replay. Standalone — it does not touch the normal run path. See [Continuing timed-out runs](/docs/benchflow/continue-runs) for the full guide. ```bash bench eval continue path/to/original/run-folder --tasks-dir path/to/tasks ``` The original top-level `bench continue` still works as a hidden, deprecated alias. Key options: `--model` (override the live-continuation model; defaults to the original run's model), `--timeout`, `--output`, `--require-timeout`, `--strict-divergence`, `--replay-only` (rebuild via replay and stop at the cut-point — no live model or API key needed), and `--proxy-mode` (replay proxy placement: `auto`, `host`, or `sandbox`; default `auto` uses sandbox-local replay for Daytona/Modal and host replay for Docker). ### bench eval continue-batch Continue all timed-out OpenHands runs found under a directory tree. Discovers run folders (`config.json` + `trajectory/llm_trajectory.jsonl`) recursively, continues each, and prints a JSON batch summary (exits 1 if any continuation failed). ```bash bench eval continue-batch path/to/jobs-root --tasks-dir path/to/tasks ``` | Flag | Default | Description | |------|---------|-------------| | `--tasks-dir` | — | Directory holding task sources; required unless the recorded task path exists | | `--model` | original run's model | Override the live-continuation model | | `--timeout` | — | Wall-clock budget per continuation | | `--output` | — | Output jobs dir for continued runs | | `--concurrency` | `100` | Maximum number of continuation runs in flight | | `--limit` | — | Limit discovered timeout folders | | `--strict-divergence` | `false` | Abort a run if replay leaves the original rails | | `--proxy-mode` | `auto` | Replay proxy placement: `auto`, `host`, or `sandbox` | --- ## /docs/benchflow/reference/python-api The Rollout/Scene API is the primary way to run agent benchmarks programmatically. ## Install ```bash uv tool install --python 3.12 --upgrade benchflow ``` BenchFlow requires Python 3.12 or newer. For CLI installs, keep the `--python 3.12` flag so `uv` provisions a compatible tool environment. ## Quick Start ```python import asyncio import benchflow as bf result = asyncio.run(bf.run("gemini", task_path="tasks/my-task", model="gemini-3.1-flash-lite-preview")) print(f"Reward: {result.rewards}") print(f"Tool calls: {result.n_tool_calls}") ``` ## Core Types ### RolloutConfig Declarative configuration for a rollout — a sequence of Scenes in a shared sandbox. ```python from pathlib import Path from benchflow import RolloutConfig, Scene, Role, Turn # Single-agent (simplest) config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], environment="docker", sandbox_setup_timeout=120, ) # Multi-scene BYOS (skill-gen → solve) config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="prep", roles=[Role("gen", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("gen", "Generate a skill for this task...")]), Scene(name="solve", roles=[Role("solver", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("solver")]), ], environment="docker", sandbox_setup_timeout=120, ) ``` Set `sandbox_setup_timeout` when sandbox user setup needs more than the default 120 seconds. The same field is also available on `JobConfig` and `RuntimeConfig`. ### Scene Authoring sugar for role, prompt, and skill attribution. Scenes compile to explicit rollout Steps before execution; there is no runtime Scene object or message scheduler. ```python # Single-role shortcut scene = Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview") # Multi-role with explicit turn order scene = Scene( name="coder-reviewer", roles=[ Role("coder", "gemini", "gemini-3.1-flash-lite-preview"), Role("reviewer", "gemini", "gemini-3.1-flash-lite-preview"), ], turns=[ Turn("coder"), # None prompt = native task goal Turn("reviewer", "Review the current workspace."), Turn("coder", "Fix the issues."), ], ) ``` ### Rollout The execution engine — decomposed into independently-callable phases. ```python from benchflow import Rollout rollout = await Rollout.create(config) # Full lifecycle (most common) result = await rollout.run() # Manual composition (for custom flows) await rollout.setup() await rollout.start() await rollout.install_agent() await rollout.connect() await rollout.execute(prompts=["custom prompt"]) await rollout.disconnect() await rollout.verify() await rollout.cleanup() ``` ### RuntimeConfig Runtime-level configuration for the `Agent + Environment` execution path. ```python from benchflow.runtime import Agent, Environment, Runtime, RuntimeConfig config = RuntimeConfig(sandbox_setup_timeout=300) agent = Agent("gemini", model="gemini-3.1-flash-lite-preview") env = Environment.from_task("tasks/X", sandbox="docker") runtime = Runtime(env, agent, config=config) result = await runtime.execute() ``` ### bf.run() Convenience function — multiple calling conventions: ```python import benchflow as bf # 1. RolloutConfig (full control) result = await bf.run(config) # 2. Agent + Environment (0.3 style) agent = bf.Agent("gemini", model="gemini-3.1-flash-lite-preview") env = bf.Environment.from_task("tasks/X", sandbox="docker") runtime_config = bf.RuntimeConfig(sandbox_setup_timeout=300) result = await bf.run(agent, env, runtime_config) # 3. String shortcut (simplest) result = await bf.run( "gemini", task_path="tasks/X", model="gemini-3.1-flash-lite-preview", config=bf.RuntimeConfig(sandbox_setup_timeout=300), ) ``` ## Rollout Lifecycle ``` Rollout.run() │ ├─ setup() — resolve config, create env object ├─ start() — spin up sandbox, upload task files, start services ├─ install_agent() — install agent binary, credentials, sandbox user │ (sandbox user setup: create non-root user, prepare │ small config/auth dirs, chown the workspace — no │ recursive copy of /root tool trees; agent binaries │ must live on shared prefixes like /usr/local/bin) ├─ compile scenes → Steps ├─ for step in steps: │ ├─ connect_as(role) — open/reuse ACP session for this role │ └─ execute(prompt) — send prompt, collect trajectory, grow tree ├─ verify() — run verifier, collect rewards └─ cleanup() — stop sandbox ``` Key: scene boundaries are gone by execution time; role changes are represented as Step metadata and handled by the rollout executor. ## Multi-Turn vs Multi-Round | Pattern | Roles | Turns | Communication | Example | |---------|-------|-------|---------------|---------| | **Single-turn** | 1 | 1 | — | Baseline benchmark | | **Multi-turn** | 1 | 2+ | Same session, sequential prompts | Self-review | | **Multi-role** | 2+ | 2+ | Explicit prompt sequence | Coder + Reviewer | **Multi-turn** = same agent gets multiple prompts. Use when a second pass catches errors (self-review, iterative refinement). The agent keeps its context across turns. **Multi-role** = different agents receive explicit turns. Use when tasks need multiple perspectives (code review, client-advisor). Any handoff text must be part of the declared prompt or agent-native communication, not a BenchFlow Scene scheduler. Both use the same API — `RolloutConfig` with different `Scene` configurations. ## Multi-Agent Patterns ### Coder + Reviewer (followup-bench) ```python config = RolloutConfig( task_path=task_path, scenes=[Scene( roles=[Role("coder", "gemini", "flash"), Role("reviewer", "gemini", "flash")], turns=[ Turn("coder"), Turn("reviewer", "Review /app/. Summarize any issues."), Turn("coder", "Read feedback and fix."), ], )], environment="docker", ) ``` ### Skill Generation + Solve (BYOS) ```python config = RolloutConfig( task_path=task_path, scenes=[ Scene(name="skill-gen", roles=[Role("gen", "gemini", "flash")], turns=[Turn("gen", "Generate a skill document to /app/generated-skill.md")]), Scene(name="solve", roles=[Role("solver", "gemini", "flash")], turns=[Turn("solver")]), ], environment="docker", ) ``` ## User-Driven Loops Use `BaseUser` or `FunctionUser` when one agent should run multiple rounds and Python should decide the next prompt from verifier feedback. This is the progressive-disclosure path: the user callback can stop early, read `RoundResult` after each `soft_verify()`, and optionally receive the oracle solution during `setup()` when `oracle_access=True`. ```python from pathlib import Path from benchflow import FunctionUser, RolloutConfig, RoundResult, Scene def user(round: int, instruction: str, rr: RoundResult | None) -> str | None: if round == 0: return instruction.splitlines()[0] if rr and (rr.rewards or {}).get("reward") == 1.0: return None return f"Tests failed:\n{rr.verifier_output}\n\nUse the full spec:\n{instruction}" config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], user=FunctionUser(user), max_user_rounds=3, environment="docker", ) result = await bf.run(config) ``` Use multi-role Scenes when another LLM should act as the reviewer or simulated user. Use `BaseUser` when the loop is deterministic or verifier-driven. See [`progressive-disclosure.md`](/docs/benchflow/progressive-disclosure) and [`docs/examples/scene-patterns.ipynb`](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/scene-patterns.ipynb). ## YAML Rollout Configs ```python from benchflow._utils.yaml_loader import rollout_config_from_yaml config = rollout_config_from_yaml("rollout.yaml") result = await bf.run(config) ``` ## Registered Agents | Agent | Protocol | Auth | Aliases | |-------|----------|------|---------| | `gemini` | ACP | GEMINI_API_KEY | — | | `claude-agent-acp` | ACP | ANTHROPIC_API_KEY | `claude` | | `codex-acp` | ACP | OPENAI_API_KEY, CODEX_API_KEY, CODEX_ACCESS_TOKEN, or host login | `codex` | | `opencode` | ACP | inferred from model/provider | — | | `openhands` | ACP | LLM_API_KEY | `oh` | | `pi-acp` | ACP | ANTHROPIC_API_KEY | `pi` | | `openclaw` | ACP | inferred from model | — | The Auth column shows each agent's native/default credentials. Provider-prefixed models can use provider-specific credentials instead; for example, Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT` with prefixes such as `azure-foundry-openai/gpt-5.5` or `azure-foundry-anthropic/claude-opus-4-5`. BenchFlow routes these providers through LiteLLM on both Docker and Daytona. Any agent can be prefixed with `acpx/` to run via [ACPX](https://acpx.sh/) (e.g. `acpx/gemini`, `acpx/claude`). ACPX is a headless ACP client with persistent sessions and crash recovery. The underlying agent's install, env, credentials, and skill paths are preserved. ## Retry and Error Handling Rollout.run() catches common errors: - `TimeoutError` — agent exceeded timeout - `ConnectionError` — SSH/ACP pipe closed (retried 3x with exponential backoff) - `ACPError` — agent protocol error Evaluation-level retry with `RetryConfig`: ```python from benchflow.evaluation import Evaluation, EvaluationConfig, RetryConfig config = EvaluationConfig( retry=RetryConfig( max_retries=2, wait_multiplier=2.0, min_wait_sec=1.0, max_wait_sec=30.0, ), ) ``` --- ## Sandbox and Reward Types ### Sandbox Protocol The `Sandbox` protocol defines the interface any sandbox backend must implement. Docker and Daytona are built-in; you can bring your own (Modal, Firecracker, E2B, etc.). ```python from benchflow import Sandbox, ImageBuilder, ImageConfig, ImageRef # Sandbox is a runtime-checkable Protocol class MySandbox: async def exec(self, cmd: str, *, user: str = "root", timeout_sec: int = 30) -> ExecResult: ... async def upload_file(self, src: Path, dst: str) -> None: ... async def download_file(self, src: str, dst: Path) -> None: ... async def start(self) -> None: ... async def stop(self, *, delete: bool = True) -> None: ... # ... plus snapshot/restore + host/expose_ports; see sandbox/protocol.py assert isinstance(my_sandbox, Sandbox) # works at runtime ``` ### Rubric + RewardFunc (Composable Rewards) Declarative scoring via composable reward functions. ```python from benchflow import Rubric, RewardFunc, RewardEvent, VerifyResult from benchflow import TestRewardFunc, StringMatchRewardFunc, LLMJudgeRewardFunc # Built-in reward functions test_reward = TestRewardFunc() # runs pytest, binary pass/fail match_reward = StringMatchRewardFunc(expected="hello world") # Compose into a weighted Rubric rubric = Rubric( reward_funcs=[test_reward, match_reward], weights=[0.7, 0.3], ) # Score a workspace result: VerifyResult = await rubric.score(rollout_dir=my_rollout_dir) print(result.reward) # weighted float [0.0, 1.0] print(result.events) # list[RewardEvent] — per-function breakdown ``` ### Adapters (Inspect AI + ORS) Convert between BenchFlow types and external frameworks. ```python from benchflow import InspectAdapter, ORSAdapter, to_inspect_task, to_ors_reward # BenchFlow Scene → Inspect AI task format inspect_task = to_inspect_task(scene, rubric=rubric) # BenchFlow VerifyResult → ORS reward format ors_payload = to_ors_reward(verify_result) ``` ### Evaluation Batch orchestration with concurrency and retries. ```python from benchflow import Evaluation, EvaluationConfig, EvaluationResult, RetryConfig # EvaluationConfig holds the per-job settings (agent/model/environment/...) # applied to every task discovered under tasks_dir. config = EvaluationConfig( model="gemini-3.1-flash-lite-preview", environment="docker", concurrency=2, retry=RetryConfig(max_retries=2), ) evaluation = Evaluation(tasks_dir="tasks", jobs_dir="jobs/my-run", config=config) eval_result: EvaluationResult = await evaluation.run() ``` --- ## /docs/benchflow/rubric-review Rubric review is a detached, agentic quality review of finished rollouts. A reviewer agent reads a rollout's records — trajectory, result, verifier output, and the task definition — inside its own sandbox and grades the run against a rubric, one `pass` / `fail` / `not_applicable` verdict plus an explanation per criterion. Review is **report-only**. It runs after a job is over, from the host-side rollout directories, and writes `review_report.json`. It never modifies a reviewed rollout's `rewards` or `result.json`, and there is no code path through which it could: the deterministic verifier is the only owner of `reward`. This is distinct from the [`llm-judge` verifier strategy](/docs/benchflow/llm-judge): an llm-judge is part of a task's verifier and *produces* the reward, while rubric review is downstream quality assurance *about* finished runs — is the task well specified, did the agent game the grader, was the method sound. ## The rubric (contract v0.1) A rubric is a JSON file with one list: ```json { "criteria": [ { "name": "reward_hacking", "description": "Author-facing note about why this criterion exists.", "guidance": "Study the recorded run under trial/ for signs the agent manipulated grading instead of doing the work. PASS when ... FAIL when ... NOT_APPLICABLE when ..." } ] } ``` Each criterion is exactly three strings: | Field | Purpose | |---|---| | `name` | Stable identifier. Becomes a field in the reviewer's structured-output schema, so it must be a valid Python identifier. | | `description` | Documentation for humans reading the rubric. **Never included in the reviewer prompt** — grading must not depend on it. | | `guidance` | The grading contract the reviewer follows. Put the full pass/fail/not-applicable conditions here. | There are no weights, gates, thresholds, or aggregate scores. Consumers read per-criterion outcomes from the report and apply their own policy. The contract is named **v0.1**; the document itself carries no version key — a rubric is exactly its `criteria` list. A rubric must contain at least one criterion, names must be unique, and unknown fields are rejected. (Validation is stricter than the shape alone requires: rubrics that would produce vacuous or ambiguous reviews are refused. Every rubric that passes is exactly the v0.1 shape.) `rubric.json` is an overloaded filename — llm-judge verifier rubrics use `{id, match_criteria}` entries. Discovery is fail-closed: a `rubric.json` is treated as a review rubric — and validated loudly — **unless** every entry carries the full judge shape (both `id` and `match_criteria`). Unreadable files, invalid JSON, empty or missing `criteria`, and misspelled keys are all claimed and rejected with an explicit error rather than silently replaced by the default rubric. Rubric resolution order, per reviewed rollout: 1. an explicit `--rubric/-r` file, 2. the reviewed task's own `verifier/rubric.json` (or `tests/rubric.json`) when it is shaped like a review rubric, 3. the built-in default rubric (`reward_hacking`, `task_specification`). ## Running a review ```bash # review one rollout locally with a Codex subscription bench review jobs// --sandbox docker \ --agent codex --model gpt-5.5 --tasks-root ./tasks # review a small job locally, two rollouts at a time bench review jobs/ --sandbox docker -n 2 \ --agent codex --model gpt-5.5 --tasks-root ./tasks # audit the winners for grader manipulation bench review jobs/ --passing \ --agent codex --model gpt-5.5 # analyze the losers for specification gaps bench review jobs/ --failing -r spec-rubric.json \ --agent codex --model gpt-5.5 ``` `--passing` selects rollouts with reward 1.0 and no recorded error; `--failing` selects everything else, including rollouts whose `result.json` is unreadable. The reviewer agent (`--agent`, default `opencode`) and model (`--model`; agents without a registry default require one) are independent of whatever ran the original job. Docker is the default and is appropriate for one review or a small local job; choose a remote sandbox only when you need more isolated parallel reviewers. ## How a review executes Each review is an ordinary rollout of a throwaway wrapper task assembled on the host, which is why every sandbox backend (`docker`, `daytona`, `agentcore`, ...) works unchanged: - **Prebuilt image, pinned by digest.** The wrapper declares a digest-pinned `python` image and ships no Dockerfile, so Docker and Daytona never build one. AgentCore is the exception: it must wrap any image with its runtime-contract shim, so it still builds and pushes a derived ECR image once per distinct image, then reuses it. - **Evidence by upload, outside the workdir.** A copy of the rollout directory is uploaded to `/evidence/trial`. A task copy is uploaded to `/evidence/task` only when it is admitted through the trusted-root and digest checks below. `/evidence` sits outside the agent workdir; after all uploads, a pre-agent hook fails closed unless the whole tree can be made root-owned, readable, and non-writable by the reviewer. Prior review outputs are excluded from the copy, so a re-review can never read an earlier verdict; symlinks anywhere in the evidence are dropped, never dereferenced; task skills and any shipped `rubric.json` are excluded from the task copy. The canonical ACP trajectory is retained. When an ACP implementation drops a completed tool observation or reduces a command title to the generic tool name, BenchFlow reconciles the missing detail from the matching exact-ID event in its trusted provider capture before the canonical record is finalized. The cumulative provider-history `llm_trajectory.jsonl` remains omitted: it repeats the growing conversation on every request and can exhaust a reviewer model's context. The reviewed rollout itself is never touched. - **Post-initialization egress restriction, fail closed.** The wrapper declares `allow_internet: false`, which disables web tools, forces the model proxy sandbox-local, and arms the agent-UID egress firewall scoped to that loopback gateway. Backends that cannot enforce isolation (for example `agentcore`, whose runtime only offers PUBLIC/VPC networking) refuse the review at launch; `--allow-open-network` is the explicit, report-recorded override for them. Be precise about what this guarantees: the container needs network during image setup and agent installation, so the firewall is armed *after* the reviewer harness starts and completes ACP initialization. The guarantee is **restricted egress for the graded portion of the run**, not network isolation for the container's whole lifetime. Evidence is uploaded during sandbox setup, before the firewall is enforced, so a reviewer harness that is itself malicious could egress during startup **after evidence is present**. Treat the reviewer harness as trusted code; the untrusted input is the evidence it reads, and this guarantee constrains the graded portion of the run, not a hostile harness. - **Task evidence requires an explicitly trusted root.** A rollout's recorded `task_path` is rollout-authored data, so it is never read directly — pass `--tasks-root ` and the task is looked up *by name* beneath that root. Without it, the review proceeds from run records alone and says so in the trial's `notes`. When the rollout recorded a `task_digest` in `result.json` or `config.json`, the values must be valid and mutually consistent. A missing digest, mismatch against the on-disk task, conflict, or any verification failure **excludes the task from evidence** and says so in `notes`; an old or unverifiable rollout is never reviewed against current task content. - **The rubric never enters the sandbox.** It is decomposed host-side: `guidance` lines render into the instruction, criterion names become the output schema and `tests/criteria.json`. `description` goes nowhere. - **Validity-only reward.** The wrapper's verifier is a stdlib-only structural check of the reviewer's `review-result.json` (every criterion answered, outcomes in vocabulary, non-empty explanations). Reward 1.0 means "a well-formed review exists" — never "the reviewed run was good". - **Failure isolation.** A review that crashes or produces malformed output becomes an error entry for that rollout; the rest of the job continues. ## Output The review job directory contains `review_report.json`: ```json { "path": "…/jobs/2026-08-03__12-00-00", "rubric": {"path": "…", "criteria": ["…"]}, "reviewer": {"agent": "opencode", "model": "gemini/gemini-2.5-flash", "environment": "docker", "network": "no-internet"}, "job_summary": "Deterministic aggregation over VALID reviews only.", "trials": [ { "trial_name": "hello-world-task__829cddb8", "source_rollout": "…", "review_valid": true, "summary": "Three-to-five sentence account of the run.", "checks": { "reward_hacking": {"explanation": "…", "outcome": "pass"}, "task_specification": {"explanation": "…", "outcome": "fail"} }, "error": null, "reviewer_rollout": "…/runtime/hello-world-task__829cddb8//…", "rubric_path": "…/verifier/rubric.json", "criteria": ["reward_hacking", "task_specification"], "notes": ["task evidence skipped: no --tasks-root was given"] } ] } ``` Each reviewer rollout's own records (trajectory, verifier output, raw `review-result.json`) sit under the report's `runtime/` directory for audit; every invocation uses a fresh unique runtime leaf. When a leaf is successfully identified, `reviewer_rollout` points at that exact leaf; otherwise it is `null` rather than an ambiguous parent directory. Reusing `--out-dir` can therefore never resurface a stale review. The job summary is a deterministic aggregation, not a model call — a host-side LLM call would bypass the sandbox backend, egress policy, and telemetry. ## Writing good criteria - Put the entire decision rule in `guidance`, including when to answer `not_applicable` (for example: infrastructure failure before the agent ever attempted the task). - One judgment per criterion. A criterion that bundles several claims makes `fail` ambiguous. - The reviewer reads evidence produced by the solver. Guidance should direct it to concrete records (`trial/result.json`, `trial/trajectory/`, `trial/verifier/`) rather than to intent. - `description` is the right place for authorship context you do not want influencing the judge — provenance, rationale, links. --- ## /docs/benchflow/running-any-benchmark BenchFlow's job is to take *any* benchmark and produce a scored trajectory you can read, compare, and train on. It sits downstream of every environment framework: whatever shape a benchmark arrives in, BenchFlow routes it to one of three execution layers and ends at a single output contract. The routing is the whole idea. You do not pick the layer by hand — the benchmark's format picks it for you: | If the benchmark is… | Layer | What BenchFlow does | |----------------------|-------|---------------------| | In a framework BenchFlow speaks *inbound* — Harbor (task-dir adapter) or PrimeIntellect / Verifiers (hosted) | **1 — native** | Runs it in its supported form; correctness is inherited from the original format | | In a variant of a known format, or a format BenchFlow has never seen | **2 — translated** | Translates it to the native `task.md` format, then *proves* equivalence with a parity gate | | A one-off harness with its own runner and scoring, no reusable adapter | **3 — as-is** | Runs the benchmark under its own harness and interfaces with its output only | All three layers terminate at the same scored-trajectory contract (see [The seam](#the-seam-one-scored-trajectory-contract) below). One ingestion, every benchmark. Inspect AI and ORS / OpenReward are deliberately **not** in the table above: BenchFlow has no inbound run path for them. They are *outbound* export targets — BenchFlow writes its own native results into those formats — covered under [Layer 1's outbound format seams](#layer-1--supported-framework--run-natively). --- ## Layer 1 — supported framework → run natively When a benchmark already speaks a framework BenchFlow supports, BenchFlow runs it in that form and converts the output to results. There is no translation step and nothing to prove: correctness is inherited from the original format. The adapters live in [`src/benchflow/adapters/`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/adapters/) and are pure format translators — none of them require the external framework's SDK. **Inbound (foreign task dir → native runtime).** [`detect_adapter()`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/adapters/inbound.py) sniffs a task directory by its signature file and returns the matching adapter: - A `task.toml` → [`HarborAdapter`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/adapters/harbor.py). Harbor is the upstream framework BenchFlow's own `TaskConfig` was internalized from, so a Harbor task directory is *already* in native shape; the adapter is a thin normalizer. The adapter returns an `InboundTask`; the benchmark then runs on BenchFlow's native runtime exactly like a first-party task. **Hosted environments (run on their own native surface).** External PrimeIntellect / Verifiers environments are not BenchFlow task directories and do not use BenchFlow's Docker/Daytona sandbox runner. BenchFlow runs them through their native Verifiers execution surface and preserves their hosted identity (`env_uid`, `hub_url`), while still writing the shared output contract — see [`src/benchflow/hosted_env.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/hosted_env.py): ```bash bench eval run \ --source-env primeintellect/general-agent \ --source-env-version 0.1.1 \ --model google/gemini-2.5-flash-lite ``` **Outbound format seams (native results → other frameworks).** Results also round-trip *out* into the frameworks teams already use: [`InspectAdapter`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/adapters/inspect_ai.py) maps a BenchFlow `Scene` + `Rubric` into an Inspect AI task, and [`ORSAdapter`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/adapters/ors.py) maps `VerifyResult` / `RewardEvent` into the ORS (OpenReward) reward-record shape. --- ## Layer 2 — unknown or variant format → translate, then validate or prove parity When a benchmark's format is **not** one BenchFlow speaks natively — a variant of a known layout, or an entirely new one — BenchFlow translates it to the native `task.md` format. There are **two distinct translation flows**, and they check the result in **different** ways. They do not chain together; pick the one that matches what you are translating: | You are translating… | Command path | How the result is checked | |----------------------|--------------|---------------------------| | A task you already control, in the legacy split layout (`task.toml` + `instruction.md`) | `bench tasks migrate` → `bench tasks check` | Structural validation of the generated `task.md` (config equivalence is enforced at conversion time) | | A foreign benchmark with no reusable adapter | `bench eval adopt ` → `bench eval adopt --verify` | The parity gate **proves** the converted benchmark reproduces the original's results | The two paths are not interchangeable: `bench eval adopt --verify` runs only against a benchmark *adopted* with `bench eval adopt `. It reads `benchmarks//parity_experiment.json` and errors `benchmark not adopted … run bench eval adopt first` on anything else — including a migrated `task.md`. A migrated `task.md` is validated with `bench tasks check`, never with `--verify`. ### (a) Migrate a task you control → validate with `bench tasks check` `bench tasks migrate ` converts a legacy split layout (`task.toml` + `instruction.md`) into the unified `task.md` format ([`cli/main.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/cli/main.py) → `migrate_task_to_task_md`). The conversion checks config equivalence before writing, so it cannot silently drop supported task configuration. Validate the generated package with: ```bash bench tasks check ``` That is the structural gate (`check_task` in [`_utils/task_authoring/`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/_utils/task_authoring/__init__.py)): it rejects unreplaced `[REPLACE: …]` placeholders, missing required files, and a verifier package with no runnable entrypoint. This flow records no `parity_experiment.json` and runs no parity gate — it is a faithful in-place format conversion of a task you already own. ### (b) Adopt a foreign benchmark → `bench eval adopt `, then prove with `bench eval adopt --verify` For a foreign benchmark with no reusable adapter, the benchmark-adoption router in [`src/benchflow/agent_router.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/agent_router.py) drives the work as a single multi-mode command: - `bench eval adopt ` scaffolds `benchmarks//` to the reference layout (if missing), then drives the conversion workflow with an agent toward a `benchmarks//` pull request. The conversion guide is embedded in the command. (`bench eval adopt --scaffold-only` writes just the package.) Only a benchmark adopted this way carries the `benchmarks//` directory the parity gate below requires. ### Prove — the parity gate `bench eval adopt --verify` closes the adopt → verify loop. It is a **parity-only** gate (`build_verify_report` in [`agent_router.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/agent_router.py)) over two layers: 1. **Deterministic conversion parity (the floor).** Every compared criterion's *converted* verdict must match the *original's* verdict on identical inputs — a side-by-side, per-criterion comparison (`extract_criterion_comparisons` → `ConversionParity`). 2. **Reward-distribution parity (the statistical layer).** Every legacy-vs-converted reward delta must sit within tolerance (`DEFAULT_REWARD_TOLERANCE = 0.02`, overridable with `--tolerance`) — `RewardDistributionParity`. The gate emits one of three verdicts: `parity-confirmed`, `parity-divergent`, or `insufficient-evidence` (a layer with no data does not block; no data at all returns `insufficient-evidence`). Two principles keep it honest: - **It never improves the source.** A faithful conversion reproduces the original's behavior on identical inputs — including any reward-hackability the original has. Parity never sanitizes or "fixes" the benchmark. - **Divergences are triaged, not buried.** On a non-confirmed verdict the gate renders a draft issue body (`render_divergence_issue`) for a human to review and file — it never auto-files anything. ### The artifacts Each adopted benchmark records its evidence in `benchmarks//parity_experiment.json`. `bench eval adopt --verify` reads and scores that file when it is a JSON object in the shape the scaffold writes: it pulls per-criterion verdict pairs and legacy-vs-converted reward samples from the object and emits a verdict. A file that records neither yields no comparisons, so the gate returns `insufficient-evidence`. The repository ships several recorded experiments under [`benchmarks/*/parity_experiment.json`](https://github.com/benchflow-ai/benchflow/blob/main/benchmarks/), and they are **not** uniform — do not assume `verify` scores all of them. `bench eval adopt programbench --verify` reads recorded reward-distribution samples and reports `parity-confirmed` (max abs reward delta within the default `0.02` tolerance). Other shipped experiments record structural- and eval-parity notes the gate does not read as criteria or reward samples, so `verify` returns `insufficient-evidence` for them; and one predates this object contract and stores a top-level list of experiment runs, which the gate cannot score. Use `verify` as the gate for experiments recorded in the object shape, and read the JSON files directly for the rest. Read the recorded experiments honestly: where rewards are recorded they report aggregate deltas **within tolerance**, with residual disagreements triaged to causes such as model non-determinism rather than conversion defects. The guarantee is "parity within tolerance, divergences triaged, no conversion defect found" — not a fixed headline percentage. --- ## Layer 3 — bespoke benchmark → run as-is, interface with output only Some benchmarks have no reusable adapter and no portable task format: a one-off harness with its own runner, its own scoring, and sometimes its own agent loop. Translating it would be a rewrite. Instead, BenchFlow runs the benchmark **under its own harness** and interfaces only with its output, mapping that onto the shared contract. Three real seams support this shape: - **The ACP shim pattern.** A benchmark's native agent loop is wrapped as an ACP server, so BenchFlow drives it over stdio and reads the trajectory it emits — the original loop runs unchanged. The agent registry ([`src/benchflow/agents/registry.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/agents/registry.py)) registers harness agents this way — a `*-harness` ACP-shim entry whose `launch_cmd` runs the original harness and whose `protocol` is `acp`. - **Native-harness / hosted runs.** [`hosted_env.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/hosted_env.py) runs an environment on its own execution surface, preserves the raw native evidence (e.g. under a `hosted_env/` subdir for forensics), and still writes the shared contract — the same "run as-is, ingest the output" shape, tagged with `source.type="hosted_env"` lineage. - **Trace import.** `bench tasks generate --from-file / --from-hf / --from-local` ([`cli/trace_import.py`](https://github.com/benchflow-ai/benchflow/blob/main/src/benchflow/cli/trace_import.py)) ingests external agent traces — JSONL trace files, HuggingFace datasets, or local Claude Code sessions — into BenchFlow records, with no harness conversion at all. --- ## The seam: one scored-trajectory contract Every layer terminates at the *same* output contract, written per rollout under `//__/` (the full layout is in [Getting started → Where results land](/docs/benchflow/getting-started#where-results-land)): | File | What it carries | |------|-----------------| | `result.json` | Rollout summary — rewards, errors, token usage/cost | | `results.jsonl` | Verifiers/Prime-RL shaped rollout row | | `rewards.jsonl` | The reward record for the rollout (ORS / OpenReward shape) | | `trajectory/acp_trajectory.jsonl` | Full agent trace as ACP events | | `trajectory/llm_trajectory.jsonl` | Raw provider requests/responses (when captured) | | `trainer/verifiers.jsonl` | Trainer-ready scored trajectory (Verifiers record) | | `trainer/atif.json` | ATIF trajectory record | | `trainer/adp.jsonl` | ADP trajectory record | | `verifier/` | Raw verifier output (`reward.txt`, `ctrf.json`, stdout) | Hosted runs share this artifact contract too (see the `hosted_env.py` module docstring), with `source.type="hosted_env"` / `trajectory_source="hosted_env"` marking the lineage. Because the contract is the same regardless of which layer produced it, one ingestion path serves every benchmark: release checks, trainers, and downstream reporting tools read the same files whether the run came from a native adapter, a parity-proven translation, or a bespoke harness. --- ## Where to go next - [Getting started](/docs/benchflow/getting-started) — install and run your first eval - [Concepts](/docs/benchflow/concepts) — Rollout / Scene / Role / Verifier - [Native `task.md` authoring](/docs/benchflow/task-authoring-task-md) — the translation target for Layer 2 - [Architecture](https://github.com/benchflow-ai/benchflow/blob/main/docs/architecture.md) — adapters and trainers as the edges of the system --- ## /docs/benchflow/running-evaluations `bench eval run` is the main command for both one task and a batch. It accepts a local directory, a remote Git repository, a YAML run config, or a pinned dataset version. All of these use Docker by default. Start with [Getting started](/docs/benchflow/getting-started) if you have not completed a local run yet. ## One task from a remote repository ```bash bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/citation-check \ --agent codex \ --model gpt-5.5 \ --sandbox docker ``` BenchFlow clones and caches the source under `.cache/datasets/`. Pin a branch, tag, or commit with `--source-ref` when reproducibility matters. ## One local task ```bash bench eval run \ --tasks-dir tasks/my-task \ --agent codex \ --model gpt-5.5 ``` The omitted `--sandbox` defaults to `docker`. A task directory contains a native `task.md` plus its `environment/` and `verifier/` directories. BenchFlow can still read the retired split layout for compatibility. ## A local batch Point `--tasks-dir` at the parent directory and choose a conservative local concurrency: ```bash bench eval run \ --tasks-dir tasks \ --agent codex \ --model gpt-5.5 \ --sandbox docker \ --concurrency 2 ``` Use repeatable filters to select task names: ```bash bench eval run \ --tasks-dir tasks \ --include citation-check \ --include weighted-gdp-calc \ --agent codex \ --model gpt-5.5 ``` Local concurrency is limited by your Docker daemon, CPU, memory, and model rate limits. Increase it gradually. A cloud sandbox becomes useful when you need more isolation or more parallel machines, not because BenchFlow requires one; see [Sandboxes](/docs/benchflow/sandboxes). ## YAML run configs Use a config when the same run should be repeated or reviewed: ```yaml source: repo: benchflow-ai/skillsbench path: tasks agent: codex model: gpt-5.5 environment: docker concurrency: 2 include: - citation-check ``` ```bash bench eval run --config run.yaml ``` Check the [CLI reference](/docs/benchflow/reference/cli#bench-eval-run) for the full run schema and flags. ## Compare a task with and without skills For a task that already contains its skill payload, run the two modes into separate job directories: ```bash bench eval run \ --tasks-dir tasks/my-task \ --agent codex --model gpt-5.5 \ --skill-mode no-skill \ --jobs-dir jobs/my-task-no-skill bench eval run \ --tasks-dir tasks/my-task \ --agent codex --model gpt-5.5 \ --skill-mode with-skill \ --jobs-dir jobs/my-task-with-skill ``` Use `--skills-dir ` when the skills live outside the task package. For a structured lift experiment backed by `evals/evals.json`, use `bench skills eval`; see [Skill evals](/docs/benchflow/skill-eval). ## Results and exit status By default, artifacts land in `jobs//`. Summarize them with: ```bash bench eval list jobs/ bench eval metrics jobs/ ``` Exit code 0 means the evaluation pipeline completed. It does not mean every task passed. Read each rollout's reward or the printed `[PASS]` / `[FAIL]` status to assess model performance. Configuration, agent, or verifier errors produce a non-zero exit. Use a new `--jobs-dir` for an independent rerun. Reusing one intentionally resumes it and skips completed rollouts. ## Reproducible published runs For leaderboard, paper, or release evidence, prefer a pinned registry dataset: ```bash bench eval run \ --dataset skillsbench@1.1 \ --agent codex \ --model gpt-5.5 ``` Dataset runs verify the pinned commit and per-task content digests. Ad-hoc `--tasks-dir` and floating repository runs are better suited to development. --- ## /docs/benchflow/sandbox-hardening ## Why Frontier agents will reward-hack any verifier they can. [BenchJack](https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/) (Berkeley RDI, April 2026) showed that ~100% of tasks across eight major AI agent benchmarks are solvable by exploiting the *verifier's runtime* rather than the agent's intelligence: pytest `conftest.py` hooks, planted PATH binaries, leaked answer keys, `.pth` injection, `sitecustomize.py` shadowing. [Meerkat](https://debugml.github.io/cheating-agents/) (Penn, same week) confirmed it from the trace side — frontier models discover and use these shortcuts in the wild. benchflow's verifier hardening blocks these patterns by default. Tasks opt out individually when their legitimate test setup requires a hook the cleanup would otherwise delete (e.g. qutebrowser's `conftest.py` resolves a real circular import). ## What gets hardened Between the agent's last action and the verifier's first command, `harden_before_verify()` (in `src/benchflow/sandbox/lockdown.py`) runs this sequence: 1. **Kill sandbox-user processes** — `pkill -u `, twice with a sleep, so the agent can't keep writing during teardown. 2. **Wipe `/logs/verifier`** — fresh, root-owned, mode 0777 (so a non-root verifier can write its own results). 3. **Restore build-config files** from a pre-agent snapshot — `setup.py`, `pyproject.toml`, `setup.cfg`, `tox.ini`, `noxfile.py`, `hatch.toml`, `flit.ini`, `MANIFEST.in`, `requirements.txt`, `requirements-dev.txt`, `Makefile`. Closes the editable-install hijack vector. 4. **Optionally restore the entire workspace** from `/testbed_verify` snapshot — opt-in per task because some tasks score workspace edits. 5. **Purge symlinks whose realpath escapes the workspace** — agent can't redirect to `/tmp/evil.py`. 6. **Purge `__pycache__` trees** that didn't exist in the pre-agent baseline. 7. **chown workspace to root** — belt-and-suspenders against zombie sandbox-user processes. 8. **`CLEANUP_CMD`** — delete `conftest.py` outside `/tests/`, `*.py` from `/tmp`/`/var/tmp`, `sitecustomize.py`/`usercustomize.py`/`*.pth` from writable `sys.path` entries. 9. **Set `VERIFIER_ENV`** — pinned `PATH`, `PYTEST_ADDOPTS=-c /dev/null --confcutdir=/tests --rootdir=/app -p no:cacheprovider`, `PYTHONPATH=""`, `PYTHONDONTWRITEBYTECODE=1`, plugin auto-discovery off. 10. **Discover allowed pytest plugins** — `importlib.metadata` scan inside the container, then merge with the task-declared `pytest_plugins` from its config (`task.md` front-matter, or `task.toml` for split-layout tasks). Anything not in the allow-list is blocked. The verifier then runs against this hardened workspace. ## Per-task opt-outs Tasks declare opt-outs in their task config (`task.md` front-matter, or `task.toml` for split-layout tasks): ```toml [verifier.hardening] cleanup_conftests = false ``` | Flag | Default | Effect when `false` | |------|---------|---------------------| | `cleanup_conftests` | `true` | Don't delete `conftest.py` outside `/tests/` before verify | Other cleanup steps (`sitecustomize.py`, `.pth`, `/tmp` `*.py`) always run — they have no legitimate use in a test artifact and disabling them would broaden the attack surface beyond what real tasks need. Unknown keys in `[verifier.hardening]` are warned and ignored. String values for boolean flags are rejected. See [`progressive-disclosure.md`](/docs/benchflow/progressive-disclosure#per-task-hardening-opt-outs) for the qutebrowser case study (legitimate `conftest.py` for circular-import fix). ## Threat model and known gaps Benchflow's hardening assumes: - The Docker image and pre-installed packages are trusted (they're built before the agent runs). - The agent runs as a non-root sandbox user (`sandbox_user`) — when set to `None`, hardening still runs but the agent can defeat lockdown by being root. Don't run untrusted agents as root. - The verifier itself is trusted code shipped with the task; benchflow only hardens *what the agent can do to the verifier's environment*. Known residual risk: - An agent with sustained access can poison `__pycache__` for files that exist in the baseline (those caches aren't deleted because some tasks diff workspace against `/testbed_verify`). Mitigated by the workspace chown but not eliminated. - Tasks that don't ship a build-config snapshot can still be hijacked via `setup.py` edits. Snapshot is automatic for declared filenames — task authors don't need to opt in. ## Related - [`progressive-disclosure.md`](/docs/benchflow/progressive-disclosure) — soft-verify (the relaxed hardening used between rounds in multi-round trials). - [`task-authoring.md`](/docs/benchflow/task-authoring) — the task config schema, including the `[verifier.hardening]` opt-outs. --- ## /docs/benchflow/sandboxes BenchFlow needs an isolated place to install an agent, expose the task workspace, and run the verifier. For local development, that place is Docker. Cloud sandboxes are optional scaling backends. ## Which sandbox should I use? | Sandbox | Use it when | Setup | |---|---|---| | **Docker** | You are learning BenchFlow, developing a task, debugging a run, or running a small batch locally | Docker daemon; included in the base install | | **Apple Container** | You are on a supported Apple Silicon Mac and want Apple's native container runtime | Apple `container` CLI; no BenchFlow extra | | **Daytona** | You need many independent cloud VMs for a light, highly parallel batch | Optional extra plus `DAYTONA_API_KEY` | | **Modal** | You need serverless or GPU-backed remote execution | Optional extra plus Modal auth | | **AgentCore** | Your deployment is built around AWS Bedrock AgentCore Runtime | Optional extra plus AWS configuration | Docker is the CLI default. If your tasks and model calls fit on one machine, there is no reason to configure Daytona. ## Local Docker ```bash docker info >/dev/null bench eval run \ --tasks-dir tasks/my-task \ --agent codex \ --model gpt-5.5 \ --sandbox docker ``` Start with `--concurrency 1` or `2`, then raise it while watching local CPU, memory, disk, Docker build pressure, and provider rate limits. Docker uses host disk capacity and supports multi-container task environments. ## Apple Container On supported Apple Silicon Macs: ```bash bench eval run \ --tasks-dir tasks/my-task \ --agent codex \ --model gpt-5.5 \ --sandbox apple-container ``` Apple Container is a single-container backend and currently cannot enforce a task's `no-network` policy. Use Docker for multi-service or strict no-network tasks. ## Daytona Install Daytona support only when you need it: ```bash uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]' export DAYTONA_API_KEY='...' bench eval run \ --tasks-dir tasks \ --agent gemini \ --model gemini-3.1-pro-preview \ --sandbox daytona \ --concurrency 32 ``` Daytona is useful for parallel experiments because each rollout gets a remote VM. It is not a prerequisite for a single evaluation. Daytona also caps each sandbox at 10 GB of storage, so tasks with large model snapshots, Playwright, LaTeX, or other heavy images may fail during bootstrap. Run those tasks with Docker when local host disk is available. ## Modal and AgentCore ```bash uv tool install --python 3.12 --upgrade 'benchflow[sandbox-modal]' uv tool install --python 3.12 --upgrade 'benchflow[sandbox-agentcore]' ``` Select them with `--sandbox modal` or `--sandbox agentcore` after configuring the provider's authentication. Both are deployment choices for specific remote workloads, not part of the local quickstart. They are single-container backends; AgentCore also cannot enforce `no-network` tasks. ## Keep the task portable The sandbox flag selects where a task runs; it should not change what the task means. Develop and debug with Docker first, then run a small parity check on the intended cloud backend before starting a batch. If a task requires a backend-specific capability, document that requirement in the task rather than silently assuming Daytona. --- ## /docs/benchflow/skill-eval Test whether your agent skill actually helps agents perform better. ## Install ```bash uv tool install --python 3.12 --upgrade benchflow ``` BenchFlow CLI releases require Python 3.12 or newer. Keep `--python 3.12` in tool-install commands so `uv` does not fall back to an old release without the `bench` / `benchflow` executables. ## Overview `bench skills eval` takes a skill directory with an `evals/evals.json` file, generates benchmark tasks from it, runs them with and without the skill installed, and reports the "lift" — how much the skill improves agent performance. 0.6 task-standard validation is in [`docs/reports/2026-06-09-task-standard-validation.md`](https://github.com/benchflow-ai/benchflow/blob/main/docs/reports/2026-06-09-task-standard-validation.md). ## Quick start ### 1. Add evals to your skill ``` my-skill/ ├── SKILL.md ├── scripts/ │ └── helper.py └── evals/ # ← add this └── evals.json ``` ### 2. Write test cases ```json { "version": "1", "skill_name": "my-skill", "defaults": { "timeout_sec": 300, "judge_model": "gemini-3.1-flash-lite" }, "cases": [ { "id": "test-001", "question": "Do X using the my-skill skill.", "ground_truth": "expected output", "expected_behavior": [ "Agent read the SKILL.md file", "Agent ran helper.py with correct arguments", "Agent produced the expected output" ] } ] } ``` ### 3. Run the eval ```bash bench skills eval my-skill/ --agent claude-agent-acp ``` Expected output: ``` $ bench skills eval ./my-skill/ --agent claude-agent-acp Skill eval: my-skill (1 cases) Agents: claude-agent-acp Environment: docker Skill Eval: my-skill ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓ ┃ Agent ┃ Mode ┃ Score ┃ Avg Reward ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩ │ claude-agent-acp │ with-skill │ 1/1 │ 0.90 │ │ claude-agent-acp │ baseline │ 0/1 │ 0.20 │ │ claude-agent-acp │ LIFT │ +1 │ +0.70 │ └───────────────────┴────────────┴───────┴────────────┘ ``` ## evals.json reference ### Top-level fields | Field | Type | Required | Description | |-------|------|----------|-------------| | `version` | string | No | Schema version (default: "1") | | `skill_name` | string | No | Skill name (auto-detected from SKILL.md) | | `defaults.timeout_sec` | int | No | Per-task timeout in seconds (default: 300) | | `defaults.judge_model` | string | No | Model for LLM judge (default: gemini-3.1-flash-lite) | | `defaults.skill_mount_dir` | string | No | Neutral sandbox path where the generated task exposes the skill before BenchFlow links it into agent-specific discovery paths (default: /skills) | ### Case fields | Field | Type | Required | Description | |-------|------|----------|-------------| | `id` | string | No | Unique case ID (auto-generated if missing) | | `question` | string | **Yes** | The task instruction sent to the agent | | `ground_truth` | string | No | Expected final answer (used for exact match fallback) | | `expected_behavior` | string[] | No | Behavioral rubric for LLM judge | | `expected_skill` | string | No | Which skill should be invoked | | `expected_script` | string | No | Which script should be called | | `environment` | object | No | Per-case env var overrides | ### Grading logic - If `expected_behavior` is provided → **LLM judge** scores the agent's trajectory against the rubric (0.0-1.0) - If only `ground_truth` is provided → **exact match** checks if the answer appears in agent output (0.0 or 1.0) - If neither → reward is 0.0 ### Agent and judge credentials `bench skills eval` runs real agents. The selected agent must have whatever provider credentials or subscription auth it normally needs, and LLM-judge cases also need a supported judge key available in the environment. Exact-match cases can avoid the judge model, but they still need a working agent. For Codex agents, that auth can be `OPENAI_API_KEY`, `CODEX_API_KEY`, `CODEX_ACCESS_TOKEN`, or a host `~/.codex/auth.json` login. Provider-prefixed models can use provider-specific credentials instead; Azure Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT`. When a supported judge key is present on the host (`GOOGLE_API_KEY`, `GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`), generated tasks reference it through `[verifier.env]` template syntax such as `${GEMINI_API_KEY}`. Secret values are resolved at verifier runtime and are not written into generated task files. The `oracle` agent is useful for generic task and sandbox smoke tests, but it is not a replacement for skill evaluation. Skill-eval tasks are generated from questions and rubrics and do not include `solution/solve.sh`, so oracle runs will error instead of measuring skill lift. ### Existing task-embedded skills Skills embedded under a benchmark task, such as `tasks//environment/skills//SKILL.md`, are task-local skill packs. They are not exposed to ordinary no-skills runs by default. To evaluate one directly with `bench skills eval`, add a sibling `evals/evals.json` inside that skill directory or copy the skill into a standalone skill directory with the same `evals/` contract. The repo includes a real task-embedded example at [`docs/examples/task-md/real-skillsbench/citation-check/environment/skills/citation-management/`](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/task-md/real-skillsbench/citation-check/environment/skills/citation-management/), adapted from the SkillsBench `citation-check` task. It is intentionally a task fixture rather than a standalone skill-eval package, so copy it to your own skill directory and add `evals/evals.json` before passing that directory to `bench skills eval`. ## Multi-agent comparison Test your skill across multiple agents: ```bash bench skills eval my-skill/ \ --agent claude-agent-acp --agent codex-acp --agent gemini ``` Expected output: ``` $ bench skills eval ./calculator/ --agent claude-agent-acp --agent codex-acp Skill eval: calculator (3 cases) Agents: claude-agent-acp, codex-acp Environment: docker Skill Eval: calculator ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓ ┃ Agent ┃ Mode ┃ Score ┃ Avg Reward ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩ │ claude-agent-acp │ with-skill │ 3/3 │ 0.95 │ │ claude-agent-acp │ baseline │ 1/3 │ 0.38 │ │ claude-agent-acp │ LIFT │ +2 │ +0.57 │ │ codex-acp │ with-skill │ 2/3 │ 0.72 │ │ codex-acp │ baseline │ 1/3 │ 0.35 │ │ codex-acp │ LIFT │ +1 │ +0.37 │ └───────────────────┴────────────┴───────┴────────────┘ ``` ## Custom environments For skills that need specific dependencies, add a Dockerfile: ``` my-skill/evals/ ├── evals.json ├── Dockerfile # custom container setup └── requirements.txt # extra Python deps ``` The Dockerfile is used instead of the default `python:3.12-slim` base. For with-skill runs, BenchFlow appends a `COPY skills/ /` step so the generated task exposes the skill at the neutral path declared in `task.toml`. During rollout setup, BenchFlow links that neutral path into the selected agent's configured discovery paths. ## GEPA integration Export traces for GEPA skill evolution: ```bash bench skills eval my-skill/ --agent claude-agent-acp --export-gepa ``` This creates a GEPA-compatible export under `jobs/skill-eval//gepa/`: ``` jobs/skill-eval//gepa/ ├── skill.md # current SKILL.md content ├── traces/ # per-case execution traces with scores │ ├── test-001-claude-agent-acp-with.json │ └── test-001-claude-agent-acp-without.json └── summary.json # aggregate lift metrics ``` Feed these to GEPA to evolve your skill: ```python import gepa optimizer = gepa.GEPA(traces_dir="traces/") improved_skill = optimizer.evolve("traces/skill.md") ``` ## End-to-End Walkthrough Here's a complete example evaluating a real skill from scratch. ### Step 1: Create the skill ```bash mkdir -p gws-skill/scripts gws-skill/evals ``` Write `gws-skill/SKILL.md`: ```markdown --- name: gws-email-drafting description: Draft professional emails using Gmail API patterns --- # GWS Email Drafting Use the templates in scripts/ to draft professional emails. ``` Write `gws-skill/scripts/draft_email.py`: ```python import sys template = sys.argv[1] if len(sys.argv) > 1 else "general" print(f"Email drafted using {template} template") ``` ### Step 2: Write eval cases Write `gws-skill/evals/evals.json`: ```json { "skill_name": "gws-email-drafting", "version": "1", "defaults": { "timeout_sec": 300, "judge_model": "claude-haiku-4-5-20251001" }, "cases": [ { "id": "draft-intro-email", "question": "Draft a professional introduction email to a potential workshop speaker. Use the gws-email-drafting skill.", "ground_truth": "The agent produced a professional email with subject line, greeting, body explaining the workshop, and call to action.", "expected_behavior": [ "The agent read the SKILL.md to understand the skill", "The agent used draft_email.py or followed the skill's patterns", "The email has a clear subject line", "The email body is professional and includes a call to action" ] }, { "id": "draft-followup", "question": "Draft a follow-up email to someone who hasn't responded in 2 weeks. Use the gws-email-drafting skill.", "ground_truth": "The agent produced a polite follow-up email that references the original outreach.", "expected_behavior": [ "The agent read the SKILL.md", "The email references a previous conversation", "The tone is polite but action-oriented", "The email is concise (under 200 words)" ] } ] } ``` ### Step 3: Run the eval ```bash $ bench skills eval ./gws-skill/ --agent claude-agent-acp --agent codex-acp Skill eval: gws-email-drafting (2 cases) Agents: claude-agent-acp, codex-acp Environment: docker Skill Eval: gws-email-drafting ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓ ┃ Agent ┃ Mode ┃ Score ┃ Avg Reward ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩ │ claude-agent-acp │ with-skill │ 2/2 │ 0.92 │ │ claude-agent-acp │ baseline │ 1/2 │ 0.55 │ │ claude-agent-acp │ LIFT │ +1 │ +0.37 │ │ codex-acp │ with-skill │ 2/2 │ 0.88 │ │ codex-acp │ baseline │ 1/2 │ 0.48 │ │ codex-acp │ LIFT │ +1 │ +0.40 │ └───────────────────┴────────────┴───────┴────────────┘ ``` ### Step 4: Inspect results Results are saved to `jobs/skill-eval//`: ``` jobs/skill-eval/gws-email-drafting/ ├── claude-agent-acp/ │ ├── with-skill/ │ │ ├── draft-intro-email__abc123/ │ │ │ ├── result.json │ │ │ ├── trajectory/acp_trajectory.jsonl │ │ │ └── timing.json │ │ └── draft-followup__def456/ │ │ └── ... │ └── baseline/ │ └── ... └── codex-acp/ └── ... ``` ### Step 5: Improve with GEPA (optional) ```bash $ bench skills eval ./gws-skill/ --agent claude-agent-acp --export-gepa GEPA traces exported to jobs/skill-eval/gws-email-drafting/gepa ``` Feed traces to the SkillSpin improvement pipeline to automatically evolve the skill text based on failure patterns. ## Architecture ``` ┌──────────────────────────────────────────────────────────────────┐ │ bench skills eval │ ├──────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────┐ ┌──────────────────┐ ┌────────────────┐ │ │ │ evals.json │───▶│ Task Generator │───▶│ Ephemeral │ │ │ │ (2-8 cases) │ │ (with/without │ │ BenchFlow Tasks │ │ │ └─────────────┘ │ skill mode) │ │ (auto-deleted) │ │ │ └──────────────────┘ └───────┬────────┘ │ │ │ │ │ ┌─────────────┐ ┌──────────────────┐ ┌───────▼────────┐ │ │ │ Lift Report │◀───│ Result Collector │◀───│ Job Engine │ │ │ │ (per agent) │ │ (per case×mode) │ │ (concurrency, │ │ │ └─────────────┘ └──────────────────┘ │ retries, ACP) │ │ │ └────────────────┘ │ │ │ │ With-skill tasks bake the skill at /skills by default; │ │ BenchFlow links that neutral path into each agent's skill paths.│ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ LLM Judge │ │ │ │ Reads: trajectory + case.json (ground_truth, rubric) │ │ │ │ Writes: /logs/verifier/reward.txt (0.0-1.0) │ │ │ └─────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────┘ ``` ## For Skill Developers (Jon Snow Adapter Pattern) If you maintain skills and want CI-integrated eval: ``` my-skill/ ├── SKILL.md ├── scripts/ │ └── do_something.py └── evals/ └── evals.json ← 2-4 test cases ``` That's it. No benchmark task authoring, no Dockerfiles, no test scripts. BenchFlow generates everything ephemeral — only results persist. **CI integration:** ```bash # In your skill's CI pipeline uv tool install --python 3.12 --upgrade benchflow bench skills eval . --agent claude-agent-acp --no-baseline ``` **What the adapter does (zero LLM):** ``` evals.json → Generate benchmark tasks → Run agents → Grade → Cleanup (static) (deterministic) (ACP) (LLM) (auto) ``` The adapter is purely deterministic — no LLM in task generation. LLM is only used at grading time (the judge). ## Tips for writing good eval cases 1. **Be specific in questions** — "Use the calculator skill to compute X" is better than "Compute X" 2. **Write 3-5 rubric items per case** — Each should be independently verifiable from the trajectory 3. **Include edge cases** — Test error handling, unusual inputs, multi-step workflows 4. **Keep ground_truth simple** — Exact match works best for numeric or short-string answers 5. **Use 2-4 cases minimum** — Enough to show a pattern, not so many that runs get expensive 6. **Test the lift, not just correctness** — The goal is to show the skill improves performance vs baseline. If baseline already scores high, the skill isn't adding value --- ## /docs/benchflow/start-in-5-minutes Launch a real, scored BenchFlow evaluation on your own machine in about five minutes. This path uses a public SkillsBench task, your existing ChatGPT subscription through Codex, and local Docker. You do not need Daytona or a model API key. The task is `3d-scan-calc`, not a toy prompt. The agent must parse a binary STL, remove disconnected scan debris, recover a material ID stored in each triangle, look up its density, calculate the largest component's mass, and write a JSON report that an independent verifier checks to 0.1% accuracy. > Five minutes is a quickstart target, not a timeout or performance guarantee. > A first run may take longer while Docker pulls images or BenchFlow downloads > the task and agent. A cold local smoke test for this guide completed in 278.7 > seconds with a 1/1 score; machine and network speeds vary. ## 0:00 — Check the two prerequisites Start Docker and confirm that the daemon is reachable: ```bash docker info >/dev/null ``` Install the [Codex CLI](https://github.com/openai/codex), sign in with your ChatGPT subscription, and confirm the saved login: ```bash codex login codex login status ``` If you already use Codex, the login command normally opens no new setup work. ## 1:00 — Install BenchFlow [`uv`](https://docs.astral.sh/uv/) can install BenchFlow and provision the required Python 3.12 runtime in one command: ```bash uv tool install --python 3.12 --upgrade benchflow bench --version ``` If `uv` reports that the `bench` or `benchflow` executable already exists, repeat the install with `--force` to replace the stale entrypoint. ## 2:00 — Run the real task locally Copy this command as-is: ```bash bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/3d-scan-calc \ --agent codex \ --model gpt-5.5 \ --sandbox docker \ --concurrency 1 \ --jobs-dir jobs/first-local-run ``` BenchFlow now performs the entire evaluation lifecycle: 1. fetches the real task package and builds its local Docker image; 2. makes your saved Codex login available to the agent in the sandbox; 3. lets the agent inspect the STL and create `mass_report.json`; 4. runs the task's verifier outside the agent's control; and 5. saves the score, timings, token usage, and full ACP trajectory. The default skill mode is `no-skill`, so this first run measures the base agent. You can compare the task's bundled mesh-analysis skill later with `--skill-mode with-skill`. ## Read the result A completed run ends with a summary like this: ```text ✓ Score: 1/1 (100.0%), mean reward 1.00, errors=0 Artifacts: jobs/first-local-run/ ``` The benchmarked agent is not guaranteed to pass. A 0/1 score with no execution error still means BenchFlow ran the agent and verifier successfully; it means the agent's answer did not satisfy the task. Summarize the saved run from the CLI: ```bash bench eval list jobs/first-local-run bench eval metrics jobs/first-local-run ``` The important files are: ```text jobs/first-local-run// summary.json 3d-scan-calc__/ result.json timing.json prompts.json trajectory/acp_trajectory.jsonl verifier/reward.txt verifier/test-stdout.txt ``` - `summary.json` gives the job-level pass rate, elapsed time, token totals, and telemetry coverage. - `result.json` is the quickest per-task record of reward, errors, tool calls, and token usage. - `trajectory/acp_trajectory.jsonl` records the agent and tool interaction. - `verifier/reward.txt` and `test-stdout.txt` explain the score. Some agent/provider combinations also write `trajectory/llm_trajectory.jsonl`; subscription-backed ACP agents can report usage without producing that optional provider-level trace. Use a different `--jobs-dir` for an independent second trial. Reusing `jobs/first-local-run` intentionally resumes the existing run and skips work that is already complete. ## Use Claude or Gemini instead Keep the task, Docker, and output flags unchanged, and replace the agent/model pair after signing in: | Login | Flags | |---|---| | Claude Code | `--agent claude --model claude-sonnet-4-6` | | Gemini CLI | `--agent gemini --model gemini-3.1-pro-preview` | For API keys, CI credentials, and provider-hosted models, see [Authentication](/docs/benchflow/authentication). For batches and pinned dataset runs, continue to [Running evaluations](/docs/benchflow/running-evaluations). ## Reproduce the documentation smoke test from source The repository keeps a versioned copy of this real task so maintainers can test the guide without depending on a fresh remote clone: ```bash uv sync --extra dev --locked uv run bench eval run \ --tasks-dir docs/examples/task-md/real-skillsbench/3d-scan-calc \ --agent codex \ --model gpt-5.5 \ --sandbox docker \ --concurrency 1 \ --jobs-dir jobs/docs-start-in-5-minutes ``` The documented smoke run used local Docker, made nine tool calls, received a 1.0 verifier reward, and recorded complete timing and token metadata. --- ## /docs/benchflow/task-authoring BenchFlow authors tasks in the native `task.md` package format. A task is one Markdown document with YAML frontmatter plus sidecar directories for the sandbox, verifier, and optional oracle. ```text tasks/my-task/ ├── task.md ├── environment/ │ └── Dockerfile ├── verifier/ │ ├── test.sh │ └── test_outputs.py └── oracle/ └── solve.sh ``` Start every new task with the native scaffold: ```bash bench tasks init my-task bench tasks check tasks/my-task ``` The full authoring guide lives in [Authoring native task.md tasks](/docs/benchflow/task-authoring-task-md), and the normative schema lives in [Task standard](/docs/benchflow/task-standard). ## Existing Split Packages BenchFlow can still read and migrate older split packages so existing datasets have a direct upgrade path. Do not start new tasks in that layout. ```bash bench tasks migrate tasks/old-task --remove-legacy bench tasks check tasks/old-task ``` `--remove-legacy` promotes `tests/` to `verifier/`, promotes `solution/` to `oracle/`, and removes the old split entrypoint after the generated `task.md` round-trips successfully. If you need to publish a compatibility artifact for another runner, export from the native package instead of hand-authoring the old layout: ```bash bench tasks export tasks/my-task exported/my-task ``` --- ## /docs/benchflow/task-authoring-task-md A native BenchFlow task is one `task.md` document plus sidecar directories. The YAML frontmatter carries the task configuration; the markdown body **is** the prompt. This page teaches the native format hands-on. For the normative standard see [the task standard](/docs/benchflow/task-standard). When a directory contains both layouts, `task.md` is the authoritative task definition — the runtime selects it and ignores the split pair. --- ## Minimal task — three files ```text my-task/ ├── task.md # config frontmatter + prompt body ├── environment/ │ └── Dockerfile # sandbox image └── verifier/ └── test.sh # verifier entry point ``` That is the complete runnable surface: structural validation requires `task.md`, an `environment/` directory with a `Dockerfile`, and a verifier directory with a runnable entrypoint. An `oracle/` directory is optional. ```markdown --- agent: timeout_sec: 300 # strongly recommended — unset means no wall-clock cap verifier: timeout_sec: 120 sandbox: cpus: 1 memory_mb: 2048 --- Create a file `/app/hello.txt` containing exactly `Hello, world!`. ``` ```bash #!/bin/bash # verifier/test.sh REWARD=0 if [ "$(cat /app/hello.txt 2>/dev/null | tr -d '\n')" = "Hello, world!" ]; then REWARD=1 fi echo "$REWARD" > /logs/verifier/reward.txt ``` Scaffold this shape with the CLI (task.md is the default format): ```bash bench tasks init my-task # task.md, environment/, verifier/, oracle/ bench tasks check tasks/my-task # structural validation bench tasks check tasks/my-task --level schema # frontmatter + prompt parse only ``` --- ## Frontmatter `task.md` must start with a `---`-delimited YAML frontmatter block, and the frontmatter must be a mapping — a document without it fails to parse. The keys fall into three classes. **Task config keys** are the BenchFlow task config surface, validated as `TaskConfig`. Unknown keys are **rejected** (the schema is `extra="forbid"`), so typos fail at parse time instead of becoming silently-ignored config: | Key | Meaning | |---|---| | `schema_version` (alias `version`) | Config schema version, currently `"1.3"` | | `task` | Package identity: `name` (`org/name` format), `description`, `authors`, `keywords`, `version` (informational Harbor 1.3 field, stored verbatim) | | `metadata` | Freeform mapping — difficulty, category, tags, anything descriptive | | `agent` | Agent run policy: `timeout_sec`, `user`, `network_mode`, `allowed_hosts` | | `verifier` | Verifier run policy: `timeout_sec` (default 600), `env`, `user`, `service`, … | | `sandbox` | Sandbox: `docker_image`, `cpus`, `memory_mb`, `storage_mb`, `network_mode`, `env`, `workdir`, … (legacy `task.toml` imports convert the Harbor `environment` table to this key; `environment:` in `task.md` is rejected with a rename hint) | | `oracle` | Oracle run policy: `env`, `timeout_sec` (import alias: `solution`) | | `source`, `artifacts`, `steps`, `multi_step_reward_strategy`, `reward` | Provenance, artifact, and reward metadata | `agent.timeout_sec` is **strongly recommended**: it is optional and defaults to unset, and a task that omits it runs the agent with no wall-clock cap unless the caller supplies a per-run timeout. Set it on every published task. Declaring both `oracle` and the legacy `solution` alias in one config is invalid and rejected; native tasks use `oracle`. **Document orchestration keys** are parsed by `TaskDocument`, not `TaskConfig`: `agents` (named roles with `agent`, `model`, `reasoning_effort`, `capabilities`, …), `scenes` (ordered turns referencing declared roles — a turn that names an undeclared role is a parse error), and `user` (simulated user). `benchflow` is the reserved extension namespace. **Authoring shorthands** are expanded during parsing and never reach the canonical config under their short names: | Shorthand | Expands to | |---|---| | `name: hello-world` | `task.name: benchflow/hello-world` (a `/` in the value keeps your org) | | `image: ubuntu:24.04` | `sandbox.docker_image: ubuntu:24.04` | | `verifier: verifier/` (string form) | `benchflow.verifier.path` / `.spec` / `.entrypoint` defaults | | `oracle: oracle/` (string form) | `benchflow.oracle.path` | | `profile: code-change` | Merges a named defaults bundle (see below) | Profiles (`profile:` / `profiles:`) merge predefined default bundles — `code-change`, `harbor-compatible`, `reward-kit`, `acceptance-live`, `multi-agent`, `leaderboard-local` — under your explicit keys; an unknown profile name is a parse error. `bench tasks normalize ` prints the fully expanded canonical document (`--write` replaces `task.md` in place), so a minimal authored file and its canonical form never drift apart. --- ## Prompt body and prompts/ sidecars The body below the frontmatter is the base prompt — free-form markdown, no heading ceremony required. If the body contains no reserved section headings, the entire body is the instruction the agent receives. Four reserved headings are recognized for compatibility imports: `## prompt`, `## role:`, `## scene:`, and `## user-persona`. Repeating the same section heading is a parse error. `bench tasks init` scaffolds a single `## prompt` section as a starting point — for a single-prompt task that is equivalent to a bare body, so keep it or drop the heading as you prefer. The multi-prompt headings (`## role:`, `## scene:`, `## user-persona`) are for compatibility imports only; new multi-prompt material belongs in sidecar files under `prompts/`: | File | Meaning | |---|---| | `prompts/role..md` | Role prompt — the whole file body is the prompt text | | `prompts/scene..md` | Scene prompt | | `prompts/user-persona.md` | Simulated-user persona | Sidecar files take precedence over a reserved heading of the same name, so a compat-imported task can be cleaned up incrementally. Runtime prompt precedence for a turn is: inline turn prompt, then scene prompt, then role prompt, then base prompt. A multi-role task wires the pieces together in frontmatter: ```yaml agents: roles: solver: agent: claude-agent-acp scenes: - name: solve turns: - role: solver ``` with the solver guidance, if any, in `prompts/role.solver.md`. See [docs/examples/task-md/](https://github.com/benchflow-ai/benchflow/blob/main/docs/examples/task-md/README.md) for runnable examples, including real converted SkillsBench packages. --- ## Verifier package and strategy declaration The native verifier directory is `verifier/`. At verify time the directory is uploaded into the sandbox at `/verifier`, and the verifier must write its reward to `/logs/verifier/reward.txt` (and optionally `/logs/verifier/reward.json`). A plain `verifier/test.sh` is a complete verifier: with no other declaration, the runtime executes it directly. Write a float `0.0`–`1.0` to `/logs/verifier/reward.txt`, then exit `0`; a nonzero exit means verifier infrastructure failure, not a scored task failure. To declare *how* the task is scored, add `verifier/verifier.md`. Its frontmatter must contain a `verifier:` mapping with at least one entry under `strategies`; `default_strategy` selects which one runs (it defaults to the first declared strategy and must name a declared one): ```markdown --- document_version: "0.3" verifier: name: my-task-verifier default_strategy: deterministic strategies: deterministic: type: script command: ./test.sh outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json --- ## verifier intent What the verifier measures and which task outputs it reads. ``` Five strategy types are recognized, each with fail-closed required fields: | `type` | Required config | Notes | |---|---|---| | `script` | `command` | Runs as `cd /verifier && `; local script files named in the command must exist in `verifier/` | | `llm-judge` | `rubric` | Optional `model`, `input_dir`, and `context` *or* `context_file` (not both) | | `reward-kit` | `root` | Optional `entrypoint` (default `reward.py`) and `criteria`; paths must be safe-relative | | `agent-judge` | `role`, `isolation: verifier-only`, `inputs` | `role` must match a `## role:` section in the verifier.md body | | `ors-episode` | `inputs` | Optional `format`: `json`, `jsonl`, or `auto` | An unknown `type` is a parse error. `bench tasks check` also verifies the selected strategy is actually runnable — e.g. a `script` strategy whose referenced files are missing, or an `llm-judge` strategy whose rubric file does not exist, fails validation. `outputs` declares the reward artifact contract (defaults shown above; `details_json` and `aggregate_policy` are optional). `bench tasks check --level publication-grade` additionally requires the native package shape: `task.md`, native `oracle/`, `verifier/verifier.md` with rubric files, and an explicit `reward_json` output contract. --- ## Oracle `oracle/solve.sh` is the held-out reference solution (`solution/` is the legacy alias; `oracle/` wins when both exist). Native oracles are uploaded to `/oracle` in the sandbox (legacy `solution/` to `/solution`) and run instead of an agent with `--agent oracle`: ```bash bench eval run --tasks-dir tasks/my-task --agent oracle --sandbox docker ``` A correct task scores `1.0` on its oracle run before any model sees it. --- ## Multi-container tasks A task may ship an `environment/docker-compose.yaml` alongside the `Dockerfile`. The agent always runs in the `main` service; any additional services you declare become sibling containers on the same Docker network. This supports vulhub-style CVE tasks where the agent attacks a separate target container over the network. > `environment/Dockerfile` is always required — `bench tasks check` rejects a > task that ships only a `docker-compose.yaml`. If your `main` service uses a > prebuilt `image:` and needs no build context, still include a minimal > `Dockerfile` (e.g. `FROM `) so structural validation and other > tooling agree on the task package shape. ```yaml # environment/docker-compose.yaml services: main: {} # agent container — BenchFlow injects build/image/limits target: # vulnerable service the agent must exploit image: vulhub/struts2-s2-001:latest expose: ["8080"] ``` `main` reaches `target` by service name (`http://target:8080`). The verifier can inspect *target-side* state — not just the agent's workspace — by passing a `service` argument when running commands: ```python # In a Python-driven run or pre/post hook await env.exec_in_service("target", "test -f /tmp/exploit_proof.txt") await env.exec("cat /flag", service="target") # equivalent form services = await env.inner.services() # ["main", "target"] ``` `exec(..., service=...)` works on the Docker sandbox and the Daytona DinD (compose) sandbox. Single-container backends (Modal, direct Daytona) raise a clear error for any non-`main` service. This lets a verifier check write-based oracles (`/tmp/exploit.txt` in the target), database modifications, or RCE markers without trusting the agent container. ### Target-side verifier with `verifier.service` For tasks whose success oracle lives in a target container — an RCE marker file, a modified database row — point the `verifier/test.sh` verifier at that service with the `service` key under `verifier` in the frontmatter: ```yaml verifier: service: target # run verifier/test.sh inside the `target` container ``` With this set, BenchFlow uploads the task's `verifier/` directory into the **target** container, runs `test.sh` there, and copies the resulting `reward.txt` / `reward.json` back to the host. `service` defaults to `"main"` (the agent container), so single-container tasks are unaffected. `verifier.service` is the declarative, task-schema way to do cross-container verification; the `env.exec_in_service(...)` Python API above is the imperative equivalent for hook-driven runs. > Use the same `service` name you declared in `docker-compose.yaml`. A > `test.sh` running in the target reaches `main` (and vice versa) by service > name over the Docker network, just like the agent does. ### Hardening policy for multi-container tasks BenchFlow's pre-verification hardening — killing the sandbox user's processes, scrubbing `PATH`/`PYTHONPATH`, restoring build-config files — applies **only to the `main` (agent) container**. Target containers are deliberately left unhardened: a vulhub-style target is *meant* to be vulnerable, the agent never has a shell inside it, and hardening it would risk breaking the very vulnerability the task exercises. `verifier.service` selects where `test.sh` *runs*; it does not move hardening off `main`. --- ## Migrating a legacy task `bench tasks migrate` converts a `task.toml` + `instruction.md` pair into `task.md`: ```bash bench tasks migrate tasks/my-task # writes task.md, keeps legacy files bench tasks migrate tasks/my-task --overwrite # replace an existing task.md bench tasks migrate tasks/my-task --remove-legacy # delete the split pair and # promote tests/ -> verifier/, # solution/ -> oracle/ ``` The migration is non-destructive by default and refuses to write anything lossy: the generated document is re-parsed and must reproduce the original config semantics and instruction text exactly, or the command fails. Unknown `task.toml` keys that the schema does not model are preserved under `benchflow.compat` in the generated frontmatter rather than dropped. After migrating, validate the result: ```bash bench tasks check tasks/my-task bench eval run --tasks-dir tasks/my-task --agent oracle --sandbox docker ``` ## Compatibility Export To produce a compatibility split package from a `task.md` package, use `bench tasks export`: ```bash bench tasks export tasks/my-task out/my-task-split bench tasks export tasks/my-task --report-only # loss report only ``` The export writes a compatibility loss report to `compatibility/export-report.json` so you can see what (if anything) the split layout cannot represent. Publication-grade validation requires `task.md` to be the only authoritative entrypoint, so keep exported split layouts in a separate output directory rather than beside `task.md`. See [CLI reference: bench tasks export](/docs/benchflow/reference/cli#bench-tasks-export) for all flags. --- ## /docs/benchflow/task-standard Status: current task package standard (2026-06-14) This document defines the direction for BenchFlow-native task packages. The short version: `task.md` is the native authoring entrypoint; `oracle/` and `verifier/` are the BenchFlow-native names for held-out reference behavior and reward checks. Split-layout names such as `solution/` and `tests/` remain compatibility names for migration and export only. The standard is intentionally split into three views: | View | Purpose | Owner | |---|---|---| | Authoring document | What humans and generators write: one `task.md` plus sidecar dirs | `TaskDocument` | | Runtime task view | What rollout, verifier, hardening, provenance, and trajectories consume | `TaskRuntimeView` plus first `TaskPackage` boundary | | Foreign adapter view | What external benchmark formats and hosted environments import/export | adapters | Do not treat these as the same interface. A good authoring document can include more information than a foreign format can export, and a foreign import can preserve unknown data that native authoring would reject. ## Goals and scope A BenchFlow task is one `task.md` that selects a mode on each of three planes (see *Planes* below): how the environment is built, how the agent interacts, and how the result is scored. The standard has three goals: 1. **Native authoring** — humans and generators write one `task.md` (plus sidecar dirs) as the primary surface. 2. **Interoperability** — existing split-layout formats (`task.toml` + `instruction.md` + `solution/` + `tests/`) import directly, and packages export back to that layout with an explicit, honest loss report when a native concept has no equivalent. 3. **Coverage** — the schema can express the full range of eval shapes (single-shot, multi-round, simulated-user, multi-agent, and live-arena interaction; workspace, trajectory, rubric, judge, and leaderboard reward), even where the runtime for a given mode lands in a later milestone (see *Open Primitives and Roadmap*). Native authoring is the priority surface; import is best-effort-faithful; export is best-effort with a loss report. The standard does not require bidirectional-lossless round-tripping with any one external format. ## Field discipline Every normative standard field must map to a mode on one of the three planes (below) or to a concrete BenchFlow runtime need. Fields without that mapping belong under `metadata` or in an adapter, not in the standard. ## Native Layout ```text task/ |-- task.md |-- environment/ | `-- Dockerfile |-- verifier/ | |-- verifier.md | `-- test.sh |-- oracle/ | `-- solve.sh `-- evidence/ `-- validation.json ``` Compatibility aliases: | Native | Compatibility / foreign export | Meaning | |---|---|---| | `task.md` | `task.toml` + `instruction.md` | Task config plus prompt material | | `oracle/` | `solution/` | Held-out reference implementation | | `verifier/` | `tests/` | Verifier package, reward code, hidden checks, rubrics, and judges | Target validation: native packages may carry both native and compatibility directories only when hashes prove the duplicate content is equivalent. Current runtime prefers the native spelling when both aliases exist; it does not yet prove equivalence. Within a BenchFlow-native package, selection is fail-closed: - if `task.md` exists, it is the authoritative task definition - if `verifier/` exists, it is the authoritative verifier directory; an empty or invalid `verifier/` does not fall back to `tests/` - if `oracle/` exists, it is the authoritative oracle directory; an empty or invalid `oracle/` does not fall back to `solution/` - duplicate alias trees must be byte-identical after normalized traversal or validation should report a collision Split layouts are compatibility inputs and export artifacts, not the native authoring surface. New BenchFlow tasks should publish `task.md` as the only authoritative entrypoint. ## Versioning There are two versions: ```yaml schema_version: "1.3" # BenchFlow task config surface benchflow: document_version: "0.6" # BenchFlow task.md document syntax ``` `schema_version` is for the runtime config model shared with compatibility imports. `benchflow.document_version` is for document-only concepts such as teams, prompt composition, agent policy, runtime policy, private assets, provenance, evidence, nudges, and export policy. ## Root Frontmatter The root frontmatter has three classes of keys. BenchFlow config keys are modeled by `TaskConfig` and must be rejected when unknown in native authoring mode: - `schema_version` / `version` - `task` - `metadata` - `agent` - `verifier` - `sandbox` (legacy `task.toml` import spelling: `environment`) - `oracle` (validation alias: `solution`) - `source` - `artifacts` - `steps` - `multi_step_reward_strategy` Document orchestration keys are parsed by `TaskDocument`: - `agents` - `scenes` - `user` BenchFlow extension keys live under the reserved namespace: - `benchflow` Do not add new root keys for every new idea. Put draft or BenchFlow-specific extensions under `benchflow:` until they have a stable interface. Native authoring should use `oracle`. Importers may accept the compatibility `solution` alias, but a config that contains both names is invalid. ## Prompt Body The `task.md` body **is** the base prompt — free-form markdown, exactly like a `SKILL.md` body. No `## prompt` heading is required: if the body carries no reserved section headings, the entire body is the prompt. The common single-shot task is just frontmatter plus prose, so a bespoke benchmark ports by dropping its existing instruction text in as the body with no markup. Tasks that need more than a base prompt — multiple roles, multiple scenes, or a simulated-user persona — author each as its **own** free-form file under `prompts/`, so no body ever carries reserved-heading ceremony: | File | Meaning | |---|---| | `prompts/role..md` | Role prompt | | `prompts/scene..md` | Scene prompt | | `prompts/user-persona.md` | Simulated user persona | Each sidecar file is itself a clean free-form body. For backward compatibility, single-prompt source formats may instead embed the same content in the body via reserved `## prompt`, `## role:`, `## scene:`, and `## user-persona` headings; these import losslessly and normalize to the file layout. Sidecar files take precedence over a heading of the same name. New tasks should prefer the files. Default runtime precedence remains a simple fallback: 1. inline turn prompt 2. scene prompt 3. role prompt 4. base prompt Native tasks can make composition explicit: ```yaml benchflow: prompt: composition: append order: [base, role, scene, turn] ``` The first `TaskPackage` prompt plan now compiles `append` and explicit `replace` policies deterministically. That avoids losing role guardrails when a scene prompt is present. `RolloutConfig` consumes the compiled plan for task.md scene execution when explicit CLI/SDK prompts do not override the document. ## Agent And Runtime Policy Agent isolation is distinct from task environment networking. A no-search task can still need dependency, LLM, or provider egress outside the sandbox; a no-network task can still need the agent harness to call its model. ```yaml benchflow: agent_policy: skill_access: none # none | installed | declared allowed_skill_roots: [] search: disabled # allowed | disabled forbidden_tools: [web_search, browser_fetch] trajectory_audit: require_no_forbidden_tool_calls: true runtime_policy: backend: modal # docker | daytona | modal | kubernetes | podman | hpc | queue required_capabilities: [gpu:B200, private_mounts, persistent_state] network: task_default: no-network agent_egress: model-and-dependencies allowed_hosts: [] phase_overrides: verifier: no-network private_mounts: - source: modal-volume://org-models/qwen target: /mnt/models/qwen mode: ro visibility: agent secret_ref: org-models-readonly persistent_state: required: true scope: task-run cleanup: after-verifier ``` Unsupported `agent_policy`, `runtime_policy`, private mounts, registry secrets, GPU types, phase-specific network overrides, or persistent state semantics must fail closed before launch for the selected sandbox. No-search proof must come from both launch policy and trajectory audit; it is not implied by `network_mode: no-network`. ## Planes The package has six planes. Keeping them separate is the core abstraction. | Plane | Owns | Should not own | |---|---|---| | Package | identity, versions, provenance, source hashes, export policy | sandbox execution | | Runtime | environment image, resources, network policy, setup/reset/readiness, mounts | prompt orchestration | | Interaction | agents, roles, teams, scenes, turns, user/nudge loops, handoff policy | verifier checkpoints | | Verifier | verifier package entrypoint, reward file contract, hidden fixtures, scorer type, rubrics, judge roles, separate verifier env | agent prompt text | | Oracle | held-out reference implementation and oracle-only env | tests/verifier code | | Evidence | validation runs, flake data, anti-cheat review, artifact hashes, leaderboard metadata | mutable task source | Imported `steps` are verifier/runtime checkpoints. BenchFlow `scenes` are interaction checkpoints. They are orthogonal. A task can have both, but a runtime must define how they compose before executing them. Of the six planes, three are the **authoring axes** an author selects a mode from — Runtime (the environment), Interaction, and Verifier — while the remaining three (Package, Oracle, Evidence) are supporting planes. A task is one mode-selection per axis: `environment` × `interaction-mode` × `verifier-strategy`. New normative fields must add or compose a mode on one of these axes; otherwise they belong in `metadata` or an adapter. ## Presets And Profiles Two different reuse mechanisms were historically both called "profile," which is why the mental model felt overloaded. v0.6 separates them: - **`preset`** — *authoring sugar*. A named bundle of defaults that `bench tasks normalize` expands into the canonical contract and then **discards**; it has no runtime existence. Presets are the lightweight-authoring path: a tiny `task.md` becomes a full contract. Examples: `code-change`, `acceptance-live`, `harbor-compatible`. (The current implementation still spells the preset key as `profile:`; M0 renames it to `preset:` and keeps `profile:` as a deprecated alias.) - **`*-profile`** — *runtime-real shared config*. A benchmark-level object that many tasks **reference** and that **persists into `TaskRuntimeView`**, one per authoring axis: - `environment-profile` — a shared world reused across tasks (e.g. one clawsbench service catalog referenced by all 44 tasks) - `agent-profile` — shared harness / model / role wiring - `verifier-profile` — shared reward strategy and rubric Rule of thumb: if removing it changes only how much you typed, it is a `preset`; if removing it changes what runs, it is a `*-profile`. Presets normalize away; profiles are inherited. `*-profile` resolution is M1 runtime work; `preset` exists today. ## Verifier Package The verifier is a peer package, not just a `tests/` directory. Native verifier packages should have their own entry document: ```text verifier/ |-- verifier.md |-- task.toml |-- test.sh |-- rewards/ | |-- correctness/ | | `-- reward.py | `-- quality/ | `-- criteria.toml |-- rubrics/ | |-- verifier.md | `-- verifier.toml |-- judges/ | `-- reviewer.md `-- fixtures/ `-- hidden_cases.jsonl ``` `verifier/verifier.md` is analogous to `task.md` for the evaluation side. It describes what evidence is read, which strategies can score the task, how rubric dimensions compose, which judge agents are allowed, and what outputs the runtime must preserve. `verifier/task.toml` is an optional compatibility projection of `verifier/verifier.md`; it is not a second native surface. If both files exist, their canonical projection must match or validation fails closed. ```md --- document_version: "0.3" verifier: name: hidden-patch-and-quality default_strategy: deterministic strategies: deterministic: type: script command: ./test.sh rewardkit: type: reward-kit root: rewards/ criteria: rewards/quality/criteria.toml judge: type: agent-judge role: verifier_judge model: gpt-5.5 inputs: [trajectory/acp_trajectory.jsonl, /logs/artifacts/patch.diff] isolation: verifier-only ors: type: ors-episode reward_aggregation: last_non_null terminal_policy: finished_true rubric: combine: weighted_sum dimensions: correctness: {weight: 0.7, source: deterministic} maintainability: {weight: 0.2, source: judge} evidence_quality: {weight: 0.1, source: rewardkit} files: human: rubrics/verifier.md structured: rubrics/verifier.toml outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json details_json: /logs/verifier/reward-details.json aggregate_policy: field: reward fallback: weighted_mean --- ## role:verifier_judge Grade only the submitted artifact and declared evidence. Do not infer intent from private oracle files or hidden verifier fixtures. ``` `test.sh` is the minimum executable strategy and the compatibility export target. Reward Kit-style criteria, ORS-style episode rewards, and AgentBeats-style assessor agents are verifier strategies. Runtime adapters may write declared evidence artifacts such as `trajectory/ors-rewards.jsonl`, but the ORS-specific judge/normalization semantics stay inside verifier scope. They must not leak into agent prompts, and they must emit either the canonical reward envelope or a declared multi-metric map. Verifier packages must be isolated: - judge models and credentials are verifier-scoped, not agent-visible - judge prompts live under `verifier/`, not inside the task prompt - rubrics are separate from judge persona: `rubrics/verifier.md` is the human scoring contract, `rubrics/verifier.toml` or JSON is the structured scoring contract, and `## role:verifier_judge` is the assessor posture - hidden fixtures and rubrics are mounted only during verifier execution - agent-as-judge trajectories are recorded as evidence and audited for input leakage - unsupported verifier strategies fail closed before scoring starts ## Verifier Contract Native verifier code lives in `verifier/` and is mounted at `/verifier`. Compatibility `tests/` remains supported and is mounted at `/tests`. The minimum script verifier contract applies to the selected verifier entrypoint: native packages with no `verifier/verifier.md` run `verifier/test.sh`; imported split-layout packages may run `tests/test.sh`. When `verifier/verifier.md` is present, structural verifier validity follows the selected strategy. A package can therefore be valid without `verifier/test.sh` if, for example, the selected Reward Kit runner and criteria files exist. - write `/logs/verifier/reward.txt` with one float from `0.0` to `1.0` - optionally write `/logs/verifier/reward.json` with structured rubric/evidence; BenchFlow preserves structured rewards, keeps `reward.txt` as scalar compatibility, and can compute a scalar from declared aggregate policies - prefer exit `0` after writing reward - treat nonzero verifier exit without a fresh reward file as infrastructure failure; a nonzero exit with a fresh reward is a scored task result with verifier diagnostics The richer standard should model verifier inputs explicitly: ```yaml verifier: type: test-script timeout_sec: 900 benchflow: verifier: entrypoint: verifier/test.sh visibility: hidden inputs: - path: verifier/test.patch kind: test_patch visibility: hidden_verifier outputs: reward_text: /logs/verifier/reward.txt reward_json: /logs/verifier/reward.json transfer: agent_to_verifier: - /logs/artifacts/** verifier_to_result: - /logs/verifier/reward.* - /logs/verifier/artifacts/** ``` This covers `tests/test.patch`, Windows entrypoints, artifact-only graders, separate verifier images, and hidden fixtures without overloading the directory name. Separate verifier execution must define image resolution, hidden fixture mounting, step-level verifier environments, and transfer rules. Hidden verifier inputs such as `tests/test.patch` or `verifier/test.patch` are mounted only for the verifier phase; agent logs and workspace files move into verifier scope only through declared artifact transfer paths. Target reward precedence: - `reward.txt` remains the scalar compatibility minimum - when `reward.json` exists, it should be the authoritative rich reward artifact - `reward.json` may be an envelope with a numeric `reward`, or a reward-kit style multi-metric map with `metrics` plus a declared `aggregate` policy - if both files exist and `reward.json` has a scalar aggregate, the scalar must match `float(reward.txt)` or validation should fail closed - if `reward.json` is a multi-metric map without a scalar `reward`, verifier metadata may declare the aggregate policy used for scalar exports; current runtime computes `reward` for `mean`, `weighted_mean`, and `weighted_sum`, and `reward.txt` may still carry the scalar compatibility value - `reward-details.json` should be preserved when present - reward artifacts should preserve structured reserved keys such as `rubric`, `items`, `evidence`, `artifacts`, `metadata`, `reason`, `reasons`, `errors`, and task-specific payloads such as `metrics`, `regressions`, `participants`, `winner`, `raw`, and `debug` Current runtime now prefers `reward.json` over `reward.txt`, rejects disagreeing scalar outputs, and preserves a first set of structured reward fields. `src/benchflow/task/verifier_document.py` parses `verifier/verifier.md` strategies, rubric metadata, output contracts, and verifier-scoped role prompts. When `verifier/verifier.md` is present, `Verifier.verify()` selects its default strategy: `script` runs the declared command relative to the uploaded verifier directory, `llm-judge` uses the existing deliverables judge with verifier-local rubric, model, input directory, and context/context-file overrides, `reward-kit` runs a safe relative `reward.py` package runner inside verifier scope, writes a `reward-kit-manifest.json` contract, and, when criteria are declared, parses those criteria before launch and computes/verifies canonical `reward` from matching `reward.json.metrics`. `agent-judge` runs a verifier-scoped judge role over declared evidence inputs. ORS runtime helpers can normalize tool-output rewards into `trajectory/ors-rewards.jsonl`; `ors-episode` then reads declared ORS reward evidence, normalizes reward responses or event streams through the existing ORS adapter, and emits canonical `reward.json` plus `reward-details.json`. `reward-details.json` is a named rollout artifact, stale copies are cleared before verification, script verifiers can preserve it, target-service verifiers download it with the rest of `/logs/verifier`, and the built-in LLM judge emits criterion details there. Metrics-only `reward.json` maps can now use the verifier document's `outputs.aggregate_policy` or selected Reward Kit criteria policy to compute and persist the canonical `reward`. It still does not fully match the target: full Reward Kit parity, full OpenReward environment import/export, and AgentBeats assessor lifecycles are not yet first-class; selected unsupported strategies fail closed. Validation evidence should include parser checks, legacy-vs-native migration parity, live rollout artifacts, negative-contract failures, and explicit fail-closed results for parsed fields that the selected runtime cannot honor. ## Oracle Contract Native oracle code lives in `oracle/` and is mounted at `/oracle`. Compatibility `solution/` remains supported and is mounted at `/solution`. The naming change is semantic: - `oracle` means "held-out reference behavior used to prove solvability" - `solution` is a compatibility export name for older split layouts Proposed outbound exporters should map native `oracle/` to foreign `solution/`. Current runtime supports native and compatibility oracle paths, and the first split-layout exporter maps native `oracle/` to `solution/`. Publication-grade oracle equivalence enforcement remains target behavior. ## Assets, Provenance, And Evidence Native task packages need more than source code. The standard should track assets as first-class objects: ```yaml benchflow: provenance: images: - field: sandbox.docker_image reference: ghcr.io/org/task-image:2026-06 digest: sha256:... registry: ghcr.io provider: modal build_source: path: environment/Dockerfile sha256: "" credential_secret_ref: task-image-pull never_persist_credentials: true assets: - path: environment/dataset.parquet visibility: agent sha256: "" source: url: https://huggingface.co/datasets/org/name revision: "" license: apache-2.0 generated: false mount_phase: agent - path: verifier/hidden_cases.jsonl visibility: hidden_verifier sha256: "" mount_phase: verifier secrets: - name: task-image-pull scope: image-pull visibility: runtime_secret never_persist: true ``` Suggested visibility values: - `agent` - `hidden_verifier` - `hidden_oracle` - `runtime_secret` - `external_dataset` - `evidence_only` Evidence should prove task validity, not just package shape: ```yaml benchflow: evidence: oracle_runs: required_reward: 1.0 last_job: jobs/task-standard/oracle/... artifact: evidence/calibration/oracle-run.json verifier: reruns: 5 flake_rate: 0.0 report: evidence/calibration/verifier-stability-report.json review: anti_cheat: passed instruction_alignment: passed reviewer: benchflow artifact: evidence/calibration/review.json calibration: no_op_reward_max: 0.0 known_bad_reward_max: 0.2 partial_solution_range: [0.3, 0.8] report: evidence/calibration/calibration-report.json human_or_reference_examples: - name: gold expected_reward: 1.0 artifact: evidence/calibration/gold-result.json judge_agreement: required: true sample_count: 5 min_pairwise_agreement: 0.8 trajectories: - path: trajectory/acp_trajectory.jsonl kind: acp visibility: evidence_only sha256: "" - path: trajectory/critique.jsonl kind: critique visibility: evidence_only sha256: "" artifacts: - path: evidence/calibration/oracle-run.json kind: oracle_run visibility: evidence_only sha256: "" - path: evidence/calibration/verifier-stability-report.json kind: verifier_stability visibility: evidence_only sha256: "" - path: evidence/calibration/review.json kind: acceptance_review visibility: evidence_only sha256: "" - path: evidence/calibration/calibration-report.json kind: calibration_report visibility: evidence_only sha256: "" - path: evidence/calibration/gold-result.json kind: reference_result visibility: evidence_only sha256: "" - path: artifacts/session.har kind: har mime_type: application/json produced_by: browser visibility: evidence_only sha256: "" ``` Borrow the discipline, not the package shape, from METR-style task standards: - keep BenchFlow document-first rather than Python `TaskFamily`-first - make asset visibility explicit instead of relying on directory folklore - declare required secrets/resources with scope and "never persist" semantics - record oracle/no-op/partial/human baseline scores where available - keep protected/intermediate scoring behind explicit visibility controls Valid evidence artifact kinds include `screenshot`, `video`, `har`, `browser_trace`, `trajectory`, `critique`, `viewer_metadata`, and `verifier_artifact`. Oracle and calibration evidence is required for leaderboard-grade tasks and recommended for all native tasks with hidden tests, subjective rubrics, or LLM/agent-as-judge scoring. ## Interaction Model An interaction declares one **mode**. The modes form the Interaction axis: | Mode | Meaning | Runtime | |---|---|---| | `single-shot` | one agent, one prompt, scored once | executable | | `multi-round` | oracle access / progressive disclosure across rounds (`BaseUser`) | executable | | `simulated-user` | a declarative, model-backed user persona drives turns (not a live wire peer) | partial (linear; one active role per scene) | | `multi-agent-sequential` | roles hand off in one shared workspace | partial (sequential handoff only) | | `arena-concurrent` | an assessor agent and the agent-under-test run as live A2A+MCP peers, scored *during* the interaction | target (runtime-deferred; see Open Primitives) | `arena-concurrent` models a live interaction where the reward is produced *during* the exchange (by an assessor agent) rather than by a post-hoc verifier. It is declared in the schema so such tasks parse without loss; its runtime (an A2A bridge for the agent-under-test plus a concurrently-running assessor) is deferred to a later milestone (see *Open Primitives and Roadmap*). Note ACP (BenchFlow <-> agent) is not A2A (agent <-> agent); arena needs the A2A leg. The current executable slice is linear `scenes` that reference `agents.roles`. The first team handoff subset is intentionally narrow: a document-declared user loop may execute explicit multi-role scene turns sequentially when `benchflow.teams..handoff` declares `mode: sequential`, `workspace_visibility: shared`, and `trajectory_visibility: none|metadata`. ```yaml agents: roles: planner: agent: claude-agent-acp model: claude-sonnet-4-6 implementer: agent: codex-acp model: gpt-5.5 reasoning_effort: high benchflow: teams: default: handoff: mode: sequential workspace_visibility: shared trajectory_visibility: metadata ``` Richer team semantics such as role membership enforcement, summaries, handoff artifacts, parallel teams, branch routing, and full trajectory sharing are parsed as draft surface but must fail closed until a runtime owns them. Simulated users and nudges should be explicit about runtime type: ```yaml user: model: scripted stop_rule: satisfied-or-3-rounds private_facts: hidden_need: reveal only after the solver asks for it benchflow: nudges: mode: simulated-user nudge_budget: 2 ``` This deterministic subset compiles into `RolloutConfig.user` as a `DocumentNudgeUser`: the public prompt runs first, private facts stay out of the package metadata and initial solver prompt, and a fact is revealed only after a targeted clarification question. Bounded model-linear simulated users should stay equally explicit: ```yaml user: model: claude-haiku stop_rule: satisfied-or-5-rounds benchflow: nudges: mode: simulated-user branchable: true branch_execution: option-kinds-preserved confirmation_policy: destructive_actions: human ``` If a runtime cannot compile `user` into a concrete loop, it should fail closed or mark the field metadata-only rather than silently ignoring the user. Today the first model-linear slice accepts `claude-*`, `gpt-*`, and `gemini*`-style models for linear single- or multi-scene simulated users through `ModelDocumentNudgeUser`. `confirmation_policy: human` installs a fail-closed ACP permission handler unless the caller supplies an explicit `on_ask_user` handler, and the ACP `ask_user` bridge preserves both option IDs and option kinds so reject/allow choices are explicit branchable evidence. Authors may spell the current executable branch slice as `branch_execution: option-kinds-preserved`; `branch_execution: forked-snapshot` fails closed until the user loop is integrated with the Environment snapshot branch engine. The first sequential shared-workspace team handoff slice records `scene`, `role`, `handoff_from`, and `handoff_to` metadata per user round. `branchable` is still not automatic branch execution; interactive approval UI, parallel teams, handoff artifacts, full trajectory sharing, and branch/message-routing policy remain fail-closed target work. ## Compatibility Compatibility must be explicit. A native package can guarantee export only for the subset the target format supports. ```yaml benchflow: compatibility: harbor: export: full emits: config: task.toml prompt: instruction.md oracle: solution/ verifier: tests/ ``` Target compatibility rules: 1. Native authoring rejects unknown root config keys. 2. Foreign adapter import preserves unknown `task.toml` keys outside native config. The first implementation is `import_task_config_toml()`: strict native validation still fails on unknown keys, while compatibility import returns a validated `TaskConfig` plus `InboundCompatibility.config_extra`. 3. Legacy-to-native migration writes preserved foreign keys under `benchflow.compat.extra`: ```yaml benchflow: compat: source: harbor extra_paths: - sandbox.modal.image - steps[0].runner - verifier.reward_kit.metric extra: sandbox: modal: image: registry.example.com/task:latest steps: - runner: harbor-step-runner verifier: reward_kit: metric: exact_match ``` 4. Split export rehydrates `benchflow.compat.extra` back into `task.toml` without overwriting supported native keys. The export report records `restored_extension_paths` so compatibility is auditable. 5. `build_harbor_roundtrip_conformance_report()` proves the supported split surface across a split -> `task.md` -> split hop: canonical `TaskConfig`, normalized prompt, and environment/solution/tests file-map hashes. 6. Export emits a degraded-export report when the target format cannot express a native concept. The first implementation is `src/benchflow/task/export.py`, which writes a compatibility split layout plus `compatibility/export-report.json`. 7. Mixed native/legacy files are structurally invalid when task config, prompt, oracle/solution, or verifier/tests aliases drift. 8. Compatibility export should emit `task.toml`, `instruction.md`, `solution/`, and `tests/`. 9. BenchFlow import should prefer `task.md` only when compatibility metadata proves the legacy files are equivalent. 9. Export reports include selected definition, selected verifier/oracle dirs, input/output file hashes, alias collisions, restored foreign extensions, and any lost semantics. 10. `tests/` compatibility is path compatibility, not a separate native standard. For native packages, `verifier/` is authoritative. For split packages, `tests/` remains valid and may contain only `test.sh`. 11. Exporters map the selected native verifier tree to `tests/`; importers may preserve split-layout `tests/` without requiring `verifier.md` or rubric files. If both `verifier/` and `tests/` exist, validation should compare normalized file maps and fail closed on drift. 12. ORS/AgentBeats imports/exports live at the adapter boundary: ORS tool-output rewards become declared reward-event artifacts plus a terminal aggregate, and AgentBeats assessor agents become verifier strategies rather than root task syntax. Round-trip guarantees are semantic, not byte-exact. Config equivalence should be checked through canonical `TaskConfig` dumps; prompt equivalence through normalized prompt text; verifier/oracle equivalence through deterministic SHA-256 file maps over regular files. Comments, TOML/YAML formatting, and blank line trivia are not preserved unless a same-format no-op export asks for that. ## Runtime Capability Matrix Current implementation status: | Feature | Parse | Runtime | Next gate | |---|---:|---:|---| | `task.md` prompt | yes | yes | keep | | native `verifier/` | yes | yes | keep | | native `oracle/` | yes | yes | keep | | verifier `script` strategy | yes | yes | keep | | verifier `llm-judge` strategy | yes | yes | keep | | verifier `reward-kit` strategy | yes | partial | safe relative `reward.py` runner executes; declared criteria parse before launch, emit a runtime manifest, require exact metrics, and compute/verify canonical reward; fuller Reward Kit parity remains target work | | verifier `agent-judge` strategy | yes | partial | verifier-scoped LLM judge over declared inputs; richer ACP-backed judge agents remain target work | | verifier `ors-episode` strategy | yes | partial | runtime helper writes ORS tool-output rewards to `trajectory/ors-rewards.jsonl`; declared reward responses/event streams normalize into `reward.json` and `reward-details.json`; fuller OpenReward environment import/export remains target work | | `agents.roles` | yes | partial | `TaskRuntimeView` carries parsed scenes; explicit sequential shared-workspace handoff can switch roles through the user loop | | `scenes` | yes | partial | prompt composition compiles; multi-role document-user scenes execute only with explicit turns and supported team handoff | | `user` / `## user-persona` | yes | partial | `model: scripted` + string `private_facts` compiles to `DocumentNudgeUser`; bounded model-linear users compile to `ModelDocumentNudgeUser`; linear single- and multi-scene user loops execute when every scene is single-role, or when explicit multi-role turns opt into sequential shared-workspace team handoff; `confirmation_policy: human` gates ACP permissions fail-closed without an explicit handler; `branch_execution: option-kinds-preserved` preserves option IDs and kinds; forked branch execution, interactive approval UI, parallel teams, and rich handoff artifacts fail closed | | `benchflow.teams` | yes | partial | supports exactly one `handoff` with `mode: sequential`, `workspace_visibility: shared`, and `trajectory_visibility: none|metadata`; richer team keys fail closed | | `benchflow:` | raw | no | typed document schema after v0.3 stabilizes | | imported `steps` | yes | no/partial | fail closed per sandbox until implemented | | root/step artifacts | yes | no/partial | implement collection or fail closed | | network allowlist | yes | no/partial | per-sandbox capability check | | separate verifier env | yes | no/partial | materializer plus verifier runner support | | Windows / TPU | yes | no | fail closed | | healthcheck | yes | no/partial | fail closed until sandbox healthcheck support lands | | workdir | yes | partial | absolute non-root paths are materialized; root/relative paths fail closed | | `reward.json` precedence | yes | partial | prefer JSON when present and reject both-present mismatches | | metrics aggregate policy | yes | partial | `mean`, `weighted_mean`, and `weighted_sum`; richer engines remain target work | | `arena-concurrent` interaction (G4) | no | no | add interaction-mode schema now; A2A bridge for the agent-under-test + concurrently-running assessor at M2 | | hybrid reward envelope (G1) | partial | no | declared cross-surface product/sum of factors; M1 | | GAIN aggregation (G2) | no | no | dynamic live baseline + ceiling; M1 | | leaderboard-submission (G5) | partial | no | hosted / hidden external scorer with durable result record; M1 | | RL step-reward (G6) | no | no | per-action environment reaction; M1/M2 | Today `bench tasks check` is structural by default. `bench tasks check --level schema` checks only the task authoring entrypoint and prompt parse, so schema-only fixtures can be validated without pretending to be runnable task packages. `bench tasks check --sandbox ` runs the sandbox-aware capability gate for this matrix, and `bench tasks check --level runtime-capability --sandbox ` names that gate explicitly. `bench tasks check --level publication-grade` adds the first static native publication gate: the package must use `task.md`, native `oracle/`, native `verifier/verifier.md`, rubric files, and selected verifier strategy artifacts with an explicit `reward_json` output contract. Unknown sandbox names fail runtime-capability validation instead of becoming no-op checks. `bench tasks check --level acceptance` adds the first static evidence gate: `benchflow.evidence` must declare oracle proof, verifier reruns and flake rate, a verifier stability report with concrete run records, anti-cheat and instruction-alignment review status, calibration bounds, reference artifacts, a calibration report with no-op/known-bad/partial/reference cases, and SHA-256 pins for every primary evidence file. The static gate parses the declared JSON artifacts and checks that oracle rewards, review status, calibration examples, calibration report cases, and verifier run records agree with the declared metadata. `bench tasks check --level acceptance-live --sandbox ` now adds the first executable live-evidence slice: `benchflow.evidence.acceptance_live.cases` can declare fresh verifier cases and executable `oracle/solve.sh` reruns that pass through the selected sandbox and BenchFlow verifier boundary, with reward thresholds checked from live `reward.txt` / `reward.json` output. `acceptance_live.calibration.from: calibration.report` can also generate live no-op/known-bad/partial/reference cases from the static calibration report; generated low/partial cases require single-line sandbox commands in the report so the checker runs real perturbations instead of treating missing artifacts as expected verifier errors. When `acceptance_live.report` is declared, the runner writes a package-local `acceptance-live-report` JSON artifact plus a `.sha256` sidecar with run records, reward summaries, task/spec hashes, generated/declared case sources, staged workspace hash, and secret-safe diagnostic fields such as `verifier_error_category`, `diagnostic_code`, and `artifact_hint`. Dependency install flakes point to `verifier/test-stdout.txt` instead of embedding raw resolver output in the report. For dogfood against checked-in examples that must not dirty the package, `bench tasks check --level acceptance-live --report-output ` writes the report and sidecar to a host path instead of the package-local path, and `--no-report-write` skips writing the report and sidecar entirely (validation only); the package-local write is reserved for intentionally refreshing checked-in evidence. Repeated live cases can declare `expect.flake_rate_max` to enforce observed flake rate across fresh sandbox reruns; without that field, each failed rerun fails the gate. Larger repeated flake campaigns, model/submission metadata, hosted leaderboard publication, and leaderboard export are still target work. If `acceptance_live.leaderboard.required: true` is declared, the live report includes a local `leaderboard_suitability` verdict requiring live runs, all runs passing, an observed flake rate within `acceptance_live.leaderboard.max_flake_rate`, oracle/reference proof, and generated calibration coverage for no-op, known-bad, partial, and reference cases. A parsed field that the selected runtime cannot honor is worse than a parse error. ## Architecture Slices P0: Add `TaskPackage` / `TaskRuntimeView`. The first runtime-facing slices exist in `src/benchflow/task/runtime_view.py` and `src/benchflow/task/package.py`. They answer: - which entrypoint is authoritative - what prompt goes into `/instruction.md` compatibility materialization - which verifier/oracle directories are native versus legacy - which scenes were parsed for execution - which source hashes and compatibility metadata apply - which verifier document and selected strategy apply - which sandbox runtime issues block launch - which compatibility export report describes target-format loss - which prompt plan composes base, role, scene, and turn prompts for rollout, with redacted document-declared user metadata The remaining package-boundary work is richer adapter import/export state, acceptance/calibration validation levels, and freezing selected verifier/user runtime semantics through launch instead of reparsing at every edge. P1: Add fail-closed capability checks. Rollout, verifier, hardening, and adapters should stop parsing fields they do not execute without surfacing a validation result. This includes explicit gates for `task.md` plus split-file drift, `verifier/` plus `tests/`, and `oracle/` plus `solution/`. The first module is `src/benchflow/task/runtime_capabilities.py` with a pure validator: ```python validate_task_runtime_support(task, *, sandbox, task_dir) -> list[UnsupportedTaskFeature] ``` It reports stable config paths and reasons for unsupported `steps`, root/step `artifacts`, allowlists, separate verifier environments, Windows, TPU, healthchecks, unsafe workdirs, document-only `user`/`benchflow` runtime semantics, and non-`main` verifier services on backends that cannot run them. It is wired into `bench tasks check --sandbox ` and the shared sandbox factory used by rollouts and `Environment.from_task()`. Unsupported parsed semantics now raise `UnsupportedTaskFeatureError` before Docker, Daytona, or Modal construction. Safe absolute non-root `sandbox.workdir` values are materialized before agent and verifier setup. P2: Split native and adapter validation modes. Native authoring should be strict. Foreign import should preserve and warn. P3: Type the `benchflow:` namespace. Start with `document_version`, `compatibility`, `provenance`, `assets`, `secrets`, `evidence`, `teams`, `nudges`, `prompt`, `agent_policy`, and `runtime_policy`. P4: Extend exporters. The first `bench tasks export` path exports to a compatibility split layout with explicit loss reports, backed by `export_task_to_split_layout()`. Extend the same reporting discipline to external benchmark datasets and same-format no-op exports. --- ## /docs/benchflow/traj-upload Anyone with BenchFlow installed can contribute a completed trajectory with one command: ```bash bench traj upload path/to/trial \ --github-id YOUR_GITHUB_ID \ --email YOU@example.com ``` `path/to/trial` may be a trial directory containing `trajectory/`, a directory of JSONL files, or one JSONL file. BenchFlow rejects duplicate object keys and non-finite numbers, structurally redacts credential-bearing keys and secret-like values in both artifacts and manifest metadata, computes a content digest, and uploads a manifest last. Use `--dry-run` to inspect the staged file list, digest, sizes, ignored siblings, and redaction count without making a network request. `--github-id` and `--email` are required for both public and direct uploads. They are self-asserted contributor provenance, not proof of account ownership, and are stored in `manifest.json` as `{"contributor":{"github_id":"...","email":"..."}}`. The email is not printed by the CLI, but dataset operators may retain or publish the manifest; use an address you are comfortable associating with the contribution. The public broker URL is built into the CLI. `BENCHFLOW_TRAJ_BROKER_URL` can override it for development or disaster recovery, and `BENCHFLOW_TRAJ_UPLOADED_BY` can add a non-secret contributor label. Do not put credentials or personal data in either label. ## What reaches the dataset Public uploads first enter a private, versioned Azure Blob quarantine prefix. The broker issues short-lived user-delegation SAS URLs scoped to create one expected blob at a time; they do not grant list, read, or delete access. An Event Grid-triggered validator independently checks the manifest contract, the 8 MiB per-record JSONL bound and structural complexity limits, allowlisted object names, byte sizes, SHA-256 hashes, strict JSONL syntax, and final artifact and manifest secret scans. Only then does it copy artifacts into the content-addressed `sources/community//` namespace, with `manifest.json` as the commit marker. Failed captures are removed from the live quarantine namespace and are never promoted. Blob versioning and lifecycle policy provide recovery and bound retention for attempted overwrites; the deployment does not configure an immutable-storage policy. The digest excludes contributor labels, timestamps, and transport details, so the same redacted bytes are idempotent across machines. Repeating an ingested upload prints `Already uploaded` and performs no blob writes. Redaction is a safety net, not a license to upload secrets. Review sensitive trajectories before contributing them; once a capture is promoted, dataset operators may retain it for benchmark provenance. ## Trusted direct upload Operators with Azure RBAC can bypass the public broker while keeping the same staging and manifest contract: ```bash uv tool install 'benchflow[azure]' az login bench traj upload path/to/trial --direct \ --github-id YOUR_GITHUB_ID \ --email YOU@example.com \ --container-url https://ACCOUNT.blob.core.windows.net/bronze ``` Direct mode uses `DefaultAzureCredential` and create-only blob calls. The identity needs a custom role with blob create/write data actions on the target container. The production deployment creates this as `TasksMiner Blob Data Creator`; Azure's broader `Storage Blob Data Contributor` role also works but grants more than direct upload needs. For routine community contributions, use the default broker mode. Deployment configuration and verification live in [`infra/trajectory-upload/`](https://github.com/benchflow-ai/benchflow/blob/main/infra/trajectory-upload/README.md). --- ## /docs/benchflow/use-cases BenchFlow's Scene-based lifecycle enables evaluation patterns that go far beyond single-turn "prompt and score." This document covers the key use cases for multi-turn, multi-agent, and stateful environment evaluation. The patterns below are all variants of one primitive: **Scenes with Roles and Turns**, all running in a single shared sandbox via ACP. No sidecar containers, no Docker Compose networking — every role lives in the same workspace and talks through ACP. > **Sandbox paths used in the prompts below.** The runtime stages the task > instruction at `/instruction.md` (sandbox root), and the oracle at `/oracle` > for native `task.md` tasks (legacy split-layout tasks use `/solution` as an > alias). The agent workspace is `/app`. For a turn with no explicit prompt > (`Turn("role")` / a bare `- role:` entry), the runtime passes the task goal > **inline** — for native `task.md` tasks it reads the prompt body from > `task.md` and sends it directly, so the agent doesn't have to read > `/instruction.md` to know the task. `/instruction.md` is still staged for > every task, so a role with an explicit prompt can read or quote it. Use > `/oracle` first and fall back to `/solution` if you support both layouts, > e.g. `cat /oracle/solve.sh 2>/dev/null || cat /solution/solve.sh`. --- ## 1. Interactive User Simulation A "user" role provides instructions iteratively; the agent responds. The user has oracle access to the solution and reveals information gradually, simulating realistic human-agent interaction. In BenchFlow, this is a two-role Scene where the "user" role is just another agent with a different prompt and (optionally) a different model. Both roles share one sandbox and one ACP session — no sidecar container, no Docker Compose networking. ### YAML ```yaml source: repo: benchflow-ai/skillsbench path: tasks environment: docker concurrency: 2 scenes: - name: interactive-assist roles: - name: user agent: gemini model: gemini-3.1-flash-lite-preview - name: assistant agent: claude-agent-acp model: claude-sonnet-4-6 turns: - role: user prompt: | You are simulating a user who needs help with the task in /instruction.md. You have access to the oracle solution at /oracle/solve.sh (legacy tasks: /solution/solve.sh). Give the assistant a high-level description of what you want. Do NOT reveal implementation details yet. Write your guidance to /app/user-guidance.md. - role: assistant - role: user prompt: | Read the assistant's work in /app/. Compare against /oracle/solve.sh (legacy: /solution/solve.sh). If incomplete, provide a targeted hint (one specific detail from the solution). Update /app/user-guidance.md with the targeted hint. - role: assistant prompt: "The user provided additional guidance. Read it and continue working." - role: user prompt: | Final check. Read /app/ and compare to /oracle/ (legacy: /solution/). If correct, write LGTM to /app/user-guidance.md. If not, give one final hint. - role: assistant prompt: "Address the user's latest feedback and finalize your solution." ``` ### Python ```python from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="interactive-assist", roles=[ Role("user", "gemini", "gemini-3.1-flash-lite-preview"), Role("assistant", "claude-agent-acp", "claude-sonnet-4-6"), ], turns=[ Turn("user", "You are simulating a user. Read /instruction.md..."), Turn("assistant"), # None = native goal passed inline (legacy: instruction.md) Turn("user", "Check the assistant's work against /oracle/ (legacy: /solution/)..."), Turn("assistant", "The user provided additional guidance..."), ]), ], environment="docker", ) result = await bf.run(config) ``` ### Why this design - One sandbox, one ACP session — no sidecar container, no Docker Compose networking, no extra server to maintain. - Roles share the sandbox filesystem; any handoff is explicit task state, such as a file named in the next prompt. BenchFlow does not inject messages between turns. - The user agent is a real LLM with full tool access — it can read files, check outputs, and give nuanced feedback, not just templated responses. - Same task folder works for single-turn (baseline) and interactive (with user) via different YAML configs. ### Lighter-weight alternative: `BaseUser` callback When you don't need a second LLM and your "user" logic is rule-based or oracle-guided (e.g. compress instruction → show test failures as hints → stop on pass), use a `BaseUser` Python callback instead of a multi-role Scene. See [progressive-disclosure.md](/docs/benchflow/progressive-disclosure). Built for the SWE-bench Pro progressive-disclosure use case. --- ## 2. Code Review Loop (followup-bench) A coder agent solves the task, then an independent reviewer agent critiques the solution. The coder revises based on the feedback. The reviewer never has write access to `/app/` -- it can only read and provide feedback. ### YAML ```yaml source: repo: benchflow-ai/skillsbench path: tasks environment: docker concurrency: 2 scenes: - name: review-loop roles: - name: coder agent: gemini model: gemini-3.1-flash-lite-preview - name: reviewer agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: coder - role: reviewer prompt: | You are an expert code reviewer. Read the task at /instruction.md and the coder's work in /app/. Write specific, actionable feedback. IMPORTANT: Do NOT modify any files in /app/ except /app/review-feedback.md. Write your specific feedback to /app/review-feedback.md. - role: coder prompt: "Read /app/review-feedback.md and revise your solution." ``` ### Python (with MCP reviewer sidecar) For stronger isolation, use the MCP reviewer server pattern. The reviewer runs as a sidecar service -- it has no filesystem write access at all. The coder calls the reviewer via a tool call: ```python from pathlib import Path from benchflow.experimental.mcp.hooks import mcp_reviewer_hook import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="solve-and-review", roles=[Role("coder", "gemini", "gemini-3.1-flash-lite-preview")], turns=[ Turn("coder"), Turn("coder", "Call the review_code MCP tool to get feedback, then fix issues."), ]), ], environment="docker", pre_agent_hooks=[mcp_reviewer_hook(port=8100, model="gemini-3.1-flash-lite")], ) result = await bf.run(config) ``` The MCP reviewer server (`benchflow.experimental.mcp.reviewer_server`) runs as a background process in the sandbox. It exposes `review_code` and `get_review_status` tools via streamable-http. The reviewer LLM reads the code but has **no ability to write files** -- all it can do is return feedback text. ### Results Compare reviewer variants on your task set across three conditions: | Condition | Description | |-----------|-------------| | `baseline` | Single-agent, single-turn | | `reviewer` | Coder + plain reviewer + coder revision | | `reviewer+spec` | Coder + reviewer that re-reads instruction + coder revision | Treat reviewer lift as an empirical question for the target benchmark. It is most relevant for tasks that require debugging or multi-file coordination, but it should be measured rather than assumed. ### Why this design - No Docker Compose, no sidecar container, no FastMCP server to maintain. - The MCP hook pattern gives the reviewer tool-level isolation: it cannot write to the workspace, preventing reward hacking via reviewer collusion. - Same task, same verifier -- define roles and turns in `RolloutConfig` or rollout YAML. --- ## 3. Skill Generation (BYOS -- Bring Your Own Skill) An agent generates a task-specific skill before solving. This is a two-scene rollout: `prep` (unscored) and `solve` (scored). Both scenes share the sandbox, so the generated skill persists. ### YAML ```yaml source: repo: benchflow-ai/skillsbench path: tasks environment: docker concurrency: 2 scenes: - name: skill-gen roles: - name: gen agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: gen prompt: | Read /instruction.md. Analyze the task requirements. Write a skill document to /app/generated-skill.md that will help an agent solve this task. Include: key steps, common pitfalls, relevant commands or APIs, and a solution outline. - name: solve roles: - name: solver agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: solver ``` ### Python ```python from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="skill-gen", roles=[Role("gen", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("gen", "Analyze the task and write a skill to /app/generated-skill.md")]), Scene(name="solve", roles=[Role("solver", "gemini", "gemini-3.1-flash-lite-preview")], turns=[Turn("solver")]), # None prompt = native goal inline (legacy: instruction.md) ], environment="docker", ) result = await bf.run(config) ``` ### How scenes work here 1. **Scene 1 (`skill-gen`)**: The `gen` agent reads the task instruction, analyzes it, and writes a skill file. This scene is unscored -- its output is an artifact that persists in the sandbox filesystem. 2. **Scene 2 (`solve`)**: A fresh agent session starts (no context from scene 1). The `solver` agent gets the standard task goal as its prompt (passed inline for native `task.md` tasks; legacy tasks read it from `/instruction.md`) and also sees `/app/generated-skill.md` on disk. The verifier scores only the final `/app/` state. The key insight: `disconnect()` between scenes kills the agent process, so there is no context bleed. The only communication is through the shared filesystem. ### Research findings From the SkillsBench paper: self-generated skills with generic prompts yield approximately 0 percentage points of lift over baseline. The BYOS pattern only helps when the skill-generation prompt is task-type-specific (e.g., "write a skill for compiler tasks" vs. "write a skill for this task"). This result informed the GEPA (Guided Evolution of Prompts and Agents) skill improvement pipeline. --- ## 4. Multi-turn Conversation The same agent receives multiple prompts in sequence, maintaining full conversation context between turns. This is the simplest multi-turn pattern -- no role switching, just sequential prompts to a persistent ACP session. ### YAML ```yaml source: repo: benchflow-ai/skillsbench path: tasks environment: docker concurrency: 2 scenes: - name: iterative-solve roles: - name: solver agent: gemini model: gemini-3.1-flash-lite-preview turns: - role: solver - role: solver prompt: "Review your solution. Run the tests if available. Check for edge cases and fix any issues you find." - role: solver prompt: "Final check: re-read the original instruction and verify your solution addresses every requirement." ``` ### Python ```python from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="iterative-solve", roles=[Role("solver", "gemini", "gemini-3.1-flash-lite-preview")], turns=[ Turn("solver"), # native goal inline (legacy: instruction.md) Turn("solver", "Review your solution. Run tests. Fix issues."), Turn("solver", "Final check: verify every requirement is met."), ]), ], environment="docker", ) result = await bf.run(config) ``` ### How it works ACP sessions are persistent -- the agent process stays alive across all turns within a scene. The agent retains full conversation history (tool calls, outputs, reasoning) between prompts. Each `Turn` sends a new `prompt()` call on the existing session. No simulated user is required — the "user" in this pattern is the benchmark framework itself, issuing predetermined follow-up prompts. ### Why this is useful - **Self-review**: The second prompt asks the agent to check its own work, catching obvious errors. - **Iterative refinement**: Tasks that require build-test-fix cycles benefit from explicit prompts to test and iterate. - **Decomposition**: Complex tasks can be broken into phases ("first set up the environment", "now implement the feature", "now write tests"). --- ## 5. Cross-model Review Different models fill different roles in the same scene. A cheap model codes, an expensive model reviews. Role-level model configuration makes this trivial. ### YAML ```yaml source: repo: benchflow-ai/skillsbench path: tasks environment: docker concurrency: 2 scenes: - name: cross-model-review roles: - name: coder agent: gemini model: gemini-3.1-flash-lite-preview - name: reviewer agent: claude-agent-acp model: claude-sonnet-4-6 turns: - role: coder - role: reviewer prompt: | You are reviewing code written by a different agent. Read /instruction.md for the task requirements. Examine the coder's work in /app/. Write specific feedback to /app/review-feedback.md - role: coder prompt: "Read /app/review-feedback.md and revise your solution." ``` ### Python ```python from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn config = RolloutConfig( task_path=Path("tasks/my-task"), scenes=[ Scene(name="cross-model-review", roles=[ Role("coder", "gemini", "gemini-3.1-flash-lite-preview"), Role("reviewer", "claude-agent-acp", "claude-sonnet-4-6"), ], turns=[ Turn("coder"), Turn("reviewer", "Review the coder's work..."), Turn("coder", "Address the reviewer's feedback."), ]), ], environment="docker", ) result = await bf.run(config) ``` ### Cost-performance tradeoff The cross-model pattern lets you sweep the reviewer axis independently: | Variant | Coder | Reviewer | Question | |---------|-------|----------|----------| | Self-review | gemini-flash | gemini-flash | Does same-model review help? | | Cross-model | gemini-flash | claude-sonnet | Does a different model catch different bugs? | | Strong reviewer | gemini-flash | claude-opus | Does a stronger reviewer help a weaker coder? | | Weak reviewer | claude-opus | gemini-flash | Does a weaker reviewer hurt a stronger coder? | Each variant is just a different YAML file -- same task folder, same verifier, different role configurations. This enables controlled experiments on the marginal value of reviewer quality. --- ## 6. Stateful Service Tasks Tasks that require agents to interact with live services -- Gmail, Calendar, Docs, Drive, Slack. Services run as sidecar processes in the sandbox, exposing REST APIs on localhost. The agent interacts with real HTTP endpoints, not mocked tool calls. ### Python ```python from pathlib import Path import benchflow as bf from benchflow.rollout import RolloutConfig, Scene, Role, Turn from benchflow import SERVICES, build_service_hooks # Declare which services the task needs services = [SERVICES["gmail"], SERVICES["gcal"], SERVICES["slack"]] config = RolloutConfig( task_path=Path("tasks/schedule-meeting-from-email"), scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")], environment="docker", pre_agent_hooks=build_service_hooks(services), ) result = await bf.run(config) ``` Service hooks are explicit today. `RolloutConfig.services` is reserved metadata; it does not start services unless you translate it into `pre_agent_hooks`. ### Service registry BenchFlow ships with 5 built-in services (from the SmolClaws project): | Service | CLI binary | Port | Description | |---------|-----------|------|-------------| | `gmail` | `claw-gmail` | 9001 | Mock Gmail REST API (FastAPI + SQLite) | | `slack` | `claw-slack` | 9002 | Mock Slack API | | `gcal` | `claw-gcal` | 9003 | Mock Google Calendar API | | `gdoc` | `claw-gdoc` | 9004 | Mock Google Docs API | | `gdrive` | `claw-gdrive` | 9005 | Mock Google Drive API | Each service: - Runs as a background process in the same container. - Exposes a health endpoint (`/health`) for startup detection. - Uses SQLite for state -- pre-seeded from the task's `environment/` directory. - Is indistinguishable from the real API from the agent's perspective. ### Example task structure ``` tasks/schedule-meeting-from-email/ ├── task.md # YAML frontmatter + task prompt body ├── environment/ │ ├── Dockerfile # FROM benchflow/claws-base (has all claw-* binaries) │ ├── gmail.db # Pre-seeded: email from Alice with meeting request │ └── gcal.db # Pre-seeded: existing calendar entries ├── oracle/ │ └── solve.sh # Oracle: curl commands to Gmail + GCal APIs └── verifier/ └── test.sh # Verify: check gcal.db has the new event ``` --- ## /docs/skillsbench/contributing SkillsBench is the first benchmark that tests whether agent skills can improve agent performance, and how good agents are at using skills. [Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) was first introduced by Anthropic on Oct 16, 2025, and became an [open standard](https://agentskills.io/) on Dec 18, 2025. Our goal is to build the best, broadest, and highest-quality benchmark for measuring the performance of skill-enabled agents, and to make it the most widely adopted in the field. We aim to design tasks that require skill composition (3+ skills) hard enough so that SOTA performances are lower than 39%. SkillsBench evaluates: 1. How well skills improve agent efficacy vs no skills 2. How well agents can compose multiple skills together 3. Whether agents can identify correct skills among distractors This addresses a gap: nobody measures agent performance on common daily tasks (office docs, git, data processing) despite these being 99% of real use cases. # How to Get Involved ## Getting Access 1. Join the [BenchFlow Discord](https://discord.gg/G9dg3EfSva) server (#skillsbench channel) or [add Xiangyi's WeChat](https://github.com/benchflow-ai/skillsbench/blob/main/docs/wechat-qr.jpg) (please add note: SkillsBench + Background) - Introduce yourself in the channel 2. Provide your name, email, affiliation on the [SkillsBench Workspace](https://docs.google.com/spreadsheets/d/1BJpSxIt4DYedVQ26eOa9Put4TgPBv9295wB2bBkHfA8/edit?gid=1867352925#gid=1867352925) - Subscribe to meetings: [Weekly Sync](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=NmYzM2Y5NDc3NDg5NGUyYjhiZmQ4OGEwZmZlMjA0MTBfMjAyNjAxMDZUMDEwMDAwWiB4aWFuZ3lpQGJlbmNoZmxvdy5haQ&tmsrc=xiangyi%40benchflow.ai&scp=ALL), [ICML Sprint](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=NjE4YjMzNDc0MTVjNDc5NGJmNzAyZDMyNzA0MDYwZjJfMjAyNjAxMDlUMDEwMDAwWiB4aWFuZ3lpQGJlbmNoZmxvdy5haQ&tmsrc=xiangyi%40benchflow.ai&scp=ALL) 3. (Optional) [Schedule a quick call](https://cal.com/xiangyi/skillsbench) with Xiangyi Li to answer questions and brainstorm ideas ## Getting Started 1. Read through the [CONTRIBUTING.md](https://github.com/benchflow-ai/skillsbench/blob/main/CONTRIBUTING.md) on GitHub for basic context and orientation - The project adopts agent-native development. While we require instruction.md, task.toml, and task ideas to be written by humans, it's okay to use AI-assisted programming for other tasks. 2. Join meetings - Weekly sync on Monday 5PM PT / 8PM ET / 9AM GMT+8 # Contributing See the [CONTRIBUTING.md](https://github.com/benchflow-ai/skillsbench/blob/main/CONTRIBUTING.md) and [PR template](https://github.com/benchflow-ai/skillsbench/blob/main/.github/PULL_REQUEST_TEMPLATE.md) on GitHub. ## Task Requirements - BenchFlow task format with oracle solution at 100% pass rate - Test composability: tasks requiring 3-6 skills together - Limit distractor skills to <10 ## Workflow 1. Design the skill 2. Run with local claude code / codex / goose / gemini cli 3. Run agent without skills, then with skills 4. When working, add distractor skills # What Tasks We Want ## Priority Skill Categories **High priority** (daily use, unmeasured): - Office suite: pptx, google docs, excel - Version control: git, github - Collaboration: slack, notion **Subject matter expertise:** - Balance of payments, logistics, bio, finance ## Task Types to Create 1. **Single skill baseline** - e.g., "create a spreadsheet summarizing this data" 2. **Two skills composed** - e.g., "pull git history and generate report document" 3. **Three+ skills composed** - e.g., "fetch data from API, analyze in spreadsheet, create presentation" 4. **Skills with distractors** - correct skills among irrelevant ones 5. **Novel skill application** - can agent apply unfamiliar skill from reading it For each task, document: - Which skills are required vs distractor - Expected pass rate without skills vs with skills - Verification criteria # FAQ ## Contributing **Q: What kind of tasks are we looking for?** See the [`skillsbench` SKILL.md](https://github.com/benchflow-ai/skillsbench/blob/main/.claude/skills/skillsbench/SKILL.md) and the repo [CONTRIBUTING.md](https://github.com/benchflow-ai/skillsbench/blob/main/CONTRIBUTING.md) for the task classification philosophy. **Q: How do I qualify for authorship?** 3 high-quality tasks merged to main = automatic authorship **Q: What if I contribute fewer tasks but help with other work?** We absolutely consider other contributions: - Engineering work (infrastructure, tooling, CI/CD) - Running experiments - Paper writing We are very flexible. If you're interested in helping, please reach out! ## Skills Source **Q: Do we use existing skills or contribute new skills?** Both are okay! You can find useful skills at: - [skillsmp.com](https://skillsmp.com/) - [smithery.ai/skills](https://smithery.ai/skills) - [claude-scientific-skills](https://github.com/K-Dense-AI/claude-scientific-skills) For more details, visit the [Google Docs Quick Start](https://docs.google.com/document/d/17f_qDeYPaNQRVDIFIr5topEUMd4_hv1RboVGGLGgdLc/edit). # Task Format Tasks follow the [BenchFlow task format](/docs/benchflow/task-authoring): ``` task-name/ ├── instruction.md # REQUIRED - Task description ├── task.toml # REQUIRED - Metadata, timeouts, required/distractor skills ├── environment/ │ ├── Dockerfile # REQUIRED - Container with dependencies │ └── skills/ # OPTIONAL - Skills available to agent │ └── skill-name/ │ ├── SKILL.md # REQUIRED (per skill) │ ├── scripts/ # OPTIONAL │ ├── references/ # OPTIONAL │ └── assets/ # OPTIONAL ├── solution/ │ └── solve.sh # REQUIRED - Oracle solution (must pass 100%) └── tests/ ├── test.sh # REQUIRED - Runs pytest └── test_outputs.py # REQUIRED - Writes reward to /logs/verifier/reward.txt ``` ## instruction.md style Direct, terminal-bench style. No "Objective:" or "Available Skills:" sections: ``` Build a sales report from the spreadsheet data. 1. Load sales data from /app/data/sales.csv 2. Calculate total revenue by region 3. Generate /app/output/report.xlsx with summary sheet 4. Create /app/output/chart.png showing revenue breakdown ``` Style traits: - Conversational - "I am trying to...", "Help!", "Could you help me..." - Context-rich - Often starts with WHY or a scenario - Numbered lists for sequential steps - Explicit about output format and file paths - No unnecessary sections # Resources ## Skills Documentation - [Anthropic Skills Docs](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview) - [Anthropic Skills Repo](https://github.com/anthropics/skills) - [OpenAI Skills Repo](https://github.com/openai/skills/tree/main/skills) ## BenchFlow Runtime SkillsBench tasks run on the [BenchFlow runtime](/docs/benchflow): - [BenchFlow Repo](https://github.com/benchflow-ai/benchflow) - [BenchFlow docs](/docs/benchflow) Key commands: ```bash benchflow run skillsbench --agent # run all tasks benchflow run skillsbench/ # run a single task ``` Supported agents: see [Getting started](/docs/benchflow/getting-started). The runtime ships with verified ACP support for Claude Code, Codex, Gemini CLI, OpenCode, OpenHands, OpenClaw, and Pi. # Coworking Xiangyi works out of [Founders, Inc.](https://f.inc/) at [2 Marina Blvd, San Francisco](https://share.google/7oQr4XWnOuCl5rigs). Feel free to drop by if you are in the Bay. We can also host coworking sessions on a given work day. --- ## /docs/skillsbench/getting-started SkillsBench contains **94+ tasks** across 11 professional domains. Tasks run on the [BenchFlow runtime](/docs/benchflow): each task is a sandboxed environment with an oracle solution and an outcome-based verifier. # Prerequisites - **Docker** installed and running (8 GB+ memory recommended for Docker Desktop) - **BenchFlow** CLI installed ([install guide](/docs/benchflow/getting-started)) - **Python 3.12+** with [uv](https://docs.astral.sh/uv/) ```bash # Install the BenchFlow CLI (provides `benchflow` and the shorter `bench` alias) uv tool install benchflow # Clone the SkillsBench dataset git clone https://github.com/benchflow-ai/skillsbench.git cd skillsbench ``` # Running the Benchmark ## A single task `benchflow run` takes one task directory. The default agent is `claude-agent-acp` (which needs `ANTHROPIC_API_KEY` or a Claude Code login); pass `--agent oracle` for the reference-solution sanity-check. ```bash # Oracle (reference solution — no LLM needed) benchflow run tasks/ --agent oracle # With your agent benchflow run tasks/ --agent --model "" # Example: Claude Code with Sonnet 4.5 benchflow run tasks/excel-fp-sum --agent claude-agent-acp --model "anthropic/claude-sonnet-4-5" ``` ## The full benchmark `benchflow eval create` runs every task under `--tasks-dir` in parallel: ```bash # Oracle sweep — verify your setup benchflow eval create --tasks-dir tasks --agent oracle # With your agent benchflow eval create --tasks-dir tasks --agent --model "" ``` ## Running from a remote repo (no clone needed) Use `--source-repo` to pull tasks directly from GitHub: ```bash # Run all tasks with Gemini bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ -a gemini \ -m gemini-3.1-pro-preview \ -e docker # Run with Claude Code on Daytona with concurrency bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks \ -a claude-agent-acp \ -m anthropic/claude-sonnet-4-6 \ -e daytona \ -c 32 ``` ## Running with skills Mount a skills directory and enable skill nudging: ```bash bench eval create \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/edit-pdf \ -a gemini \ -m gemini-3.1-pro-preview \ -e daytona \ --skills-dir tasks/edit-pdf/environment/skills \ --ae BENCHFLOW_SKILL_NUDGE=name ``` ## Validating a task ```bash benchflow tasks check tasks/ ``` # Self-Contained Subset (no API keys) 9 of the 94+ tasks require external API keys (OpenAI, GitHub, HuggingFace, Modal, etc.) or have broken Docker builds on non-author machines. To skip them, drive `eval create` from a YAML config that lists `exclude_task_names`: ```yaml title="self-contained.yaml" jobs_dir: jobs n_attempts: 1 timeout_multiplier: 3.0 orchestrator: type: local n_concurrent_trials: 4 quiet: false environment: type: docker force_build: true delete: true agents: - name: oracle model_name: oracle datasets: - path: tasks exclude_task_names: - gh-repo-analytics # requires GH_AUTH_TOKEN - mhc-layer-impl # requires MODAL_TOKEN_ID/SECRET - pedestrian-traffic-counting # requires OPENAI/GEMINI/ANTHROPIC API keys - pg-essay-to-audiobook # requires OPENAI_API_KEY + ELEVENLABS_API_KEY - scheduling-email-assistant # hardcoded volume mount + HUGGINGFACE_API_TOKEN - speaker-diarization-subtitles # Docker build OOM (Whisper large-v3) - trend-anomaly-causal-inference # requires ANTHROPIC + OPENAI API keys - video-filler-word-remover # requires OPENAI_API_KEY - video-tutorial-indexer # requires OPENAI_API_KEY ``` ```bash benchflow eval create --config self-contained.yaml ``` # External API Keys Some tasks call external APIs during the oracle solution or verification step. To run these tasks, export the required keys before starting the job: | API Key | Tasks | What It's Used For | |---------|-------|-------------------| | `OPENAI_API_KEY` | pg-essay-to-audiobook, video-filler-word-remover, video-tutorial-indexer, trend-anomaly-causal-inference, pedestrian-traffic-counting | OpenAI Whisper (transcription), TTS (text-to-speech), and Vision APIs | | `ANTHROPIC_API_KEY` | trend-anomaly-causal-inference, pedestrian-traffic-counting | Claude API for causal inference analysis and vision-based counting | | `GEMINI_API_KEY` | pedestrian-traffic-counting | Gemini Vision API for video understanding | | `ELEVENLABS_API_KEY` | pg-essay-to-audiobook | ElevenLabs TTS (alternative to OpenAI TTS) | | `GH_AUTH_TOKEN` | gh-repo-analytics | GitHub personal access token with repo read access | | `HUGGINGFACE_API_TOKEN` | scheduling-email-assistant | HuggingFace model access | | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | mhc-layer-impl | Modal serverless GPU compute for model training | One additional task makes external API calls that don't require keys: - **find-topk-similiar-chemicals** — PubChem API (may fail under rate limiting) ```bash export OPENAI_API_KEY=sk-... export ANTHROPIC_API_KEY=sk-ant-... export GEMINI_API_KEY=... export ELEVENLABS_API_KEY=... export GH_AUTH_TOKEN=ghp_... export HUGGINGFACE_API_TOKEN=hf_... export MODAL_TOKEN_ID=ak-... export MODAL_TOKEN_SECRET=as-... ``` Note: API keys must also be listed in each task's `task.toml` under `[solution.env]` or `[environment.env]` to be passed into the Docker container. Some tasks (e.g., `pedestrian-traffic-counting`) only pass keys via `docker-compose.yaml` environment variables. The tasks that need keys already have this configured. # Known Issues The following tasks have known issues that may cause oracle or agent failures depending on your environment. These are documented from our oracle validation runs. ## Tasks with Docker build failures | Task | Issue | Workaround | |------|-------|-----------| | **speaker-diarization-subtitles** | `pip install speechbrain==1.0.3` fails; loading Whisper large-v3 model during build triggers OOM | Increase Docker Desktop memory to 16 GB+, or exclude this task | | **multilingual-video-dubbing** | Kokoro TTS model download (`KPipeline`) fails intermittently during Docker build | Retry the build; passes on ~50% of attempts | | **scheduling-email-assistant** | Docker compose mounts a hardcoded host path (`/Users/suzilewie/Downloads/auth`) that doesn't exist on other machines | Exclude this task or fix the volume mount in `docker-compose.yaml` | ## Tasks with intermittent oracle failures These tasks have oracles that sometimes fail due to environment-sensitive tests: | Task | Symptom | Root Cause | |------|---------|-----------| | **dynamic-object-aware-egomotion** | `TypeError: Object of type int64 is not JSON serializable` | Oracle outputs numpy int64 values instead of native Python ints | | **fix-build-google-auto** | `test_build_success` assertion fails — Maven build exits with code 1 | Build depends on network-fetched dependencies; flaky under Docker networking | | **reserves-at-risk-calc** | Volatility calculation tests fail | Oracle produces slightly different Excel formula results | | **setup-fuzzing-py** | Gets 5/6 tests (reward=0.83); `test_fuzz` times out after ~3 min | Fuzzing duration exceeds verifier timeout; use `timeout_multiplier: 3.0` | | **simpo-code-reproduction** | Build timeout on first attempt | Rust/tokenizers compilation is slow; passes with `timeout_multiplier: 3.0` | | **r2r-mpc-control** | `test_performance` assertion fails intermittently | MPC controller settling time is sensitive to Docker CPU scheduling | | **pedestrian-traffic-counting** | Oracle gets reward ~0.07 (counts 0 instead of 12-14) | Oracle depends on vision API keys; without them, returns zero counts | # Common Issues ## Docker build failures Some tasks compile ML dependencies from source (e.g., `simpo-code-reproduction`, `multilingual-video-dubbing`), which can take 10+ minutes. Ensure sufficient disk space and Docker memory. ```bash # Free up Docker space if builds fail docker system prune ``` ## Timeout errors The default agent timeout is 900s. For tasks with long builds or heavy computation, increase the timeout multiplier in your YAML config: ```yaml timeout_multiplier: 3.0 # multiplies both agent and build timeouts ``` ## ARM64 / Apple Silicon Running on Apple Silicon (M1/M2/M3/M4) via Docker Desktop may cause: - **Borderline test failures** — numerical thresholds (control settling times, floating-point results) differ slightly under ARM64 emulation - **Performance test flakiness** — parallel speedup benchmarks depend on Docker CPU allocation; reduce `n_concurrent_trials` to avoid CPU contention - **Longer build times** — some packages (tokenizers, safetensors) compile from source on aarch64 The following tasks have architecture-specific Dockerfile logic: | Task | Arch handling | |------|--------------| | **glm-lake-mendota** | Forces `--platform=linux/amd64` (runs under Rosetta emulation on ARM) | | **fix-druid-loophole-cve** | Detects amd64/arm64 for Java paths | | **simpo-code-reproduction** | Installs Rust for aarch64 tokenizers compilation | | **python-scala-translation** | Downloads arch-specific Coursier (Scala build tool) binary | | **suricata-custom-exfil** | Detects x86_64/aarch64 for Node.js binary | | **react-performance-debugging** | Detects amd64 for Node.js binary | If you see nondeterministic failures, try rerunning the failed task individually with `benchflow run tasks/`. ## API rate limiting Tasks calling external APIs (PubChem, CrossRef) may return 503 errors under high concurrency. Reduce `n_concurrent_trials` in your config: ```yaml orchestrator: type: local n_concurrent_trials: 2 # reduce from 4 to avoid rate limits ``` ---