# Core Concepts



Harbor has the following core concepts:

## Task

A task is a single instruction, container environment, and test script. Tasks are used to evaluate agents and models. A task is implemented as a directory of files in the [Harbor task format](/docs/tasks).

## Dataset

A [dataset](/docs/datasets) is a collection of tasks. Datasets are used to evaluate agents and models. Usually, a dataset corresponds to a benchmark (e.g. Terminal-Bench, SWE-Bench Verified, etc.). Datasets can optionally be distributed via the Harbor registry.

## Agent

An [agent](/docs/agents) is a program that completes tasks. Agents are defined by implementing the `BaseAgent` or `BaseInstalledAgent` interfaces. Tasks can configure which OS user the agent runs as via the `agent.user` field in `task.toml`.

## Container environment

Environments in Harbor are containers, typically defined as Docker images using a `Dockerfile`. The `BaseEnvironment` interface provides a unified interface for interacting with environments. Many cloud container runtimes are already supported out of the box, including [Daytona](https://www.daytona.io/), [Modal](https://modal.com/), [E2B](https://e2b.dev/), [Runloop](https://runloop.ai/), [Tensorlake](https://docs.tensorlake.ai/sandboxes/harbor), [LangSmith](https://docs.langchain.com/langsmith/harbor-integrations), [Blaxel](https://blaxel.ai/), [Novita Sandbox](https://novita.ai/), EC2, [Beam](https://beam.cloud/), and [Hyperbrowser](https://www.hyperbrowser.ai/). Other container runtimes can be supported by implementing the `BaseEnvironment` interface.

The target container OS is declared per task via `[environment].os` in `task.toml` (`"linux"` by default; set to `"windows"` for Windows containers — see [Windows tasks](/docs/tasks/windows-container-support)).

## Trial

A trial is an agent's attempt at completing a task. Trials can be configured using the `TrialConfig` class.

Essentially, a trial is a rollout that produces a reward.

## Job

A job is a collection of trials. Jobs are used to evaluate agents and models. A job can consist of multiple datasets, agents, tasks, and models. Jobs can be configured using the `JobConfig` class.

Once you define your `job.yaml` or `job.json` file, you can run it using the following command:

```bash
harbor run -c "<path/to/job.yaml>"
```

Alternatively, you can create an adhoc job by configuring the `harbor run` flags.

Under the hood, a job generates a bunch of `TrialConfig` objects and runs them in parallel.


# Getting Started



Harbor is a framework for evals, post-training, and prompt optimization using agentic environments.

## Installation

<CodeBlockTabs defaultValue="uv">
  <CodeBlockTabsList>
    <CodeBlockTabsTrigger value="uv">
      uv
    </CodeBlockTabsTrigger>

    <CodeBlockTabsTrigger value="pip">
      pip
    </CodeBlockTabsTrigger>

    <CodeBlockTabsTrigger value="uv upgrade">
      uv upgrade
    </CodeBlockTabsTrigger>
  </CodeBlockTabsList>

  <CodeBlockTab value="uv">
    ```bash
    uv tool install harbor
    ```
  </CodeBlockTab>

  <CodeBlockTab value="pip">
    ```bash
    pip install harbor
    ```
  </CodeBlockTab>

  <CodeBlockTab value="uv upgrade">
    ```bash
    uv tool uninstall harbor
    uv tool install harbor
    ```
  </CodeBlockTab>
</CodeBlockTabs>

## Nightly builds (not frequently used for most users)

We publish a dev release from the latest `main` to PyPI daily. Install it with a pre-release flag (stable installs are unaffected):

<CodeBlockTabs defaultValue="uv">
  <CodeBlockTabsList>
    <CodeBlockTabsTrigger value="uv">
      uv
    </CodeBlockTabsTrigger>

    <CodeBlockTabsTrigger value="pip">
      pip
    </CodeBlockTabsTrigger>

    <CodeBlockTabsTrigger value="uv upgrade">
      uv upgrade
    </CodeBlockTabsTrigger>
  </CodeBlockTabsList>

  <CodeBlockTab value="uv">
    ```bash
    uv tool install --prerelease explicit 'harbor>=0.dev0'
    ```
  </CodeBlockTab>

  <CodeBlockTab value="pip">
    ```bash
    pip install --pre harbor
    ```
  </CodeBlockTab>

  <CodeBlockTab value="uv upgrade">
    ```bash
    uv tool uninstall harbor
    uv tool install --prerelease explicit 'harbor>=0.dev0'
    ```
  </CodeBlockTab>
</CodeBlockTabs>

## Getting started

Run the following command to see a list of all available commands:

```bash
harbor --help
```

## Running an eval

The primary command is `harbor run`, which is used to run evals or generate rollouts.

```bash
harbor run --help
```

To view registered datasets, run

```bash
harbor dataset list
```

### Running a registered dataset

To evaluate an agent and model one of these datasets, you can use the following command:

```bash
harbor run -d "<org/name>" -m "<model>" -a "<agent>"
```

Harbor will automatically download registered datasets.

### Running a local dataset

Local datasets (directories of tasks) can also be run using

```bash
harbor run -p "<path/to/dataset>" -m "<model>" -a "<agent>"
```

### Running a cloud sandbox

To run using a cloud sandbox provider like Daytona, you can use the following command:

```bash
harbor run -d "<org/name>" -m "<model>" -a "<agent>" --env "daytona" -n 32
```

If you run a cloud sandbox using an API model, trials become I/O bounded rather than compute bounded, which means you can typically parallelize far above your CPU count (the example command above runs 32 trials concurrently).

Sandboxed agent evaluations are often slow, because they can require many turns to complete and each command requires time to execute. Horizontal scaling becomes the only viable way to accelerate experimentation, so we recommend using a cloud sandbox provider like Daytona.


# Motivation



Harbor is a framework for evaluating and optimizing agents and models in container environments.

When we released Terminal-Bench in May, we were surprised to see it used in unexpected ways like building custom evals, optimizing prompts, running RL, generating SFT traces, and CI/CD agent testing.

We also learned that defining and managing containerized tasks at scale is hard. We built Harbor to make it easy.

Harbor provides:

* Simple, modular interfaces for environments, agents, and tasks
* All popular CLI agents pre-integrated
* A registry of popular benchmarks and datasets
* Integrations with cloud sandbox providers like [Daytona](https://www.daytona.io/), [Modal](https://modal.com/), [E2B](https://e2b.dev/), [Runloop](https://runloop.ai/), [Tensorlake](https://docs.tensorlake.ai/sandboxes/harbor), [LangSmith](https://docs.langchain.com/langsmith/harbor-integrations), [Blaxel](https://blaxel.ai/), [Novita Sandbox](https://novita.ai/), EC2, [Beam](https://beam.cloud/), [Hyperbrowser](https://www.hyperbrowser.ai/), and [Vercel Sandbox](https://vercel.com/docs/sandbox) for horizontal scaling
* Integrations with frameworks like SkyRL and GEPA for optimizing agents


# Migrating from Terminal-Bench



Harbor is created by the same team as Terminal-Bench. The Harbor task format is an iteration on the format proposed by Terminal-Bench and addresses some of the limitations of the Terminal-Bench format.

The difference in file trees is shown below:

A detailed description of the differences between the two task formats is available [here](/docs/tasks/task-difference).

import { File, Folder, Files } from 'fumadocs-ui/components/files';

### Terminal-Bench Task Format

<Files>
  <File name="docker-compose.yaml" />

  <File name="task.yaml" />

  <File name="solution.sh" />

  <File name="run-tests.sh" />

  <Folder name="tests" defaultOpen>
    <File name="test_outputs.py" />

    <File name="..." />
  </Folder>

  <File name="..." />
</Files>

### Harbor Task Format

<Files>
  <File name="instruction.md" />

  <File name="task.toml" />

  <Folder name="environment" defaultOpen>
    <File name="Dockerfile" />

    <File name="..." />
  </Folder>

  <Folder name="solution" defaultOpen>
    <File name="solve.sh" />

    <File name="..." />
  </Folder>

  <Folder name="tests" defaultOpen>
    <File name="test.sh" />

    <File name="..." />
  </Folder>
</Files>

## Migration Guide

Harbor provides a mapper from the Terminal-Bench task format to the Harbor task format. Note that this mapper is not perfect, and some particularly custom tasks may require manual migration.

```bash
harbor task migrate -i "<path/to/terminal-bench/task(s)>" -o "<path/to/harbor/task(s)>"
```

To understand the differences between the two task formats, see the [differences](/docs/tasks/task-difference) page.

> **Linux by default.** Tasks migrated from Terminal-Bench remain Linux-targeted unless you explicitly set `[environment].os = "windows"` in `task.toml`. The new `os` field defaults to `"linux"`, so the migration mapper requires no changes for existing tasks. See [Windows tasks](/docs/tasks/windows-container-support) for Windows-targeted tasks.


# Usage Stats



Harbor collects minimal anonymous usage statistics to understand how the CLI is used, which execution paths are reliable, and where development work should be prioritized.

Usage stats are enabled by default. You can opt out at any time:

```bash
HARBOR_TELEMETRY=off harbor run ...
```

## Identity

Harbor generates a random installation ID on first use and stores it in the user config directory as `telemetry_id`. This ID is used as the analytics `distinct_id` so Harbor can group events from the same local installation across sessions.

The installation ID is not derived from your IP address, username, email, GitHub account, job name, task name, or file paths.

## Delivery

Events are handed to a short-lived detached helper process that sends them after the command returns, so telemetry never delays a command. The helper makes a single delivery attempt and exits; events that cannot be delivered, for example while offline, are dropped.

## Events

Harbor sends two lifecycle events:

* `harbor.command_finished`
* `harbor.job_finished`

### `harbor.command_finished`

This event describes a completed Harbor CLI invocation. It is emitted from the CLI boundary after a command returns, errors, or is interrupted.

Collected fields include:

* Harbor version, install source, Python version, operating system, architecture, and whether the process appears to run in CI
* launch source: `cli`, `viewer`, or `unknown`
* the AI coding agent driving the invocation, when a known agent environment marker is present: a fixed label such as `claude-code`, or `unknown-ai-agent` for the generic marker convention; environment variable values are never collected
* sanitized command and command path, such as `run`, `view`, or `job start`
* command flag names, such as `--task`, `--config`, or `-t`; only flag names registered on the invoked command are recorded, and flag values and positional arguments are never collected
* status: `completed`, `errored`, or `interrupted`
* duration in seconds
* exit code
* exception type only, when available
* job IDs of the Harbor jobs the command runs, so multi-job commands such as sweeps stay linked to their invocation

### `harbor.job_finished`

This event describes the resolved job configuration, final job result, and aggregate runtime measurements. It is emitted when the job lifecycle ends, including normal completion, handled errors, and keyboard interruption when Harbor can run cleanup.

Collected fields include:

* Harbor version, install source, Python version, operating system, architecture, and whether the process appears to run in CI
* job ID, which is the runtime UUID already used by Harbor's `JobResult.id`
* launch source: `cli`, `viewer`, `programmatic`, or `unknown`
* the AI coding agent driving the invocation, when a known agent environment marker is present (same fixed labels as `harbor.command_finished`)
* whether the job is a resumed job
* task count, total planned trials, attempts, concurrency, retry settings, install-only mode, and verification-disabled mode
* environment type and resource override counts
* whether GPU or TPU resources are requested
* agent names, agent count, model providers, and model names
* whether a simulated user agent is configured, its agent name, model provider, and model name, the bridge kind, and whether a custom persona, prompt template, bridge prompt, or bridge kwargs are set; paths, contents, and kwargs themselves are never collected
* whether custom agents, custom environments, custom verifiers, custom metrics, MCP, skills, artifacts, or extra instructions are used
* dataset source types such as local, package, registry, repo, or git
* Harbor-owned registry/package dataset refs and package task refs
* whether custom registry URLs or registry paths are used
* aggregate task-shape fields such as multi-step task count, separate-verifier task count, and network modes
* status
* duration in seconds
* completed, errored, cancelled, pending, and running trial counts
* retry count
* exception types only
* aggregate input tokens, cache tokens, output tokens, and cost in USD when agents report them
* aggregate trial duration statistics: min, mean, p50, p95, and max
* average standard reward for the conventional `reward` key

## What Harbor Does Not Collect

Harbor does not intentionally collect:

* raw command arguments
* raw job configuration
* local dataset or task paths
* repo URLs, repo org/name values, or repo subdirectories
* custom registry URLs or registry file paths
* dataset include/exclude task-name filters
* job names or trial names
* local file paths
* repository URLs
* task instructions or prompts
* task names for local/private tasks
* environment variable names or values
* agent or environment kwargs
* MCP server names or URLs
* artifact paths
* exception messages or tracebacks
* usernames, emails, or organization names
* IP address or GeoIP-derived location

Telemetry events are sent with PostHog person profile processing disabled and GeoIP disabled.

## Launch Source

Harbor records launch source as runtime metadata, not as part of the replayable job configuration:

* normal terminal commands are marked as `cli`
* jobs started from the viewer run launcher are marked as `viewer`
* direct Python use of `Job.create(...).run()` is marked as `programmatic`

If `HARBOR_LAUNCH_SOURCE` is set to an unrecognized value, Harbor records `unknown`.


# Contributing



Please consider contributing to Harbor!

## Reporting issues

If you find a bug or have a feature request, please open an issue on the [GitHub repository](https://github.com/laude-institute/harbor/issues).

## Contributing code

If you want to contribute code to Harbor, please open a pull request on the [GitHub repository](https://github.com/laude-institute/harbor/pulls).

## Discord

Even if you don't want to contribute code, you can still [join our Discord server](https://discord.gg/QVvyhRw5UQ) to discuss ideas, ask questions, and get help.


# Roadmap



Here's what we're working on and where Harbor is headed.

## Benchmarks & Tasks

* [ ] **Adapt all compatible benchmarks into the Harbor registry** — Run any major benchmark through Harbor with a single command.
* [ ] **Build new benchmarks** — Develop new agent benchmarks like Terminal-Bench 3.0.
* [ ] **Help researchers and companies build & share benchmarks** — We want to work directly with users on this.
* [ ] **Multi-turn conversation evaluation** — Both step-by-step verification and simulated users.
* [ ] **Task creation tooling** — Capture production agent state and map it to tasks.

## Integrations

* [ ] **Integrate with more sandbox providers** — e.g. Vercel sandbox.
* [ ] **Integrate with Tinker for training** — RL on Harbor tasks.
* [ ] **Integrate with SkyRL for training** — RL on Harbor tasks.
* [x] **Support LLM-as-a-judge** — Add first-class support for using language models as evaluation judges.

## Infrastructure

* [ ] **Improve visualization and analysis tooling** — Help people debug their jobs fast.
  * [ ] **Improved sandbox snapshotting** — Snapshotting the state of an environment for debugging. (E.g. snapshotting the filesystem, asciinema recordings, etc.)
* [ ] **Build a hosted storage layer** — PyPI for Harbor tasks. Make it easy to version, share, distribute, track metrics, and run tasks. This includes sharable private registries.
* [ ] **Build hosted rollout infra** — Managed infrastructure for running evaluations at scale.


# ACP Registry Agents



Harbor can run agents from the [ACP registry](https://github.com/agentclientprotocol/registry) through the built-in generic `acp` runner.

## CLI

Use the `acp:<id>[@version]` shorthand anywhere `--agent` is accepted:

```bash
harbor run \
  --path examples/tasks/hello-world \
  --agent acp:opencode@1.3.9 \
  --model openai/gpt-5.4 \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY
```

If the version is omitted, Harbor resolves the latest `agent.json` from the registry's `main` branch during agent setup.

## Config Files

The same shorthand works in YAML or JSON configs and is preserved in the persisted `config.json`:

```yaml
agents:
  - name: acp:opencode@1.3.9
    model_name: openai/gpt-5.4
    kwargs:
      auth_policy: auto
      permission_mode: allow
```

## SDK

SDK users can use the same declarative agent name:

```python
from harbor.models.trial.config import AgentConfig

agent = AgentConfig(
    name="acp:opencode@1.3.9",
    model_name="openai/gpt-5.4",
    env={"OPENAI_API_KEY": "${OPENAI_API_KEY}"},
)
```

Registry resolution happens asynchronously in `AcpAgent.setup()`, so creating configs and agents does not perform network I/O.

## Options

Common ACP kwargs:

* `auth_policy`: `auto`, `explicit`, or `disabled`
* `permission_mode`: `allow` or `deny`
* `distribution_preference`: comma-separated preference among `binary`, `npx`, `uvx`, and `local`
* `registry_ref`: ACP registry git ref for latest-version resolution
* `registry_cache_dir`: local cache directory for fetched registry entries

You can also run the generic ACP agent with an explicit registry entry:

```yaml
agents:
  - name: acp
    kwargs:
      registry_entry_path: /path/to/agent.json
```

For agents already installed in the task environment, use Harbor's `local`
distribution.

```yaml
agents:
  - name: acp
    kwargs:
      registry_entry:
        id: local-agent
        name: Local Agent
        version: local
        description: ACP agent provided by the task image
        distribution:
          local:
            cmd: node
            args: [/opt/local-agent/server.js]
```

## Outputs

ACP runs write these files under the agent log directory:

* `acp.txt`
* `acp-events.jsonl`
* `acp-summary.json`
* `trajectory.json`

Because Harbor generates `trajectory.json`, standard trace export works:

```bash
harbor traces export -p jobs/<job-name> --recursive
```


# Agents



How to evaluate on existing agents and integrate your own. This is particularly useful for benchmarking your agent, optimizing its prompts, using it as a scaffold for RL, or using it to generate SFT datasets.

## Existing agents

Harbor comes with most popular agents pre-integrated. List them with:

```bash
harbor agent list
```

Right now, Harbor includes Terminus-2, Claude Code, Copilot CLI, Codex CLI, Gemini CLI, Grok Build, OpenHands, Antigravity SDK, Mini-SWE-Agent, fx, MCode, fx-dev, and more.

Inspect an agent's accepted options, types, and defaults with:

```bash
harbor agent schema claude-code
```

Pass options with `--ak` / `--agent-kwarg`, or as `kwargs` in a job config. Harbor rejects unknown or invalid options before starting an environment. Omitted reasoning and thinking options use the agent's own defaults.

### Native configuration

Codex, Claude Code, and DeerFlow accept a local config file or inline JSON:

```bash
harbor run ... --ak "config=$PWD/config.toml"
harbor run ... --ak 'config={"model_context_window": 200000}'
```

Inline JSON is converted to the agent's native TOML, JSON, or YAML format.

## Integrating your own agent

Harbor supports integrating your own agent without having to modify the Harbor source code.

There are two types of agents:

1. **External agents** which interface with the environment through the `BaseEnvironment` interface, typically by executing bash commands via the `exec` method.
2. **Installed agents** which are agents that are installed directly into the container environment and are executed in headless mode. This is how most agents are integrated and comes with the advantage of bringing custom tools.

### External agents

To build an external agent, you need to implement the `BaseAgent` interface which involved defining the following methods:

```python title="my_external_agent.py"
from harbor.agents.base import BaseAgent

class MyExternalAgent(BaseAgent):
    @staticmethod
    def name() -> str:
        """The name of the agent."""
        pass

    def version(self) -> str | None:
        """The version of the agent."""
        pass

    async def setup(self, environment: BaseEnvironment) -> None:
        """
        Run commands to setup the agent & its tools.
        """
        pass

    async def run(
        self,
        instruction: str,
        environment: BaseEnvironment,
        context: AgentContext,
    ) -> None:
        """
        Runs the agent in the environment. Be sure to populate the context with the
        results of the agent execution. Ideally, populate the context as the agent
        executes in case of a timeout or other error.

        Args:
            instruction: The task instruction.
            environment: The environment in which to complete the task.
            context: The context to populate with the results of the agent execution.
        """
        pass
```

### Installed agents

To build an installed agent, you need to implement the `BaseInstalledAgent` interface which involves defining the following methods:

```python title="my_installed_agent.py"
from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext

class MyInstalledAgent(BaseInstalledAgent):
    async def install(self, environment: BaseEnvironment) -> None:
        """
        Install the agent in the environment. Use exec_as_root for system
        packages and exec_as_agent for user-level installs.
        """
        await self.exec_as_root(environment, command="apt-get update && apt-get install -y curl")
        await self.exec_as_agent(environment, command="pip install my-agent")

    @with_prompt_template
    async def run(
        self, instruction: str, environment: BaseEnvironment, context: AgentContext
    ) -> None:
        """
        Run the agent in the environment. The @with_prompt_template decorator
        automatically applies prompt template rendering to the instruction.
        Use exec_as_agent to execute commands as the configured agent user.
        """
        await self.exec_as_agent(
            environment,
            command=f"my-agent run {shlex.quote(instruction)}",
        )

    def populate_context_post_run(self, context: AgentContext) -> None:
        """
        Populate the context with the results of the agent execution.
        Called after run() completes. Typically involves parsing trajectory files.
        """
        pass
```

The `exec_as_root` and `exec_as_agent` helpers handle logging, environment variable merging, `set -o pipefail`, and error handling automatically. `exec_as_agent` runs commands as the task's configured agent user (see [`agent.user` in task.toml](/docs/tasks#configuration--metadata)).

### Running a custom agent

To run a custom agent, you can use the following command:

```bash
harbor run -d "<dataset@version>" --agent path.to.agent:SomeAgent
```


# Terminus-2



## Overview

Terminus-2 is Harbor's reference agent implementation, designed as a research-preview agent for evaluating language models' capabilities in terminal environments. It operates entirely autonomously within sandboxed environments and serves as a high-performance neutral test bed for understanding language model agent capabilities.

## Key Features

### Mono-tool Design

Terminus-2 uses a unique **single-tool approach** - an interactive tmux session - allowing it to:

* Send keystrokes and navigate environments flexibly
* Scroll through output and use arrow keys to navigate menus
* Launch additional shells within the environment
* Interact with any terminal-based application naturally

This design philosophy enables the agent to work with virtually any command-line interface without requiring specialized tools for each interaction pattern.

### Independent Execution

The agent's logic runs in a **separate Python process** from the Docker container, enabling:

* Remote connection to arbitrary computer environments
* Dockerized execution environments for safety and isolation
* Flexible deployment across different infrastructure setups
* Clean separation between agent logic and task environment

### Autonomy-First Approach

Terminus-2 is designed to operate without human intervention:

* **Will never ask for user input** during task execution
* Independently attempts to complete tasks end-to-end
* Currently recommended only for sandboxed environments due to full autonomy
* Makes decisions and recovers from errors without guidance

## Using Terminus-2 with Harbor

### Basic Usage

Run Terminus-2 on a task using the `--agent terminus-2` flag:

```bash
harbor run \
  --agent terminus-2 \
  --model openai/gpt-5 \
  --path examples/tasks/ \
  --task-name hello-world
```

### Configuration Options

Terminus-2 supports various configuration options through the agent config:

```python
from harbor.models.trial.config import AgentConfig
from harbor.models.agent_name import AgentName

agent_config = AgentConfig(
    name=AgentName.TERMINUS_2,
    model_name="openai/gpt-5",
    kwargs={
        # Parser configuration
        "parser_name": "json",  # "json" or "xml" (default: "json")

        # API configuration
        "api_base": "https://your-vllm-server.com",  # Custom API endpoint
        "temperature": 0.7,  # Sampling temperature (default: 0.7)

        # Episode/turn limits
        "max_turns": 100,  # Maximum number of episodes (default: 1000000)

        # Summarization configuration
        "enable_summarize": True,  # Enable context summarization (default: True)
        "proactive_summarization_threshold": 8000,  # Free tokens threshold for summarization (default: 8000)

        # RL training configuration (default: False)
        # If enabled, token ids and logprobs are collected in result and persisted in trajectories
        "collect_rollout_details": False,

        # Advanced model configuration
        "reasoning_effort": "medium",  # "none", "minimal", "low", "medium", "high", "xhigh", "max", or "default" (default: None)
        "max_thinking_tokens": 2048,  # For Anthropic extended thinking mode (minimum: 1024, default: None)

        # Optional: Register custom model info with LiteLLM
        # LiteLLM doesn't recognize uncommon models like custom models. For metrics
        # tracking and context summarization to work properly, provide model_info following
        # https://docs.litellm.ai/docs/completion/token_usage#9-register_model
        "model_info": {
            "max_input_tokens": 128000,
            "max_output_tokens": 4096,
            "input_cost_per_token": 0.000003,
            "output_cost_per_token": 0.000015,
        },  

        # Session tracking (included in the LLM request body unless LLM provider doesn't support)
        "session_id": "custom-session-id",  # Custom session ID (default: auto-generated UUID)
    }
)
```

## Conversation History Management

Terminus-2 implements intelligent conversation history management to handle long-running tasks efficiently while staying within context window limits.

### Standard Summarization Process

Both proactive and passive summarization use a 3-step subagent process to generate high-quality summaries:

```
┌─────────────────────────────────────────────────────────────────┐
│                   Standard Summarization Flow                    │
└─────────────────────────────────────────────────────────────────┘

  Previous History
        │
        ▼
  ┌─────────────────────┐
  │ 1. Summary Subagent │
  │   Input: Previous   │
  │   Output: Summary   │
  └─────────────────────┘
        │
        ▼
  ┌─────────────────────┐
  │ 2. Question Subagent│
  │   Input: Summary    │
  │   Output: Questions │
  └─────────────────────┘
        │
        ▼
  ┌─────────────────────┐
  │ 3. Answer Subagent  │
  │   Input: Previous + │
  │   Summary + Qs      │
  │   Output: Answers   │
  └─────────────────────┘
        │
        ▼
  ┌─────────────────────┐
  │   Main Agent        │
  │   Context:          │
  │   • System prompt   │
  │   • Task            │
  │   • Summary         │
  │   • Questions       │
  │   • Answers         │
  └─────────────────────┘
```

**Step 1 - Summary Subagent**: Receives the full previous conversation history and generates an initial summary.

**Step 2 - Question Subagent**: Receives only the summary (not the full history) and generates clarifying questions about any missing critical information.

**Step 3 - Answer Subagent**: Receives the previous history, summary, and questions, then answers the questions to fill in the gaps.

The main agent then continues with a compressed context containing: system prompt, task description, summary, questions, and answers.

### Proactive Summarization

When free tokens (max input tokens - current context length) drop below the `proactive_summarization_threshold` (default: 8000), Terminus-2:

1. Pauses execution
2. Runs the standard 3-step summarization process on the conversation history
3. Replaces the middle portion of the conversation history with the summary + Q\&A
4. Keeps the system prompt and task description intact
5. Resumes execution with the compressed history

The threshold can be configured via `proactive_summarization_threshold` in agent config.

### Passive Summarization

When a `ContextLengthExceededError` occurs, Terminus-2 uses a 3-way fallback strategy to recover and continue execution:

```
┌─────────────────────────────────────────────────────────────────┐
│              Passive Summarization Fallback Flow                 │
└─────────────────────────────────────────────────────────────────┘

              ContextLengthExceededError
                           │
                           ▼
            ┌──────────────────────────────┐
            │ 1. Unwind to Free Tokens     │
            │    Remove recent messages    │
            │    from end until enough     │
            │    space (keeps first msg)   │
            └──────────────────────────────┘
                           │
                           ▼
            ┌──────────────────────────────┐
            │ 2. Standard Summarization    │
            │    (3-step subagent process) │
            └──────────────────────────────┘
                           │
                  ┌────────┴────────┐
                  │                 │
              Success            Failure
                  │                 │
                  │                 ▼
                  │    ┌──────────────────────────┐
                  │    │ 3. Fallback Summary      │
                  │    │    Only: System prompt + │
                  │    │    Task + Current state  │
                  │    └──────────────────────────┘
                  │                 │
                  │        ┌────────┴────────┐
                  │        │                 │
                  │    Success            Failure
                  │        │                 │
                  │        │                 ▼
                  │        │    ┌──────────────────────┐
                  │        │    │ 4. Ultimate Fallback │
                  │        │    │    System prompt +   │
                  │        │    │    Task + State only │
                  │        │    │    (Continue without │
                  │        │    │     summarization)   │
                  │        │    └──────────────────────┘
                  │        │                 │
                  └────────┴─────────────────┘
                           │
                           ▼
                   Continue execution with
                   compressed/recovered context
```

**Step 1 - Unwind**: Remove recent messages from the end of the conversation (in pairs of user + assistant) until there are enough free tokens for summarization. Always keeps at least the first message.

**Step 2 - Standard Summarization**: Run the 3-step subagent process. If successful, replace the unwound messages with the summary + Q\&A and continue execution.

**Step 3 - Fallback**: If standard summarization fails, attempt a simpler summary using only system prompt, task description, and current state. If successful, continue with this compressed context.

**Step 4 - Ultimate Fallback**: If fallback also fails, continue execution with only system prompt, task description, and current state (no summary).

This recovery mechanism allows Terminus-2 to continue executing even when context limits are exceeded. Enable with `enable_summarize=True` in agent config.

## Reinforcement Learning Support

Terminus-2 is designed with RL training in mind and collects detailed rollout information for use in RL pipelines.

### Rollout Details Collection

During execution, Terminus-2 can collect and export:

#### Token Information

* **Prompt Token IDs**: List of token ID sequences, one per turn. Each sequence contains the full prompt including chat history.
* **Completion Token IDs**: List of token ID sequences, one per turn. Each sequence contains the response tokens for that turn.
* **Logprobs**: List of log probability sequences corresponding to each completion.

These are stored as a list of `RolloutDetail` objects in the agent result metadata:

```python
# First RolloutDetail contains main agent conversation
rollout_detail = trial_result.agent_result.metadata["rollout_details"][0]

# Access turn-by-turn data
prompt_token_ids = rollout_detail["prompt_token_ids"]  # List[List[int]]
completion_token_ids = rollout_detail["completion_token_ids"]  # List[List[int]]
logprobs = rollout_detail["logprobs"]  # List[List[float]]
```

#### Rewards

Terminus-2 integrates with Harbor's verifier system to collect rewards:

```python
# Access rewards from trial results
reward = trial_result.verifier_result.rewards.get("reward", 0)
```

## Trajectory Format

Terminus-2 automatically generates trajectories in the **Agent Trajectory Interchange Format (ATIF)**, Harbor's standardized trajectory format. This enables:

* **SFT dataset generation**: Convert successful trajectories to supervised fine-tuning data
* **RL training**: Use complete action sequences and rewards for policy optimization
* **Debugging**: Inspect detailed step-by-step execution logs
* **Visualization**: Replay agent actions in Harbor's trajectory viewer

See the [Agent Trajectory Format](/docs/agents/trajectory-format) documentation for details on the ATIF specification.

### Trajectory Configuration

Terminus-2 supports a `TrajectoryConfig` that controls how trajectories are recorded and formatted. This is particularly important when generating SFT datasets or when context summarization occurs.

#### Configuration Options

**`raw_content` (default: `False`)**

Controls whether to save raw LLM responses or parsed structured data in the trajectory.

* **`raw_content=False` (default)**: Saves parsed, structured data with separate `message` and `tool_calls` fields. Best for trajectory analysis and debugging.
* **`raw_content=True`**: Saves the exact raw LLM response in the `message` field without parsing. Essential for SFT data export where you need the exact model outputs.

**`linear_history` (default: `False`)**

Controls how trajectories are split when context summarization occurs. The key difference is whether you can **recover the true LLM conversation history** from the trajectory files.

During agent execution, the actual conversation history sent to the LLM looks like this:

```
Turn 1:
  {"user": "You are an AI assistant tasked with solving command-line tasks...
           Task Description: Create a file called hello.txt
           Current terminal state: ..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}

Turn 2:
  {"user": "You are an AI assistant tasked with solving command-line tasks...
           Task Description: Create a file called hello.txt
           Current terminal state: ..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
  {"user": "New Terminal Output:\nroot@container:/app# ..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}

// Context summarization happens here

Turn 3:
  {"user": "You are an AI assistant tasked with solving command-line tasks...
           Task Description: Create a file called hello.txt
           Current terminal state: ..."}
  {"user": "You are picking up work from a previous AI agent on this task:
           **Original Task:** Create a file called hello.txt
           **Summary from Previous Agent:** ...
           **Current Terminal Screen:** ...
           Please begin by asking several questions..."}
  {"assistant": "1. What files have been created so far?\n2. ..."}
  {"user": "Here are the answers the other agent provided.\n\n[answers]\n\nContinue working on this task..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
```

Notice how in Turn 3, the conversation history was reset and compressed - the system prompt is followed by the question prompt (which includes the task, summary, and terminal screen), then model questions, then the handoff prompt with answers, skipping all the intermediate conversation steps.

**When `linear_history=False` (default):**

All main agent steps are stored in a single `trajectory.json` file in a human-readable format, while summarization subagents are stored in separate files. However, you **cannot recover the true conversation history** from the main trajectory file - the handoff prompt appears to be a continuation of the previous conversation, but in reality, the LLM context was reset.

File structure:

```
trajectory.json                              # All main agent steps
trajectory.summarization-1-summary.json      # First summarization: summary subagent
trajectory.summarization-1-questions.json    # First summarization: questions subagent
trajectory.summarization-1-answers.json      # First summarization: answers subagent
```

If multiple summarizations occur, you'll see:

```
trajectory.json
trajectory.summarization-1-*.json            # First summarization
trajectory.summarization-2-*.json            # Second summarization
...
```

**When `linear_history=True`:**

The trajectory is split into separate files when summarization occurs. Each file represents a continuous, unambiguous linear history that was actually sent to the LLM.

`trajectory.json`:

```json
[
  {"user": "You are an AI assistant tasked with solving command-line tasks...\nTask Description: Create a file called hello.txt\nCurrent terminal state: ..."},
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"},
  {"user": "New Terminal Output:\nroot@container:/app# ..."},
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
]
```

`trajectory.cont-1.json`:

```json
[
  {"user": "You are an AI assistant tasked with solving command-line tasks...\nTask Description: Create a file called hello.txt\nCurrent terminal state: ..."},
  {"user": "You are picking up work from a previous AI agent on this task:\n**Original Task:** ...\n**Summary from Previous Agent:** ...\n**Current Terminal Screen:** ...\nPlease begin by asking several questions..."},
  {"assistant": "1. What files have been created so far?\n2. ..."},
  {"user": "Here are the answers the other agent provided.\n\n[answers]\n\nContinue working on this task..."},  // No ambiguity!
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
]
```

File structure:

```
trajectory.json                              # Before first summarization
trajectory.cont-1.json                       # After first summarization
trajectory.cont-2.json                       # After second summarization (if any)
trajectory.summarization-1-*.json            # First summarization subagents
trajectory.summarization-2-*.json            # Second summarization subagents (if any)
```

**Use cases:**

* **`linear_history=False`**: Simpler structure, easier to see the full agent execution in one file. Good for debugging and human analysis.
* **`linear_history=True`**: Each file represents the exact LLM context. Essential for SFT training where you need unambiguous input/output sequences.

#### Example Configuration

```python
from harbor.agents.terminus_2 import Terminus2
from harbor.models.agent.trajectory_config import TrajectoryConfig

# For SFT dataset generation
trajectory_config = TrajectoryConfig(
    raw_content=True,      # Preserve exact LLM responses
    linear_history=True    # Split on summarization for clean sequences
)

agent = Terminus2(
    logs_dir=Path("logs"),
    model_name="anthropic/claude-3-5-sonnet-20241022",
    trajectory_config=trajectory_config
)
```

Common configurations:

**For debugging and analysis:**

```python
TrajectoryConfig(
    raw_content=False,     # Structured, parsed data
    linear_history=False   # Single file, easier to navigate
)
```

**For SFT data export:**

```python
TrajectoryConfig(
    raw_content=True,      # Raw model outputs
    linear_history=True    # Clean input/output sequences
)
```

**For RL training:**

```python
TrajectoryConfig(
    raw_content=False,     # Either works (token IDs are always exact), but structured helps debugging
    linear_history=False   # Full episode in one file
)
```

## Related Documentation

* [Agents Overview](/docs/agents) - General agent integration guide
* [Agent Trajectory Format](/docs/agents/trajectory-format) - ATIF specification and usage
* [RL Training](/docs/training-workflows/rl) - Using Terminus-2 for reinforcement learning
* [SFT Datasets](/docs/training-workflows/sft) - Generating supervised fine-tuning data


# Agent Trajectory Format (ATIF)



## Overview

The **Agent Trajectory Interchange Format (ATIF)** is a standardized, JSON-based specification for logging the complete interaction history of autonomous LLM agents. ATIF unifies the data requirements of conversational logs, action sequences, and replayable data structures, ensuring collected data is immediately usable across debugging, visualization, Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL) pipelines.

For the complete specification, see the [ATIF RFC](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md).

## Key Features

ATIF provides a comprehensive format that captures:

* **Complete interaction history**: User messages, agent responses, tool executions, and environment feedback
* **Multi-turn conversations**: Support for both single-turn tasks and extended conversational interactions
* **LLM metrics**: Token usage, costs, logprobs, and other operational metrics
* **Tool calls and observations**: Structured logging of agent actions and their results
* **Multi-agent systems**: Support for subagent delegation and hierarchical architectures
* **Extensibility**: Optional `extra` fields at all levels for custom metadata

## Harbor Support

Harbor provides first-class support for ATIF through:

1. **Pydantic models** for type-safe trajectory construction and validation
2. **Trajectory validator** for validating trajectory files against the ATIF schema
3. **Automatic trajectory generation** by integrated agents

### Supported Agents

The following agents in Harbor automatically generate ATIF-compliant trajectories:

* **Terminus-2** - Harbor's reference agent implementation
* **OpenHands** - Converts OpenHands event logs to ATIF format
* **Mini-SWE-Agent** - Software engineering agent with trajectory support
* **Gemini CLI** - Google's Gemini agent interface
* **Claude Code** - Anthropic's code agent
* **Codex** - OpenAI's Codex agent with trajectory support

### OpenHands Example

OpenHands is a great example of how Harbor converts agent-specific formats to ATIF. The OpenHands agent reads event files from the agent's execution and converts them to a standardized ATIF trajectory:

```python
# From harbor/agents/installed/openhands.py
def populate_context_post_run(self, context: AgentContext) -> None:
    """Convert OpenHands events to ATIF trajectory format."""
    # Get the session directory
    session_dir = self._get_session_dir()
    events_dir = session_dir / "events"

    # Convert events to trajectory
    trajectory = self._convert_events_to_trajectory(events_dir)

    # Write trajectory.json file using Pydantic's to_json_dict method
    trajectory_path = self.logs_dir / "trajectory.json"
    with open(trajectory_path, "w") as f:
        json.dump(trajectory.to_json_dict(), f, indent=2)

    # Populate context from trajectory
    if trajectory.final_metrics:
        context.cost_usd = trajectory.final_metrics.total_cost_usd
        context.n_input_tokens = trajectory.final_metrics.total_prompt_tokens
        context.n_output_tokens = trajectory.final_metrics.total_completion_tokens
```

The conversion process:

1. Reads OpenHands event files
2. Maps events to ATIF steps (system/user/agent)
3. Converts accumulated metrics to per-step deltas
4. Creates a complete `Trajectory` object using Pydantic models
5. Exports to JSON format

## Data Classes

Harbor provides Pydantic models for all ATIF schema components in `harbor.models.trajectories`:

### Core Models

**Trajectory** - Root-level trajectory object

```python
from harbor.models.trajectories import Trajectory, Agent, Step

trajectory = Trajectory(
    schema_version="ATIF-v1.8",
    session_id="session-123",
    agent=Agent(
        name="my-agent",
        version="1.0.0",
        model_name="claude-3-5-sonnet-20241022"
    ),
    steps=[
        # ... steps
    ]
)
```

**Agent** - Agent configuration

```python
from harbor.models.trajectories import Agent

agent = Agent(
    name="openhands",
    version="0.9.0",
    model_name="gpt-4",
    extra={"agent_class": "CodeActAgent"}
)
```

**Step** - Individual interaction step

```python
from harbor.models.trajectories import Step

# User step
user_step = Step(
    step_id=1,
    timestamp="2025-01-15T10:30:00Z",
    source="user",
    message="Create a file called hello.txt with 'Hello, world!' as the content."
)

# Agent step with tool calls
agent_step = Step(
    step_id=2,
    timestamp="2025-01-15T10:30:02Z",
    source="agent",
    model_name="claude-3-5-sonnet-20241022",
    message="I'll create the file for you.",
    reasoning_content="The user wants a simple text file. I'll use the file_write tool.",
    tool_calls=[
        ToolCall(
            tool_call_id="call_1",
            function_name="file_write",
            arguments={"path": "hello.txt", "content": "Hello, world!"}
        )
    ],
    observation=Observation(
        results=[
            ObservationResult(
                source_call_id="call_1",
                content="File created successfully"
            )
        ]
    ),
    metrics=Metrics(
        prompt_tokens=520,
        completion_tokens=80,
        cached_tokens=200,
        cost_usd=0.00045
    )
)
```

**ToolCall** - Tool/function invocation

```python
from harbor.models.trajectories import ToolCall

tool_call = ToolCall(
    tool_call_id="call_price_1",
    function_name="financial_search",
    arguments={"ticker": "GOOGL", "metric": "price"}
)
```

**Observation** - Environment feedback

```python
from harbor.models.trajectories import Observation, ObservationResult

observation = Observation(
    results=[
        ObservationResult(
            source_call_id="call_price_1",
            content="GOOGL is currently trading at $185.35"
        )
    ]
)
```

**Metrics** - LLM operational metrics

```python
from harbor.models.trajectories import Metrics

metrics = Metrics(
    prompt_tokens=520,
    completion_tokens=80,
    cached_tokens=200,
    cost_usd=0.00045,
    logprobs=[-0.1, -0.05, -0.02],  # Optional
    completion_token_ids=[1722, 310, 5533]  # Optional
)
```

**FinalMetrics** - Trajectory-level aggregate metrics

```python
from harbor.models.trajectories import FinalMetrics

final_metrics = FinalMetrics(
    total_prompt_tokens=1120,
    total_completion_tokens=124,
    total_cached_tokens=200,
    total_cost_usd=0.00078,
    total_steps=3
)
```

### Export to JSON

All models provide a `to_json_dict()` method for clean JSON export:

```python
from harbor.models.trajectories import Trajectory
import json

# Build trajectory using Pydantic models
trajectory = Trajectory(...)

# Export to JSON (excludes None values by default)
trajectory_dict = trajectory.to_json_dict()

# Write to file
with open("trajectory.json", "w") as f:
    json.dump(trajectory_dict, f, indent=2)

# Include None values if needed
trajectory_dict_full = trajectory.to_json_dict(exclude_none=False)
```

## Validation

Harbor provides a trajectory validator for validating ATIF trajectory files:

### Command Line

```bash
# Validate a trajectory file
python -m harbor.utils.trajectory_validator trajectory.json
```

Output:

```
✓ Trajectory is valid: trajectory.json
```

Or for invalid trajectories:

```
✗ Trajectory validation failed: trajectory.json

Found 2 error(s):
  - trajectory.steps.0.step_id: expected 1 (sequential from 1), got 0
  - trajectory.agent.name: required field is missing
```

### Programmatic Usage

```python
from harbor.utils.trajectory_validator import TrajectoryValidator

validator = TrajectoryValidator()

# Validate from file path
is_valid = validator.validate("trajectory.json")

# Validate from dict
trajectory_dict = {...}
is_valid = validator.validate(trajectory_dict)

# Validate from JSON string
trajectory_json = '{"schema_version": "ATIF-v1.8", ...}'
is_valid = validator.validate(trajectory_json)

# Check errors
if not is_valid:
    for error in validator.get_errors():
        print(f"Error: {error}")
```

The validator:

* Validates against the complete ATIF schema using Pydantic models
* Checks required fields, types, and constraints
* Validates sequential step IDs (starting from 1)
* Validates tool call references in observations
* Validates ISO 8601 timestamps
* Ensures agent-only fields are only present on agent steps
* Collects all errors before returning (not just the first error)

## Building Custom Trajectories

Here's a complete example of building an ATIF trajectory:

```python
from harbor.models.trajectories import (
    Trajectory,
    Agent,
    Step,
    ToolCall,
    Observation,
    ObservationResult,
    Metrics,
    FinalMetrics,
)
import json

# Build the trajectory
trajectory = Trajectory(
    schema_version="ATIF-v1.8",
    session_id="025B810F-B3A2-4C67-93C0-FE7A142A947A",
    agent=Agent(
        name="my-agent",
        version="1.0.0",
        model_name="claude-3-5-sonnet-20241022",
    ),
    steps=[
        # Step 1: User message
        Step(
            step_id=1,
            timestamp="2025-01-15T10:30:00Z",
            source="user",
            message="What is the current trading price of Alphabet (GOOGL)?",
        ),
        # Step 2: Agent action with tool call
        Step(
            step_id=2,
            timestamp="2025-01-15T10:30:02Z",
            source="agent",
            message="I will search for the current trading price for GOOGL.",
            reasoning_content="The request requires the current stock price. I will execute a tool call to retrieve this information.",
            tool_calls=[
                ToolCall(
                    tool_call_id="call_price_1",
                    function_name="financial_search",
                    arguments={"ticker": "GOOGL", "metric": "price"},
                )
            ],
            observation=Observation(
                results=[
                    ObservationResult(
                        source_call_id="call_price_1",
                        content="GOOGL is currently trading at $185.35",
                    )
                ]
            ),
            metrics=Metrics(
                prompt_tokens=520,
                completion_tokens=80,
                cached_tokens=200,
                cost_usd=0.00045,
            ),
        ),
        # Step 3: Agent response
        Step(
            step_id=3,
            timestamp="2025-01-15T10:30:05Z",
            source="agent",
            message="Alphabet (GOOGL) is trading at $185.35.",
            metrics=Metrics(
                prompt_tokens=600,
                completion_tokens=44,
                cost_usd=0.00033,
            ),
        ),
    ],
    final_metrics=FinalMetrics(
        total_prompt_tokens=1120,
        total_completion_tokens=124,
        total_cached_tokens=200,
        total_cost_usd=0.00078,
        total_steps=3,
    ),
)

# Export to JSON
with open("trajectory.json", "w") as f:
    json.dump(trajectory.to_json_dict(), f, indent=2)

# Validate the trajectory
from harbor.utils.trajectory_validator import validate_trajectory

is_valid = validate_trajectory(trajectory.to_json_dict())
print(f"Trajectory is valid: {is_valid}")
```

## Audio content (v1.8+)

A `ContentPart` can reference audio the same way it references an image, so a
trajectory can capture speech input — and, as models gain speech output, a
spoken agent response:

```python
from harbor.models.trajectories import AudioSource, ContentPart

parts = [
    ContentPart(type="text", text="Answer the question in the recording."),
    ContentPart(
        type="audio",
        source=AudioSource(
            media_type="audio/wav",
            path="audio/question.wav",
            duration_sec=3.2,
        ),
    ),
]
```

Audio files live beside the trajectory, conventionally in an `audio/`
subdirectory, mirroring `images/`. Supported media types:

| Media type   | Notes                                     |
| ------------ | ----------------------------------------- |
| `audio/wav`  | Widely accepted; the safest default       |
| `audio/mpeg` | MP3                                       |
| `audio/mp4`  | m4a / AAC-in-MP4                          |
| `audio/aac`  | Raw ADTS AAC                              |
| `audio/ogg`  | Ogg Vorbis/Opus                           |
| `audio/flac` | Lossless                                  |
| `audio/webm` | What a browser's `MediaRecorder` produces |
| `audio/aiff` | Accepted by Gemini audio understanding    |

Common alternate spellings are normalized on the way in, so a producer copying a
provider's `mime_type` verbatim still validates. `audio/mp3` is the one that
matters most — it is not a registered MIME type, but the Gemini API emits it, and
it is stored as `audio/mpeg`:

```python
AudioSource(media_type="audio/mp3", path="a.mp3").media_type
# 'audio/mpeg'
```

`duration_sec` is optional. It is worth populating when you have it: several
providers bill audio per second, and unlike an image's dimensions the duration
cannot be recovered without decoding the file.

A transcript belongs in a separate `text` part rather than on the source, which
keeps `AudioSource` purely about locating bytes.

Two deliberate exclusions: streaming/realtime audio (raw PCM, G.711) is not
representable, because those encodings are not self-describing and would need
sample-rate and channel metadata; and ATIF has no video content type.

## Schema Versions

ATIF follows semantic versioning. The current version is **v1.8**. Supported versions:

* **ATIF-v1.8** (current) - Added `audio` as a `ContentPart` type with a corresponding `AudioSource` (`media_type`, `path`, optional `duration_sec`)
* **ATIF-v1.7** - Added `subagent_trajectories` and `trajectory_id` on `Trajectory` for single-file subagent embedding; added `extra` on `ToolCall` and `ObservationResult`; added `llm_call_count` on `Step`; relaxed `session_id` to optional and clarified it as run-scoped
* **ATIF-v1.6** - Added multimodal content support via `ContentPart` and `ImageSource`; extended `message` and observation `content` to accept arrays of content parts
* **ATIF-v1.5** - Added optional `tool_definitions` field to `Agent` for storing tool/function definitions used in SFT pipelines
* **ATIF-v1.4** - Added optional `prompt_token_ids` field for storing prompt token IDs
* **ATIF-v1.3** - Added optional `completion_token_ids` field for RL training
* **ATIF-v1.2** - Extended observation field to support system steps
* **ATIF-v1.1** - Added optional `extra` field at root level
* **ATIF-v1.0** - Initial specification

## Related Resources

* [ATIF RFC Specification](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md)
* [Real-world trajectory examples](https://github.com/harbor-framework/harbor/tree/main/tests/golden)
* [Harbor Pydantic models](https://github.com/harbor-framework/harbor/tree/main/src/harbor/models/trajectories)
* [Trajectory validator source](https://github.com/harbor-framework/harbor/blob/main/src/harbor/utils/trajectory_validator.py)


# Adapters (Human Guide)



import { Callout } from 'fumadocs-ui/components/callout';
import { File, Folder, Files } from 'fumadocs-ui/components/files';

To add a new benchmark or dataset to Harbor, you create an [adapter](https://github.com/harbor-framework/harbor/tree/main/adapters) that translates the original benchmark's tasks into Harbor format.

<Callout title="Using AI to build your adapter?">
  AI agents should follow the spec at [Adapter AI Guideline](/docs/datasets/adapters)
  instead of this page. That document contains the complete schema,
  all edge cases, and machine-verifiable examples.
  Do not use the tutorial below as your source of truth.
</Callout>

<Callout title="Need help or want to contribute?">
  Join our [Discord](https://discord.gg/QVvyhRw5UQ) (`#adapters-announcements`) and reach out to [Lin Shi](mailto:ls2282@cornell.edu). Check the [Adapter List](https://docs.google.com/spreadsheets/d/1mJbiASPm32DDNzEnV6eDGwpEf3FlMUe5dhkZmFjjSoo/edit?gid=0#gid=0) for available benchmarks. We cover API costs for parity experiments.
</Callout>

## Quick Start

```bash
# List available datasets
harbor dataset list

# Scaffold a new adapter interactively
harbor adapter init

# Or with arguments
harbor adapter init my-adapter --name "My Benchmark"
```

## Steps at a Glance

| # | Step                                                               | Goal                                                                         |
| - | ------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| 1 | [Understand the benchmark](#1-understand-the-original-benchmark)   | Identify instructions, environments, tests, and solutions                    |
| 2 | [Complete the adapter code](#2-complete-the-adapter-code)          | Fill in the scaffolded adapter to generate Harbor-format task directories    |
| 3 | [Verify oracle solutions](#3-verify-oracle-solutions)              | All oracle solutions pass at 100% reward                                     |
| 4 | [Plan parity & implement agents](#4-plan-parity--implement-agents) | Coordinate with the team; set up agents on both sides                        |
| 5 | [Run parity experiments](#5-run-parity-experiments)                | Compare Harbor vs. original benchmark scores                                 |
| 6 | [Record parity results](#6-record-parity-results)                  | Save results to `parity_experiment.json`                                     |
| 7 | [Upload results](#7-upload-results)                                | Push to HuggingFace parity dataset                                           |
| 8 | [Register the dataset](#8-register-the-dataset)                    | Prepare dataset with `harbor init` and `dataset.toml`, submit for publishing |
| 9 | [Document & submit](#9-document--submit)                           | Write README, submit PR for review                                           |

***

## 1. Understand the Original Benchmark

Before coding, study the original benchmark and identify four key components:

1. **Task Instructions:** How are tasks described? What do agents need?
2. **Environments:** What setup is required? (Docker, dependencies, file structures)
3. **Tests:** How are solutions evaluated? (unit tests, LLM-as-a-Judge, etc.)
4. **Solutions:** What are the oracle/reference solutions?

## 2. Complete the Adapter Code

<Callout title="Supervising AI on format compliance">
  If you are using an AI agent to help write your adapter, make sure it strictly follows the format specs. Harbor runs automated parsing scripts to extract key information from `adapter_metadata.json`, `parity_experiment.json`, and other structured files. Format mismatches will cause extraction failures. Detailed explanations and context belong in the `"notes"` fields of JSON files or in the README, not in the structured data fields.
</Callout>

### 2.1 Fork and branch

```bash
git clone https://github.com/{you}/harbor.git
cd harbor
git checkout -b {adapter-name}-adapter
```

### 2.2 Generate adapter boilerplate

Run `harbor adapter init` to scaffold your adapter. This creates the following package structure under `harbor/adapters/{adapter-name}/`:

```
adapters/{adapter-name}/
├── .python-version              # (Optional)
├── pyproject.toml               # Python package config
├── README.md                    # Requirements checklist and final documentation
├── adapter_metadata.json        # Structured metadata about the adapter
├── parity_experiment.json       # Parity results (filled in later)
├── run_{adapter-name}.yaml      # Reference config to run the full adapted dataset
└── src/{adapter_name}/          # adapter-name with dashes replaced by underscores
    ├── __init__.py
    ├── adapter.py               # Core logic: parse benchmark data, generate task dirs
    ├── main.py                  # CLI entry point (--output-dir, --limit, --overwrite, --task-ids)
    └── task-template/           # Template files copied into each generated task
        ├── task.toml
        ├── instruction.md
        ├── environment/
        │   └── Dockerfile
        ├── solution/
        │   └── solve.sh
        └── tests/
            └── test.sh
```

### 2.3 Fill in adapter code

Complete `adapter.py` and `main.py` so that running the adapter produces a valid task directory for each benchmark task. Each generated task must satisfy Harbor's [task format](/docs/tasks) by following this structure:

<Files>
  <Folder name="<adapter-name>" defaultOpen>
    <Folder name="<task-id>" defaultOpen>
      <File name="task.toml" />

      <File name="instruction.md" />

      <Folder name="environment" defaultOpen>
        <File name="Dockerfile" />
      </Folder>

      <Folder name="solution" defaultOpen>
        <File name="solve.sh" />
      </Folder>

      <Folder name="tests" defaultOpen>
        <File name="test.sh" />

        <File name="test_*.py (optional)" />
      </Folder>
    </Folder>
  </Folder>
</Files>

Each `task.toml` must contain a valid, unique `name` field that identifies the task in the registry. Sanitize upstream identifiers (lowercase, replace special characters with hyphens) so the resulting names are stable and registry-safe. The author name and email in `task.toml` refer to the original benchmark authors, not the adapter contributor. See [§8 Tips](#8-register-the-dataset) for the full naming guidance.

**Running the adapter:**

```bash
uv run python -m {adapter_name}.main --output-dir <path>
```

**Tips:**

* Minor prompt tweaks (e.g., "write files in place without asking") are fine, as long as they apply to both the original benchmark and Harbor sides.
* Adapting only a subset of tasks is acceptable if documented in the README.
* If your benchmark requires GPU, add a `docker-compose.yaml` with nvidia device reservations in the task's `environment/` directory for Docker runs. For cloud/Modal runs, also set `gpus` in `task.toml`. See the [featurebench adapter](https://github.com/harbor-framework/harbor/tree/main/adapters/featurebench) for a comprehensive example with separate CPU/GPU/Modal configs.

### 2.4 Create `run_{adapter-name}.yaml` config files

Create one or more YAML config files that serve as the entry point for running experiments. Keep the oracle agent as the default (uncommented) and include other agents as commented-out alternatives so anyone can quickly switch.

If your benchmark has multiple variants (CPU vs. GPU, different splits, different cloud providers), create separate config files for each. The [featurebench adapter](https://github.com/harbor-framework/harbor/tree/main/adapters/featurebench) is a good example with configs for different scenarios:

| Config file                    | Purpose                                    |
| ------------------------------ | ------------------------------------------ |
| `featurebench_docker_cpu.yaml` | 156 CPU-only tasks, Docker environment     |
| `featurebench_docker_gpu.yaml` | 44 GPU-dependent tasks, Docker environment |
| `featurebench_modal.yaml`      | All 200 tasks, Modal environment           |
| `featurebench_parity.yaml`     | Parity validation subset                   |
| `featurebench_lite_*.yaml`     | Lite split variants (30 tasks)             |

## 3. Verify Oracle Solutions

Run your adapter with the oracle agent and confirm **100% reward on all tasks**. Validating oracle solutions is a straightforward way to check:

1. **Adaptation correctness:** a wrong adaptation usually causes oracle failures.
2. **Oracle solution bugs:** cross-validate with the original benchmark to determine whether a failure is due to the solution itself or the Harbor adaptation.
3. **Environment issues:** Docker build failures or broken verification tests make tasks impossible to solve, so catching these early is critical.

```bash
# Single task
harbor trial start -p datasets/<adapter-name>/<task-id>

# Entire dataset
harbor run -p datasets/<adapter-name>

# With a config file (recommended for reproducibility)
harbor run -c adapters/<adapter-name>/<config>.yaml -a <agent> -m <model>
```

Once oracle passes, create a **WIP PR** titled `[WIP] Adapter: {adapter_name}` with a screenshot of the 100% pass results in the PR description.

<details>
  <summary>
    **Broken oracles in the original benchmark?**
  </summary>

  If a fix is straightforward, propose it on the original fork and upstream repo as a GitHub Issue or PR, then document the fix clearly in the adapter README. This is more robust and transparent than patching on the Harbor side. For tasks that cannot be reliably fixed or verified, document them and exclude them from the dataset.
</details>

<details>
  <summary>
    **Original benchmark does not provide solutions?**
  </summary>

  Usually we require adapter contributors to build the oracle solutions themselves with the help of AI. Building oracle solutions is a separate process from running parity experiments (specified below), so they can theoretically progress in parallel. However, before running parity, you need to validate that the tasks are theoretically solvable by agents with no environment or test issues.

  A recommended approach is to use a cheap agent and model to do a pass over all the tasks. For building oracle solutions, you can also directly use the successful agent solutions from parity experiments and then complete the remaining ones with more powerful AI plus human supervision.

  If things become more complicated than this, please reach out to the team on Discord and discuss case-by-case.
</details>

## 4. Plan Parity & Implement Agents

Reach out to the team (e.g., **Lin Shi**) on [Discord](https://discord.gg/QVvyhRw5UQ) **before** running parity experiments. They will help decide:

* Which agents and models to use
* How many runs are needed
* API key provisioning

### Agent design

Depending on your benchmark, you'll fall into one of three scenarios:

**Scenario 1: Compatible agents exist.** The original benchmark already supports Harbor-compatible agents (OpenHands, Codex, Claude-Code, etc.). No extra work needed. Example: [ADEBench](https://github.com/harbor-framework/harbor/tree/main/adapters/adebench), where the original benchmark already supports Claude Code.

**Scenario 2: LLM-based, no compatible agents.** Fork the original benchmark, implement a Harbor-compatible agent there, and document it. Example: [EvoEval](https://github.com/harbor-framework/harbor/tree/main/adapters/evoeval), which forked the repo to add codex agent support for parity.

**Scenario 3: Custom agents.** The original benchmark uses custom agents unavailable in Harbor. Implement the custom agent under `adapters/{name}/` and run parity experiments with it. Also run experiments with standard agents (Codex, Claude-Code) to show the adapter generalizes. There are two sub-cases here:

* Some benchmarks require registering a separate dataset for CLI-agent compatibility. Example: [BixBench](https://github.com/harbor-framework/harbor/tree/main/adapters/bixbench), [FinanceAgent](https://github.com/harbor-framework/harbor/tree/main/adapters/financeagent) (also demonstrates LLM-as-a-Judge verification).
* Others do not need a separate dataset. Example: [MedAgentBench](https://github.com/harbor-framework/harbor/tree/main/adapters/medagentbench) supports a custom HTTPAgent matching the original GET/POST/FINISH interaction semantics.
* For multi-agent workflows where multiple agents coordinate in parallel (e.g., via Redis messaging and Docker sidecars), see [CooperBench](https://github.com/harbor-framework/harbor/tree/main/adapters/cooperbench). Note that multi-agent benchmarks may not be compatible with standard single-agent CLI agents.

<Callout title="Large benchmarks">
  For expensive benchmarks, you can run parity on a representative subset. Discuss sampling strategy with the team first. Use `--split parity` in your adapter and ask the team to publish the parity subset under the `parity` tag so users can run `-d {name}@parity`. See the versioning tip in [§8](#8-register-the-dataset).
</Callout>

## 5. Run Parity Experiments

The purpose of parity experiments is to prove result equivalence between Harbor and the original benchmark. Run the **same agents, models, and settings** on both the original benchmark and your Harbor adapter, multiple times each. Report results as **mean ± sample SEM** (sample standard error of the mean) on both sides. They should be **comparable** to demonstrate equivalence.

```bash
# Harbor side
harbor run -p datasets/<adapter-name> -a <agent> -m <model>
```

See the [AI adapter guide](/docs/datasets/adapters#reporting-format-mean--sample-sem) for the exact SEM formula and why we report SEM rather than sample std.

## 6. Record Parity Results

Create `parity_experiment.json` in your adapter directory:

```json
[
  {
    "adapter_name": "<adapter-name>",
    "agent": "<agent>@<version>",
    "model": "<model-version>",
    "date": "<date>",
    "adapted_benchmark_size": "<total-tasks-converted>",
    "parity_benchmark_size": "<tasks-used-for-parity>",
    "number_of_runs": "<runs-per-side>",
    "notes": "<any special notes>",
    "original_parity_repo": "<fork-url>",
    "adapter_pr": ["<pr-url>"],
    "dataset_pr": ["<pr-url>"],
    "parity_pr": ["<hf-pr-url>"],
    "metrics": [
      {
        "benchmark_name": "<name>",
        "metric": "<metric>",
        "original": "<mean ± sample SEM>",
        "harbor": "<mean ± sample SEM>",
        "original_runs": ["<run1>", "<run2>", "..."],
        "harbor_runs": ["<run1>", "<run2>", "..."]
      }
    ]
  }
]
```

Also include a summary table in your README. Values are formatted as `mean ± sample SEM`:

```markdown
| Agent | Model | Metric | Runs | Dataset Size | Original (mean ± SEM) | Harbor (mean ± SEM) |
|-------|-------|--------|------|--------------|-----------------------|---------------------|
| codex@0.1.2 | gpt-5 | pass@1 | 5    | 2000 (100%)  | X ± Y                 | X ± Y               |
```

## 7. Upload Results

Upload parity and oracle results to the [HuggingFace parity-experiments dataset](https://huggingface.co/datasets/harborframework/parity-experiments). The [parity upload skill](https://github.com/harbor-framework/harbor/tree/main/skills/upload-parity-experiments) can automate this workflow.

```
adapters/<adapter_name>/
├── README.md
├── config.yaml
├── original_parity/
├── harbor_parity/
├── oracle/
└── results_collection/
    ├── result_{original/harbor}_trial1.json
    └── ...
```

## 8. Register the Dataset

A dataset is a collection of tasks, and the two have a many-to-many relationship: the same task can live in multiple datasets, and one dataset can aggregate tasks from multiple adapters. Pick an organization name following the guidance in the Tips section below. The dataset is named `{organization}/{dataset}`, and tasks are named `{organization}/{task-id}`.

**8.1.** Generate the dataset directory with your adapter code. Store it in the [Github repo](https://github.com/laude-institute/harbor-datasets), or in the [HuggingFace repo](https://huggingface.co/datasets/harborframework/harbor-datasets) if the dataset is too large for GitHub.

```bash
git clone https://github.com/{you}/harbor-datasets.git
cd harbor/adapters/<adapter-name>
uv run python -m <adapter_name>.main --output-dir /path/to/harbor-datasets/datasets/<adapter-name>
```

**8.2.** Generate `dataset.toml` once your generated tasks are finalized.

```bash
cd harbor-datasets/datasets/<adapter-name>
harbor init
# Select "dataset" when prompted, then enter the dataset name as <org>/<dataset>.
```

**8.3.** Edit the generated `dataset.toml` to fill in the description. Include the parity results summary, adapter author credits, and any acknowledgments.

**8.4.** Verify the dataset runs locally before submitting, using the `-p` (path) parameter:

```bash
harbor run -p /path/to/your/dataset
```

<Callout title="Registry testing is only available post-publish">
  You cannot test against the registry (using `-d`) until the dataset has been published. Use `-p` for all pre-publish testing.
</Callout>

**8.5.** Open a PR to `harbor-datasets` with the tasks directory and `dataset.toml`. Request `@Slimshilin` for review. Once approved, the Harbor team will publish the dataset to the registry.

**8.6.** After publishing, verify the dataset loads and runs from the registry:

```bash
harbor run -d <organization-name>/<adapter-name>
```

**Tips:**

* **Authors:** if there are many benchmark authors, list the first authors only.
* **Organization:** the `organization` namespace disambiguates tasks that share a name across adapters. Prefer the benchmark's owning organization (e.g., `openai/mmmlu`). If there's no clear single owner or there are multiple, use the benchmark name itself as the org (e.g., `terminal-bench/terminal-bench`).
* **Task names:** every task must have a `name` field in `task.toml` to be included in a dataset. If the original benchmark lacks stable identifiers, create your own deterministic scheme (e.g., `{dataset}-1`, `{dataset}-2`, ...).
* **Versioning:** set the package version under `[dataset]` in `dataset.toml`; semantic versions are recommended but not required. Registry tags remain separate: tell the Harbor team in your PR which tags you'd like (e.g., `v1.0`, `parity`). Users resolve a tag via `-d <org>/<adapter_name>@<tag>`.

## 9. Document & Submit

Fill in the README that was generated by `harbor adapter init`. It should cover:

* Benchmark bugs discovered and how they were handled
* Special treatments (prompt tweaks, environment adjustments)
* Deviations from the original and why
* Agent implementation details
* Known limitations
* Reproduction scripts for parity experiments on both the original benchmark and Harbor sides

If you forked the original benchmark repository for parity (Scenario 2 or 3), also update the fork's README to include reproduction scripts for running Harbor parity experiments. This makes it easy for others to reproduce results on the original benchmark side.

When ready, update your PR title from `[WIP]` to `[Ready for Review]` and request review from `@Slimshilin`.

***

## Appendix: Terminal-Bench Migration

If you're converting a Terminal-Bench adapter, here are the key differences:

| Aspect       | Terminal-Bench               | Harbor                                             |
| ------------ | ---------------------------- | -------------------------------------------------- |
| Config       | `task.yaml`                  | `task.toml`                                        |
| Instruction  | In `task.yaml`               | Separate `instruction.md`                          |
| Dockerfile   | Root level                   | `environment/Dockerfile`                           |
| Solution     | `solution.sh`                | `solution/solve.sh`                                |
| Tests        | `run-tests.sh` + `tests/`    | `tests/test.sh`                                    |
| Verification | Exit code (pytest)           | Reward file: `/logs/verifier/reward.txt`           |
| Output dir   | `tasks/`                     | `datasets/`                                        |
| Registry     | Dataset-level `dataset_path` | `dataset.toml` + `harbor init` publishing workflow |
| CLI          | `tb run --dataset`           | `harbor run -d` / `harbor run -t` /`harbor run -p` |
| Metrics      | Binary pass/fail             | Float rewards, multiple metrics                    |

**Important:** If Terminal-Bench used a tweaked metric, re-implement to support the **original** benchmark metrics. Harbor supports multiple metrics as rewards.

Migration checklist:

1. Convert `task.yaml` → `task.toml` + `instruction.md`
2. Reorganize files into `environment/`, `solution/`, `tests/` subdirs
3. Update test scripts to write rewards to `/logs/verifier/reward.txt`
4. Change output directory from `tasks/` to `datasets/`
5. Update registry format using `harbor init` and `dataset.toml`

***

## Resources

* [Harbor docs](/docs/getting-started): Running tasks and jobs
* [Harbor repo](https://github.com/harbor-framework/harbor): Examples and configs
* [Agent tutorial](/docs/agents): Creating custom agents
* [Discord](https://discord.gg/QVvyhRw5UQ): Ask questions in `#adapters-spam`


# Adapters (Agent Guide)



import { Callout } from 'fumadocs-ui/components/callout';

<Callout title="Looking for the human-readable guide?">
  This page is the comprehensive spec optimized for AI agents. For a concise walkthrough, see the [Adapters (Human Guide)](/docs/datasets/adapters-human).
</Callout>

## Purpose

An adapter translates an existing benchmark into Harbor's task format. This document is the authoritative reference for building one. Follow steps 1-9 in order.

Check the [Adapter List](https://docs.google.com/spreadsheets/d/1mJbiASPm32DDNzEnV6eDGwpEf3FlMUe5dhkZmFjjSoo/edit?gid=0#gid=0) for available benchmarks. Contact [Lin Shi](mailto:ls2282@cornell.edu) or join [Discord](https://discord.gg/QVvyhRw5UQ) `#adapters-announcements` for coordination. The team covers API costs for parity experiments.

## Quick Start

```bash
harbor dataset list                              # list available datasets
harbor adapter init                              # interactive scaffold
harbor adapter init my-adapter --name "My Name"  # non-interactive scaffold
```

## Required Directory Structures

### Generated task directory (one per task)

```
<adapter-name>/
└── <task-id>/
    ├── task.toml              # task configuration and metadata
    ├── instruction.md         # task instructions for the agent
    ├── environment/
    │   └── Dockerfile         # container environment definition
    ├── solution/
    │   └── solve.sh           # oracle solution script
    └── tests/
        ├── test.sh            # test execution script
        └── test_*.py          # (optional) pytest test files
```

**Task naming requirement:** Every generated `task.toml` **must** contain a `name` field. Harbor uses this field to identify the task when it's added to a dataset; tasks without a `name` cannot be registered. Adapter code is responsible for deriving a valid, unique, registry-safe name for every task: sanitize upstream identifiers (lowercase, replace spaces/slashes/special characters with hyphens). See [§Step 8 Naming rules](#naming-rules) for the full naming contract, and the [task format](/docs/tasks) for the rest of the task structure. Each generated directory must contain at minimum `task.toml`, `instruction.md`, `environment/Dockerfile`, `solution/solve.sh`, and `tests/test.sh`.

### Adapter code directory

Generated by `harbor adapter init`, this is a Python package using `src` layout:

```
harbor/adapters/<adapter-name>/
├── .python-version            # Python version (optional, created by uv init)
├── pyproject.toml             # Python package config (created by uv init)
├── README.md                  # final documentation (step 9)
├── adapter_metadata.json      # structured metadata (step 9)
├── parity_experiment.json     # parity results (step 6)
├── run_<adapter-name>.yaml    # reference config to run the full adapted dataset
└── src/
    └── <adapter_name>/         # adapter-name with dashes → underscores
        ├── __init__.py
        ├── adapter.py         # main logic: parse benchmark, generate task dirs
        ├── main.py            # CLI entry point (must support --output-dir)
        └── task-template/     # template files copied into each task
            ├── task.toml
            ├── instruction.md
            ├── environment/
            │   └── Dockerfile
            ├── solution/
            │   └── solve.sh
            └── tests/
                └── test.sh
```

### Key requirements for `main.py`

* Must support `--output-dir` to specify where generated tasks are written.
* Must support `--limit`, `--overwrite`, and `--task-ids` flags.
* Run via `uv run python -m <adapter_name>.main --output-dir <path>`.

***

## Step 1. Understand the Original Benchmark

Identify these four components for every task in the benchmark:

| Component        | What to find                                                                  |
| ---------------- | ----------------------------------------------------------------------------- |
| **Instructions** | How tasks are described; what information agents receive                      |
| **Environments** | Docker setup, system dependencies, file structures                            |
| **Tests**        | Evaluation method: deterministic unit tests, LLM-as-a-Judge, etc.             |
| **Solutions**    | Oracle/reference solutions; if none exist, whether LLM generation is feasible |

Study the benchmark's repository, documentation, and code structure.

**Step complete when:** You can describe, for each task, the instruction text, environment setup, test/verification method, and oracle solution.

***

## Step 2. Fork and Develop Adapter Code

Run `harbor adapter init` to scaffold your adapter. The generated `README.md` doubles as a requirements checklist.

```bash
git clone https://github.com/{your-github-username}/harbor.git
cd harbor
git checkout -b {your-adapter-name}-adapter
```

Develop your adapter under `adapters/{adapter-name}/`. Refer to existing adapters in that directory.

### Adapter component reference

| Component                           | Description                                                                                                               |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `src/<adapter_name>/adapter.py`     | Core logic: parse benchmark data, generate task directories.                                                              |
| `src/<adapter_name>/main.py`        | CLI entry point. Must support `--output-dir`, `--limit`, `--overwrite`, `--task-ids`.                                     |
| `src/<adapter_name>/task-template/` | Template files copied into each generated task.                                                                           |
| `parity_experiment.json`            | Parity results (see [Step 6](#step-6-record-parity-results) for full schema)                                              |
| `run_{adapter-name}.yaml`           | Reference config to run the full adapted dataset                                                                          |
| `README.md`                         | Write last before PR submission. Fill in the README generated by `harbor adapter init`.                                   |
| Metrics / Rewards                   | Harbor supports multiple float-valued metrics as rewards (RL-compatible). Use the same metrics as the original benchmark. |

> **Targeting Windows containers.** Adapters that emit Windows-targeted tasks must set `[environment].os = "windows"` in `task.toml` and ship `solve.bat` / `test.bat` instead of `.sh`. Users who need PowerShell can call it from within a `.bat` file. Linux is the default and requires no change. See [Windows tasks](/docs/tasks/windows-container-support).

### Task file reference

**`task.toml`:** Every task must include this configuration file. The `name` field is required for registry. The `version` field must stay `"1.0"`. Adjust timeouts to match your benchmark's complexity.

```toml
version = "1.0"

[task]
name = "my-benchmark/task-001"

[metadata]
author_name = "Original benchmark authors' names"
author_email = "benchmark-authors@email.com"
difficulty = "medium"
category = "programming"
tags = ["debugging", "python"]

[agent]
timeout_sec = 1800.0

[verifier]
timeout_sec = 120.0

[environment]
build_timeout_sec = 600.0
cpus = 1
memory_mb = 2048
storage_mb = 10240
```

For LLM-as-a-Judge verifiers, add a `[verifier.env]` section to pass the judge model and API key:

```toml
[verifier.env]
OPENAI_API_KEY = "${OPENAI_API_KEY}"
MODEL_NAME = "openai/gpt-5-2025-08-07"
```

**`tests/test.sh`:** Must write a numeric reward (integer or float, 0 to 1) to `/logs/verifier/reward.txt`. Harbor mounts `/logs/verifier/` at runtime. The script should exit 0 on success and non-zero on failure.

```bash
#!/bin/bash
pytest /tests/test_*.py
if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```

**`instruction.md`:** Write agent-actionable instructions, not raw benchmark descriptions. The agent reads this file to understand what to do. Include the goal, constraints, expected output location, and any files the agent should modify. Do not include test answers or oracle solutions.

**`environment/Dockerfile`:** Set up the container the agent will work in. Install system and Python dependencies, copy any benchmark-specific data files, and set the working directory. The agent and verifier both run inside this container. No terminal-bench or harbor canary string should occur here.

```dockerfile
FROM python:3.13-slim
WORKDIR /workspace

RUN apt-get update && apt-get install -y \
    git \
    && rm -rf /var/lib/apt/lists/*

# Install benchmark-specific dependencies
# RUN pip install --no-cache-dir <packages>

# Copy task-specific files
# COPY . /workspace/
```

### GPU tasks

If your benchmark includes tasks that require GPU (e.g., CUDA, Triton kernels), add a `docker-compose.yaml` in the task's `environment/` directory with nvidia device reservations for Docker runs. For cloud/Modal runs, also set `gpus` in `task.toml`. See the [featurebench adapter](https://github.com/harbor-framework/harbor/tree/main/adapters/featurebench) for a comprehensive example; it handles 44 GPU tasks across multiple repos with separate CPU/GPU/Modal config files.

### Writing `run_{adapter-name}.yaml`

This config file serves as the single entry point for all experiments: oracle verification, parity runs, and general benchmarking. Keep the oracle agent as the default (uncommented) and include other agents as commented-out alternatives so anyone can quickly switch.

```yaml
datasets:
  - path: datasets/<adapter-name>

# Default: oracle agent for verification
agents:
  - name: oracle

# Uncomment to run with other agents:
# agents:
#   - name: codex
#     model_name: openai/gpt-5-mini
#
# agents:
#   - name: claude-code
#     model_name: claude-sonnet-4-5-20250929

environment:
  type: docker
  delete: true

orchestrator:
  type: local
  n_concurrent_trials: 4
```

You can also create additional config files for different scenarios (e.g., parity subsets, CPU-only vs GPU, Modal). For example, featurebench provides `featurebench_docker_cpu.yaml`, `featurebench_docker_gpu.yaml`, `featurebench_modal.yaml`, and `featurebench_parity.yaml`.

Usage:

```bash
# Oracle verification (default)
harbor run -c adapters/<adapter-name>/run_<adapter-name>.yaml

# Switch agent by uncommenting the desired agent block
```

### Rules

* Prompt modifications (e.g., "write files in place without asking") are acceptable **if applied to both the original benchmark and Harbor adapter**.
* Adapting a subset of tasks is acceptable (e.g., only SWE-Bench-Verified). **Document all exclusions in the README.**

**Step complete when:** `main.py` produces a valid task directory for each task containing `task.toml`, `instruction.md`, `environment/Dockerfile`, `solution/solve.sh`, and `tests/test.sh`.

***

## Step 3. Verify Oracle Solutions

Run your adapter with the oracle agent and confirm **100% reward on all tasks**. Validating oracle solutions is a straightforward way to check:

1. **Adaptation correctness:** a wrong adaptation usually causes oracle failures.
2. **Oracle solution bugs:** cross-validate by running oracle on the original benchmark side as well to determine whether a failure is due to the solution itself or the Harbor adaptation.
3. **Environment issues:** Docker build failures or broken verification tests make tasks impossible for agents to solve, so catching these early is critical.

### Run commands

| Method                 | Command                                                                         | When to use                                                                                                      |
| ---------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Single task            | `harbor trial start -p datasets/<adapter-name>/<task-id> -a <agent> -m <model>` | Testing individual tasks                                                                                         |
| Entire dataset         | `harbor run -p datasets/<adapter-name> -a <agent> -m <model>`                   | Full oracle verification                                                                                         |
| Config file            | `harbor run -c adapters/<adapter-name>/<config>.yaml -a <agent> -m <model>`     | Reproducible runs (see [example configs](https://github.com/harbor-framework/harbor/tree/main/examples/configs)) |
| Registry: single task  | `harbor run -t <organization-name>/<task-id> -a <agent> -m <model>`             | Post-publish single task                                                                                         |
| Registry: full dataset | `harbor run -d <organization-name>/<adapter-name> -a <agent> -m <model>`        | Post-publish full dataset (after [Step 8](#step-8-register-the-dataset))                                         |

Write a reference config YAML for your adapter to ensure reproducibility.

**README ordering note:** In the final adapter README, list the registry method (Option 5) first; it is the primary user-facing run method. Adapter code and local-path methods are for development/reproduction.

### After oracle passes

1. Create a WIP PR titled `[WIP] Adapter: {adapter_name}`.
2. Include a screenshot of the terminal showing 100% oracle pass results.

### Broken oracles in the original benchmark

If a fix is straightforward, propose it on the original fork and upstream repo as a GitHub Issue or PR, then document the fix clearly in the adapter README. This is more robust and transparent than patching on the Harbor side. For tasks that cannot be reliably fixed or verified, document them and exclude them from the dataset.

### Benchmarks without oracle solutions

Usually we require adapter contributors to build the oracle solutions themselves with the help of AI. Building oracle solutions is a separate process from running parity experiments (Step 5), so they can progress in parallel. However, before running parity, you need to validate that the tasks are theoretically solvable by agents with no environment or test issues.

A recommended approach is to use a cheap agent and model to do a pass over all the tasks. For building oracle solutions, you can also directly use the successful agent solutions from parity experiments and then complete the remaining ones with more powerful AI plus human supervision. If things become more complicated than this, reach out to the team on Discord and discuss case-by-case.

**Step complete when:** All oracle solutions pass with 100% reward, and a WIP PR titled `[WIP] Adapter: {adapter_name}` is created with a screenshot of the passing results.

***

## Step 4. Discuss Parity Plans and Implement Agents

Contact the team (e.g., **Lin Shi** on [Discord](https://discord.gg/QVvyhRw5UQ)) **before** running parity experiments. They determine agents, models, number of runs, and API key provisioning.

### Agent implementation scenarios

**Scenario 1: Compatible agents exist.** The original benchmark already supports Harbor-compatible agents (OpenHands, Codex, Claude-Code, Gemini-CLI). No extra work needed; run parity with identical settings on both sides. Example: [ADEBench](https://github.com/harbor-framework/harbor/tree/main/adapters/adebench), where the original benchmark already supports Claude Code.

**Scenario 2: LLM-based, no compatible agents.** Fork the original benchmark, implement a Harbor-compatible agent there, and document it in the fork's README. Example: [EvoEval](https://github.com/harbor-framework/harbor/tree/main/adapters/evoeval), which forked the repo to add codex agent support for parity.

**Scenario 3: Custom agents.** The original benchmark uses custom agents unavailable in Harbor. Implement the custom agent under `adapters/{name}/` and run parity experiments with it. Also run experiments with standard agents (Codex, Claude-Code) to show the adapter generalizes. There are two sub-cases here:

* Some benchmarks require registering a separate dataset for CLI-agent compatibility. Example: [BixBench](https://github.com/harbor-framework/harbor/tree/main/adapters/bixbench), [FinanceAgent](https://github.com/harbor-framework/harbor/tree/main/adapters/financeagent) (also demonstrates LLM-as-a-Judge verification).
* Others do not need a separate dataset. Example: [MedAgentBench](https://github.com/harbor-framework/harbor/tree/main/adapters/medagentbench) supports a custom HTTPAgent matching the original GET/POST/FINISH interaction semantics.
* For multi-agent workflows where multiple agents coordinate in parallel (e.g., via Redis messaging and Docker sidecars), see [CooperBench](https://github.com/harbor-framework/harbor/tree/main/adapters/cooperbench). Note that multi-agent benchmarks may not be compatible with standard single-agent CLI agents.

Keep links to any forked repositories and document the approach in the README.

### Large or expensive benchmarks

If running the full benchmark is too expensive, run parity on a representative subset. Requirements:

* Document in README how the subset was selected and that parity ran on a subset.
* Support `--split parity` in `main.py` to generate only the parity subset.
* Ask the team to publish the parity subset under the `parity` tag so users can run `-d <adapter_name>@parity`. See [Versioning](#versioning) below.

```bash
uv run python -m <adapter_name>.main --split parity --output-dir /path/to/output  # parity subset
uv run python -m <adapter_name>.main --output-dir /path/to/output                 # full dataset
```

**Step complete when:** Parity plan is agreed with the team (agents, models, number of runs), and any required agent implementations are working on both the original benchmark and Harbor sides.

***

## Step 5. Run Parity Experiments

### Definition

Parity is the property that the Harbor adaptation and the original benchmark measure the same quantity. Under identical conditions — same agent, model, prompts, and configuration — scores on both sides should be statistically indistinguishable. An adapter without demonstrated parity cannot be accepted, because its results would not be comparable to results reported against the upstream benchmark.

### Matching criterion

The two sides match if and only if their run-score ranges overlap:

```
max(side_A_runs) >= min(side_B_runs)  AND  max(side_B_runs) >= min(side_A_runs)
```

Equivalently, neither side's lowest run may exceed the other side's highest run. Close means with non-overlapping ranges do not constitute a match. Non-overlap should be treated as a systematic adaptation error until evidence indicates otherwise.

### Reporting format: mean ± sample SEM

All scalar-uncertainty fields in `parity_experiment.json` and the README parity table (`original`, `harbor`, and any `Score ± …` column) MUST be reported as **mean ± sample standard error of the mean (sample SEM)** — not sample standard deviation. Parity is a claim about how well each side estimates the true benchmark score; SEM is the uncertainty of that estimate and shrinks as runs are added, while sample std describes per-run spread and does not. Reporting sample std inflates apparent uncertainty and can mask a genuinely diverging adapter.

For `n ≥ 2` runs with per-run scores `x₁, x₂, …, xₙ` and mean `x̄`:

```
sample SEM = sqrt( Σ (xᵢ - x̄)² / ( n (n - 1) ) )
```

Notes: SEM is undefined for `n < 2` (require ≥ 2 runs per side; 3+ preferred). `original_runs` / `harbor_runs` are the source of truth — reviewers recompute from them to verify the reported string. Keep units consistent with the raw runs (don't mix `45.2` and `45.2%`).

### Checklist BEFORE any parity run

<Callout title="Required execution order" type="warn">
  Full runs must be completed symmetrically. The required order is: sanity check on 5-10 tasks (both sides) → one full run (both sides) → scale to three runs (both sides). Completing three runs on one side before the other is not permitted, because an adaptation bug discovered later forces the work to be re-run and incurs avoidable cost.
</Callout>

Before starting a parity run, confirm each item below on both the original-benchmark side and the Harbor side, and record the values in the `notes` field of `parity_experiment.json` (or the parity notes section of the README) for reproducibility.

| # | Check                            | Requirement                                                                     | Verification                                                                                                                                                     |
| - | -------------------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | Harbor is on the latest upstream | `main` is not behind `origin/main`                                              | Run `git fetch origin && git status`; pull if behind. Stale checkouts miss recent adapter and agent updates and are a common source of spurious parity failures. |
| 2 | Agent version                    | Pinned and identical on both sides                                              | For example, `codex@0.1.0`. Avoid `latest` and unpinned installs.                                                                                                |
| 3 | Model ID                         | Exact dated identifier, identical on both sides                                 | For example, `claude-sonnet-4-5-20250929`. Avoid floating aliases.                                                                                               |
| 4 | Installation script              | Identical on both sides                                                         | Diff the install commands (package-manager or curl lines).                                                                                                       |
| 5 | Execution / entry command        | Identical flags, environment variables, and working directory                   | Diff the launch command used on each side.                                                                                                                       |
| 6 | Tools, arguments, and parameters | Identical on both sides                                                         | Includes temperature, max tokens, tool list, system prompt, turn limit, and timeouts.                                                                            |
| 7 | Prompt modifications             | Applied symmetrically                                                           | Any prompt change on one side must be applied on the other.                                                                                                      |
| 8 | Ramp-up order                    | sanity (5-10 tasks, both sides) → 1 full run (both sides) → 3 runs (both sides) | See the execution-order callout above.                                                                                                                           |

### Run command

```bash
harbor run -p datasets/<adapter-name> -a <agent> -m <model>
```

### Debug playbook (when parity does not match)

If scores do not overlap per the matching criterion, work through the steps below in order. Each step narrows the hypothesis space; re-running without diagnosis is not an acceptable substitute.

| # | Action                                        | When to run                                                              | Signal to look for                                                                                                                                                                       | Decision                                                                                                                                                                   |
| - | --------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | Resolve errors                                | Always first                                                             | Crashes, timeouts, Docker build failures, or verifier errors in the logs                                                                                                                 | Fix infrastructure and re-run before interpreting scores. A failed run should not be treated as a low score.                                                               |
| 2 | Inspect agent trajectories                    | After the logs are clean                                                 | On successes, whether the success is legitimate. On failures, whether the failure reflects a genuine capability gap or a harness issue (e.g., incorrect path, misread test output).      | If a harness issue is found, fix the adapter. Otherwise proceed.                                                                                                           |
| 3 | Overlap analysis                              | After trajectories look reasonable                                       | Per-task resolution on each side; list tasks resolved on A-only and on B-only.                                                                                                           | A roughly symmetric split suggests sampling noise (proceed to step 4). A lopsided split suggests systematic adaptation error (proceed to step 7 or return to the adapter). |
| 4 | Distinguish randomness from systematic error  | Disagreements persist across runs                                        | Tasks that fail on Harbor in every run (or succeed in every run) while flipping on the original side indicate a systematic issue. Disagreements that shuffle across runs indicate noise. | Systematic issues typically originate in the environment, instruction, test, or solution. Otherwise proceed to step 5.                                                     |
| 5 | Characterize high-variance tasks individually | A small number of tasks show large variance but appear correctly adapted | Run those tasks 5-10 times in isolation to estimate their variance                                                                                                                       | Use the resulting distribution to decide whether variance alone explains the gap. Avoid scaling the full dataset to investigate a few noisy tasks.                         |
| 6 | Lock configuration for all repeats            | Any time you re-run                                                      | Same Harbor commit, agent version, Docker base image, and environment variables                                                                                                          | Look up and record these values before running. Mixing configurations across repeats invalidates the comparison.                                                           |
| 7 | Diff the agent implementation                 | Tasks appear correctly adapted but scores still diverge                  | Differences in wrappers, tool definitions, prompt construction, retry logic, turn limits, or tool-output handling                                                                        | Any such difference is a candidate root cause and should be resolved before attributing the gap to the adapter.                                                            |
| 8 | Scale up only after fixes                     | After any corrective change                                              | 5-10 task sanity check → 1 full run (both sides) → 3 runs (both sides)                                                                                                                   | Do not complete three runs on one side before the other. See the execution-order callout above.                                                                            |

### Common sources of adaptation error

Any asymmetry between the two sides that is not attributable to sampling noise constitutes adaptation error. Check the following in order:

* Environment: Dockerfile, installed packages, working directory, file permissions, mounted paths.
* Instruction: wording changes, missing context, leaked solutions, additional constraints.
* Tests: differing assertions, tolerances, reward mappings, or timeouts.
* Solution: oracle passes on one side but not the other.
* Agent: installation, entry command, tools, arguments, system prompt, retry and turn logic.
* Harbor version: stale checkout missing a recent adapter or feature fix (revisit checklist item 1).

**Step complete when:** multiple runs on both sides satisfy the overlap criterion, and any remaining gaps are documented with evidence that they arise from sampling noise rather than adaptation error.

***

## Step 6. Record Parity Results

Create `parity_experiment.json` in your adapter directory. The file is a JSON array; each entry is one agent+model parity experiment.

### `parity_experiment.json` field reference

| Field                    | Type       | Required | Description                                                        |
| ------------------------ | ---------- | -------- | ------------------------------------------------------------------ |
| `adapter_name`           | `string`   | Yes      | Adapter name (e.g., `"swe-bench"`)                                 |
| `agent`                  | `string`   | Yes      | Agent with version (e.g., `"codex@1.0"`)                           |
| `model`                  | `string`   | Yes      | Full model identifier (e.g., `"gpt-5-2025-06-01"`)                 |
| `date`                   | `string`   | Yes      | Experiment date (e.g., `"2025-06-15"`)                             |
| `adapted_benchmark_size` | `integer`  | Yes      | Total tasks converted by adapter (full set)                        |
| `parity_benchmark_size`  | `integer`  | Yes      | Tasks used for parity. Equals `adapted_benchmark_size` if full set |
| `number_of_runs`         | `integer`  | Yes      | Runs per side. Should be identical for original and Harbor         |
| `notes`                  | `string`   | No       | Additional explanations                                            |
| `original_parity_repo`   | `string`   | Yes      | Fork URL for reproducing parity on original benchmark              |
| `adapter_pr`             | `string[]` | Yes      | All adapter PR links in `harbor` repo                              |
| `dataset_pr`             | `string[]` | Yes      | All PR links in `harbor-datasets` repo                             |
| `parity_pr`              | `string[]` | Yes      | All PR links to HuggingFace parity dataset                         |
| `metrics`                | `object[]` | Yes      | Metric comparison objects (see below)                              |

### `metrics` entry fields

| Field            | Type       | Required | Description                                                                                                         |
| ---------------- | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------- |
| `benchmark_name` | `string`   | Yes      | Original benchmark name                                                                                             |
| `metric`         | `string`   | Yes      | Metric name (e.g., `"pass@1"`, `"resolve_rate"`)                                                                    |
| `original`       | `string`   | Yes      | `mean ± sample SEM` on original (e.g., `"45.2 ± 1.3"`). See [Reporting format](#reporting-format-mean--sample-sem). |
| `harbor`         | `string`   | Yes      | `mean ± sample SEM` on Harbor (e.g., `"44.8 ± 1.1"`). See [Reporting format](#reporting-format-mean--sample-sem).   |
| `original_runs`  | `number[]` | Yes      | Individual scores per run on original                                                                               |
| `harbor_runs`    | `number[]` | Yes      | Individual scores per run on Harbor                                                                                 |

### Example

```json
[
  {
    "adapter_name": "my-benchmark",
    "agent": "codex@1.0",
    "model": "gpt-5-2025-06-01",
    "date": "2025-06-15",
    "adapted_benchmark_size": 500,
    "parity_benchmark_size": 500,
    "number_of_runs": 3,
    "notes": "None",
    "original_parity_repo": "https://github.com/user/my-benchmark-fork",
    "adapter_pr": ["https://github.com/harbor-framework/harbor/pull/123"],
    "dataset_pr": ["https://github.com/laude-institute/harbor-datasets/pull/45"],
    "parity_pr": ["https://huggingface.co/datasets/harborframework/parity-experiments/discussions/12"],
    "metrics": [
      {
        "benchmark_name": "my-benchmark",
        "metric": "pass@1",
        "original": "45.2 ± 1.3",
        "harbor": "44.8 ± 1.1",
        "original_runs": [44.0, 45.5, 46.1],
        "harbor_runs": [43.8, 45.0, 45.6]
      }
    ]
  }
]
```

### README parity table

Include this table in the adapter README. Scores are `mean ± sample SEM` as defined in [Reporting format](#reporting-format-mean--sample-sem):

```markdown
| Agent | Model | Metric | Runs | Dataset Size | Original (mean ± SEM) | Harbor (mean ± SEM) |
|-------|-------|--------|------|--------------|-----------------------|---------------------|
| codex@1.0 | gpt-5 | pass@1 | 5    | 2000 (100%)  | 45.2 ± 1.3            | 44.8 ± 1.1          |
```

Also include links to: original benchmark repo, forked repo (if applicable), dataset PR, HuggingFace parity PR, adapter PR.

**Step complete when:** `parity_experiment.json` is valid JSON, all required fields are populated, `original` and `harbor` are reported as `mean ± sample SEM` and are consistent with the raw `original_runs` / `harbor_runs` arrays, the run-score ranges overlap per the [matching criterion](#matching-criterion), and the README includes the parity summary table with links. If scores diverge significantly, investigate before proceeding.

***

## Step 7. Upload Parity Results

Upload parity and oracle results to [harborframework/parity-experiments](https://huggingface.co/datasets/harborframework/parity-experiments) on HuggingFace.

**Recommended:** Use the [parity upload skill](https://github.com/harbor-framework/harbor/tree/main/skills/upload-parity-experiments) to automate this. It handles sparse checkouts, LFS tracking, and HF-specific PR refs.

### Required directory structure

```
adapters/
└── {adapter_name}/
    ├── README.md
    ├── config.yaml
    ├── original_parity/
    ├── harbor_parity/
    ├── oracle/
    └── results_collection/
        ├── result_{original/harbor}_trial1.json
        ├── result_{original/harbor}_trial2.json
        └── result_{original/harbor}_trial{N}.json
```

**Step complete when:** PR to the HuggingFace parity-experiments dataset is submitted with all result files in the expected directory structure.

***

## Step 8. Register the Dataset

A dataset is a collection of tasks with a **many-to-many** relationship: the same task can appear in multiple datasets, and a dataset can aggregate tasks from multiple adapters. Both datasets and tasks are namespaced as `{organization}/{name}`, for example `{organization}/{dataset}` for a dataset and `{organization}/{task-id}` for a task.

**Step 1.** Generate the dataset directory with your adapter. Store it in the [harbor-datasets GitHub repo](https://github.com/laude-institute/harbor-datasets), or the [HuggingFace mirror](https://huggingface.co/datasets/harborframework/harbor-datasets) if it's too large for GitHub.

```bash
git clone https://github.com/{your-github-username}/harbor-datasets.git
cd harbor/adapters/<adapter-name>
uv run python -m <adapter_name>.main --output-dir /path/to/harbor-datasets/datasets/<adapter-name>
```

**Step 2.** Create `dataset.toml` at the root of the dataset directory (e.g., `harbor-datasets/datasets/<adapter-name>/dataset.toml`).

```bash
cd /path/to/harbor-datasets/datasets/<adapter-name>
harbor init  # select "dataset"; enter the dataset name as <org>/<dataset>
```

**Step 3.** Edit `dataset.toml` to fill in the description: parity results summary, adapter author credits (first authors only if the list is long), and acknowledgments.

**Step 4.** Verify the dataset runs locally before submitting, using the `-p` (path) parameter:

```bash
harbor run -p /path/to/your/dataset
```

**Note:** Registry testing (`-d`) is only available after publishing. Use `-p` for all pre-publish testing.

**Step 5.** Open a PR to `harbor-datasets` with the tasks directory and `dataset.toml`. Request `@Slimshilin` for review. The Harbor team publishes the dataset to the registry after approval.

**Step 6.** After publishing, verify the dataset loads and runs from the registry:

```bash
harbor run -d <organization-name>/<adapter-name>
```

#### Naming rules

| Rule             | Requirement                                                                                                                                                                                                            |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Dataset ID       | `<org>/<dataset_name>`, e.g., `openai/mmmlu`. Entered interactively during `harbor init`.                                                                                                                              |
| Task ID          | `<org>/<task_name>`. Every generated `task.toml` **must** contain a `name` field. Tasks without a name cannot be added to a dataset.                                                                                   |
| Choosing `<org>` | Prefer the benchmark's owning organization (e.g., `openai/mmmlu`). If there is no clear single owner or there are multiple, use the benchmark name itself as the organization (e.g., `terminal-bench/terminal-bench`). |
| Name stability   | Task names must be **unique** within the dataset and **stable** across adapter runs. Unstable names churn registry digests on republish.                                                                               |
| Fallback scheme  | If the upstream benchmark lacks stable task identifiers, mint a deterministic scheme in adapter code (e.g., `{dataset}-1`, `{dataset}-2`, ...) derived from a reproducible sort of upstream tasks.                     |
| Sanitization     | Sanitize upstream identifiers before using them as names: lowercase, replace spaces/slashes/special characters with hyphens, avoid leading/trailing separators.                                                        |

**Agent instruction:** before writing `dataset.toml`, verify every generated `task.toml` contains a `name` field. If any are missing, fix `main.py` and regenerate. Do not hand-edit generated task directories. Treat `main.py` as the source of truth for task names.

#### Versioning

Dataset manifests declare a package version with `version` under `[dataset]` in `dataset.toml`. Semantic versions are recommended, but any non-empty string is accepted. Registry snapshots also have publish-time tags: users resolve a tag with `-d <org>/<adapter_name>@<tag>`, and every publish receives the `latest` tag automatically. Individual tasks declare their package versions with `version` under `[task]` in `task.toml`.

| Tag                     | When to use                                                         |
| ----------------------- | ------------------------------------------------------------------- |
| `v1.0`                  | Default for the first release                                       |
| `v1.1`, `v2.0`, ...     | Subsequent releases; previous tags stay pinned to their snapshots   |
| `verified`, `lite`, ... | Mirror upstream naming when the original benchmark has named splits |
| `parity`                | Parity subset (generated via `--split parity`)                      |

To request a version, state the desired tag(s) in your adapter PR description. To cut a new version later (e.g., a bug fix), open a follow-up PR and request the new tag.

**Agent instruction:** set `[dataset].version` in `dataset.toml` and `[task].version` in each `task.toml`. In `task.toml`, top-level `schema_version` identifies the config format and is distinct from `[task].version`. Request any desired registry tags in the PR description.

**Step complete when:** Dataset is published to the registry, `harbor run -d <organization-name>/<adapter-name>` passes oracle tests, and the PR to `harbor-datasets` is merged.

***

## Step 9. Document and Submit

### README requirements

<Callout title="The README is parsed by automation" type="warn">
  Downstream tooling parses this README to generate parity summaries and registry metadata, so the template produced by `harbor adapter init` must be followed as written.

  Required:

  * Fill in every section the template defines.
  * Place any additional context — caveats, deviations, or commentary — in the **Notes** section, or in the `notes` field of `parity_experiment.json` or `adapter_metadata.json`.

  Not permitted:

  * Adding new top-level sections.
  * Renaming, reordering, or removing template sections.

  Deviations from the template prevent automated parsing; when unsure where to place content, use the **Notes** section.
</Callout>

The required content must appear in the following template-defined locations:

| Content                                                            | Location              |
| ------------------------------------------------------------------ | --------------------- |
| Benchmark bugs discovered and how they were handled                | Notes                 |
| Special treatments (prompt modifications, environment adjustments) | Notes                 |
| Deviations from the original benchmark and the rationale           | Notes                 |
| Agent implementation details (if custom agents were added)         | Agents section        |
| Known limitations                                                  | Notes                 |
| Reproduction scripts for parity experiments (both sides)           | Parity result section |

### Update the original benchmark fork's README

If you forked the original benchmark repository for parity (Scenario 2 or 3), update the fork's README to include reproduction scripts for running Harbor parity experiments. This makes it easy for others to reproduce results on the original benchmark side. Also include the reproduction steps in your adapter's README for completeness.

### `adapter_metadata.json` schema

Create `harbor/adapters/{adapter_name}/adapter_metadata.json`.

**Top-level fields:**

| Field                | Type       | Required | Description                                                       |
| -------------------- | ---------- | -------- | ----------------------------------------------------------------- |
| `adapter_name`       | `string`   | Yes      | Adapter name                                                      |
| `adapter_builders`   | `string[]` | Yes      | Builder names with email, e.g., `["Jane Doe (jane@example.com)"]` |
| `original_benchmark` | `object[]` | Yes      | Original benchmark split descriptors                              |
| `harbor_adapter`     | `object[]` | Yes      | Harbor adapter split descriptors                                  |

**`original_benchmark` entry fields:**

| Field              | Type       | Required | Description                                    |
| ------------------ | ---------- | -------- | ---------------------------------------------- |
| `split`            | `string`   | Yes      | Split name (use `"full"` if none)              |
| `size`             | `integer`  | Yes      | Number of tasks in Harbor context              |
| `harness`          | `string`   | Yes      | `"agent"`, `"llm"`, or `"None"`                |
| `supported_agents` | `string[]` | Yes      | Use `agent@version` format. `["None"]` if none |
| `adaptable`        | `boolean`  | Yes      | Whether this split can be converted            |
| `notes`            | `string`   | No       | Additional clarification. `"None"` if N/A      |

**`harbor_adapter` entry fields:**

| Field                      | Type       | Required | Description                                                 |
| -------------------------- | ---------- | -------- | ----------------------------------------------------------- |
| `split`                    | `string`   | Yes      | Corresponding split. `"full"` if collective                 |
| `adapted_benchmark_size`   | `integer`  | Yes      | Tasks convertible by adapter                                |
| `parity_benchmark_size`    | `integer`  | Yes      | Tasks used for parity                                       |
| `parity_sampling_rate`     | `number`   | Yes      | `parity_benchmark_size / adapted_benchmark_size`            |
| `registry_benchmark_size`  | `integer`  | Yes      | Exact task count in registry                                |
| `added_agents`             | `string[]` | Yes      | Custom agents added. `["None"]` if none                     |
| `parity_matching_agents`   | `string[]` | Yes      | Agents with comparable scores (`agent@version+model`)       |
| `parity_unmatching_agents` | `string[]` | Yes      | Agents without comparable scores. `["None"]` if all matched |
| `parity_costs`             | `string`   | Yes      | Total USD (e.g., `"$150"`)                                  |
| `notes`                    | `string`   | No       | `"None"` if N/A                                             |

If parity ran across three systems (Harbor ↔ Terminal-Bench ↔ Original), include a `"tb_adapter"` key with the same structure.

### Example

```json
[
  {
    "adapter_name": "my-benchmark",
    "adapter_builders": ["Jane Doe (jane@example.com)"],
    "original_benchmark": [
      {
        "split": "full",
        "size": 500,
        "harness": "agent",
        "supported_agents": ["codex@0.1.0"],
        "adaptable": true,
        "notes": "None"
      }
    ],
    "harbor_adapter": [
      {
        "split": "full",
        "adapted_benchmark_size": 500,
        "parity_benchmark_size": 100,
        "parity_sampling_rate": 0.2,
        "registry_benchmark_size": 500,
        "added_agents": ["None"],
        "parity_matching_agents": ["codex@0.1.0+gpt-5-2025-06-01"],
        "parity_unmatching_agents": ["None"],
        "parity_costs": "$150",
        "notes": "Parity on 100-task subset. Averaged over 3 trials."
      }
    ]
  }
]
```

### Submit

1. Change PR title from `[WIP] Adapter: {adapter_name}` to `[Ready for Review] Adapter: {adapter_name}`.
2. Request review from `@Slimshilin`.

**Step complete when:** PR title is `[Ready for Review] Adapter: {adapter_name}`, README covers all required sections, `adapter_metadata.json` passes schema validation, and review is requested from `@Slimshilin`.

***

## Reference: Terminal-Bench Migration

**Important:** The Harbor adapter must be isolated from the Terminal-Bench repo. Do not write a mechanical translation script. Write fresh adapter code following the Harbor process.

| Aspect         | Terminal-Bench                           | Harbor                                                                |
| -------------- | ---------------------------------------- | --------------------------------------------------------------------- |
| Config         | `task.yaml`                              | `task.toml`                                                           |
| Instruction    | In `task.yaml`                           | Separate `instruction.md`                                             |
| Dockerfile     | Root level                               | `environment/Dockerfile`                                              |
| Solution       | `solution.sh`                            | `solution/solve.sh`                                                   |
| Tests          | `run-tests.sh` + `tests/test_outputs.py` | `tests/test.sh`                                                       |
| Docker Compose | `docker-compose.yaml` in task root       | `environment/docker-compose.yaml` for GPU tasks; not needed otherwise |
| Verification   | Exit code (pytest)                       | Reward file: `/logs/verifier/reward.txt`                              |
| Output dir     | `tasks/`                                 | `datasets/`                                                           |
| Registry       | Dataset-level `dataset_path`             | Task-level via `dataset.toml` + `harbor init`                         |
| CLI            | `tb run --dataset`                       | `harbor run -d` / `-t` / `-p`                                         |
| Metrics        | Binary pass/fail                         | Float rewards, multiple metrics                                       |

**Important:** If Terminal-Bench used a tweaked metric, re-implement for the **original** benchmark metrics.

### Migration steps

1. Convert `task.yaml` to `task.toml` + `instruction.md`
2. Move files: `Dockerfile` → `environment/`, `solution.sh` → `solution/solve.sh`, `run-tests.sh` → `tests/test.sh`
3. For GPU tasks, move `docker-compose.yaml` into `environment/`; for non-GPU tasks, remove it
4. Update test scripts to write rewards to `/logs/verifier/reward.txt` (Harbor mounts `/logs/verifier` at runtime)
5. Update adapter code: change output dir from `tasks/` to `datasets/`, create subdirectories (`environment/`, `solution/`, `tests/`), split instruction into `instruction.md`, convert YAML generation to TOML
6. Use `harbor init` + `dataset.toml` for registry (replaces the old `registry.json`)

### Registry format conversion

**Before (Terminal-Bench registry.json):**

```json
{
    "name": "my-adapter",
    "version": "head",
    "description": "...",
    "github_url": "https://github.com/laude-institute/terminal-bench-datasets.git",
    "dataset_path": "datasets/my-adapter",
    "task_id_subset": null
}
```

**After (Harbor):**

```bash
harbor init  # select "dataset", creates dataset.toml
# Edit dataset.toml with descriptions, authors, credits
# Then submit to Harbor team for publishing
```

See [Step 8](#step-8-register-the-dataset) for the full publishing workflow.

### task.yaml → task.toml conversion example

**Before:**

```yaml
instruction: |
  Your task instruction here...
author_email: original-bench-author@email.com
author_name: Original Bench Author Name
difficulty: hard
category: programming
tags:
  - debugging
  - python
parser_name: swebench
max_agent_timeout_sec: 3000.0
max_test_timeout_sec: 3000.0
```

**After (task.toml):**

```toml
version = "1.0"

[task]
name = "my-benchmark/task-001"

[metadata]
author_email = "original-bench-author@email.com"
author_name = "Original Bench Author Name"
difficulty = "hard"
category = "programming"
tags = ["debugging", "python"]

[agent]
timeout_sec = 3000.0

[verifier]
timeout_sec = 3000.0
```

**After (instruction.md):**

```markdown
Your task instruction here...
```

### test.sh conversion example

**Before (Terminal-Bench):**

```bash
#!/bin/bash
pytest tests/ > test_results.txt
if [ $? -eq 0 ]; then echo "PASSED" > /tmp/test_marker.txt; else echo "FAILED" > /tmp/test_marker.txt; fi
```

**After (Harbor):**

```bash
#!/bin/bash
pytest /tests/test_*.py
if [ $? -eq 0 ]; then echo 1 > /logs/verifier/reward.txt; else echo 0 > /logs/verifier/reward.txt; fi
```

Key differences:

* Harbor mounts `/logs/verifier` for test outputs at runtime.
* Write numeric reward (can be float) to `/logs/verifier/reward.txt`.
* Can still use pytest, but final output must be the reward file.

***

## Resources

* [Harbor docs](/docs/getting-started): running tasks and jobs
* [Harbor repo](https://github.com/harbor-framework/harbor): examples and configs
* [Agent tutorial](/docs/agents): creating custom agents
* [Discord](https://discord.gg/QVvyhRw5UQ): `#adapters-spam` for questions


# Git Repository Datasets



import { Callout } from 'fumadocs-ui/components/callout';

Harbor can resolve datasets directly from Git repositories using the `--repo` flag. This lets you run benchmarks hosted in any GitHub, GitLab, or Hugging Face repo without publishing to the Harbor registry first.

## Quick start

```bash
harbor run --repo org/repo-name -d my-dataset -a claude-code -m anthropic/claude-sonnet-4
```

This clones the repository, finds `registry.json` (the dataset manifest), and resolves the named dataset.

## Specifying the repository

The `--repo` flag accepts several formats:

```bash
# GitHub shorthand (defaults to github.com)
--repo org/repo-name

# Pinned to a branch, tag, or commit
--repo org/repo-name@v1.0
--repo org/repo-name@main
--repo org/repo-name@abc1234

# Full URL
--repo https://github.com/org/repo-name

# Subdirectory via /tree/ path
--repo https://github.com/org/repo-name/tree/main/benchmarks

# Hugging Face
--repo https://huggingface.co/datasets/org/repo-name

# GitLab
--repo https://gitlab.com/org/repo-name
```

When no `@ref` is specified, the repository's default branch is used.

## How it works

1. Harbor parses the `--repo` value to extract the host, org, name, ref, and optional subdirectory.
2. It resolves the ref to an immutable Git SHA via `git ls-remote`.
3. The repository is sparse-checked-out into a local cache at `~/.harbor/cache/`.
4. Harbor reads the `registry.json` in the repo (or subdirectory) to discover available datasets.
5. The `--dataset` name is matched against the registry and tasks are resolved from the repo.

Cached checkouts are keyed by SHA, so pinned refs (`@v1.0`, `@abc1234`) are fetched once and reused. Branch refs re-resolve the SHA each run.

## Selecting a dataset

With `--repo`, the `--dataset` flag takes a bare name (no `org/` prefix):

```bash
# Run the "lite" dataset from the repo's registry.json
harbor run --repo org/my-benchmarks -d lite -a claude-code -m anthropic/claude-sonnet-4

# Run a specific version of the dataset
harbor run --repo org/my-benchmarks -d lite@1.2 -a claude-code -m anthropic/claude-sonnet-4
```

If the repo's `registry.json` contains only one dataset, `--dataset` can be omitted.

## Combining with other flags

Standard dataset flags work with `--repo`:

```bash
# Include specific tasks
harbor run --repo org/benchmarks -d suite \
  -i "task-a" -i "task-b" \
  -a claude-code -m anthropic/claude-sonnet-4

# Limit task count
harbor run --repo org/benchmarks -d suite \
  -l 10 \
  -a claude-code -m anthropic/claude-sonnet-4

# Custom registry.json path within the repo
harbor run --repo org/benchmarks \
  --registry-path benchmarks/registry.json \
  -d suite -a claude-code -m anthropic/claude-sonnet-4
```

<Callout type="warn" title="Mutual exclusivity">
  `--repo` cannot be combined with `--registry-url` or `--task` / `--task-git-url`.
</Callout>

## Repository layout

A git repository used with `--repo` should contain a `registry.json` at its root (or in the targeted subdirectory). This is the same format used by local `--registry-path`:

```
my-benchmarks/
├── registry.json
├── task-a/
│   ├── task.toml
│   ├── instruction.md
│   ├── environment/
│   └── tests/
├── task-b/
│   └── ...
└── ...
```

The `registry.json` maps dataset names to task lists. See the [Harbor task format](/docs/tasks) for task directory structure.

## Authentication

Public repositories work without any configuration. For private repositories, Harbor uses the Git credentials available in your environment (SSH keys, credential helpers, `GIT_ASKPASS`, etc.). If `git ls-remote` can reach the repo, `--repo` will work.


# Datasets



A [Harbor task](/docs/tasks) is an instruction, sandbox environment, and test script. A dataset is a collection of tasks for evals and training. Datasets sometimes define custom metrics that aggregate rewards across tasks.

Tasks can belong to multiple datasets. You can create datasets to be targeted eval or training groups. For example, you may want to grab 10 tasks from a few different benchmarks to create a composite benchmark.

There are three ways to use datasets:

1. **Local datasets**: run a local directory of tasks.
2. **Published datasets**: run a dataset from the [Harbor registry](https://hub.harborframework.com/).
3. **Git repository datasets**: run a dataset from any Git repository.

## Local datasets

Run a local dataset with `--path` or `-p`:

```bash
harbor run -p "<path/to/dataset>" -a "<agent>" -m "<model>"
```

## Published datasets

Datasets can be published and shared with members of your organization or publicly on the [Harbor registry](https://hub.harborframework.com/). If you publish your dataset privately, all members of your organization can run it. If you publish it publicly, anyone can run it.

Run a published dataset with `--dataset` or `-d`:

```bash
harbor run -d "my-org/my-dataset@1.0" -a "<agent>" -m "<model>"
```

To learn how to create and publish a dataset, see [Publishing a dataset](/docs/datasets/publishing).

## Git repository datasets

Run a dataset directly from any Git repository with `--repo`:

```bash
harbor run --repo org/repo-name -d "my-dataset" -a "<agent>" -m "<model>"
```

This resolves `registry.json` from the repo and runs the named dataset. Supports GitHub, GitLab, and Hugging Face URLs, with optional ref pinning (`@v1.0`, `@main`).

See [Git Repository Datasets](/docs/datasets/git-repos) for the full guide.

## Related docs

* [Git Repository Datasets](/docs/datasets/git-repos)
* [Publishing a dataset](/docs/datasets/publishing)
* [Custom Metrics](/docs/datasets/metrics)


# Metrics



Metrics define how to aggregate rewards across tasks in a job or dataset. By default, Harbor averages rewards across tasks and treats missing rewards as 0.

However, you may want to define custom logic or handle missing rewards differently. You can do this by creating a `metric.py`. If you are working with a `dataset.toml`, you can run the following command to create a `metric.py` file and automatically add it to the `[[files]]` section of your `dataset.toml`:

```bash
harbor init --dataset "<org>/<name>" --with-metric
```

See [Publishing a dataset](/docs/datasets/publishing) for more details.

If you are running a local dataset, it will automatically use the `metric.py`.

## Custom metrics with `metric.py`

A `metric.py` script must accept the following arguments:

* `-i, --input-path`: rewards JSONL file
* `-o, --output-path`: output JSON file with computed metrics

Example:

```python title="metric.py"
# /// script
# dependencies = []
# ///
# To add a dependency: uv add --script metric.py <dependency>

import argparse
import json
from pathlib import Path


def main(input_path: Path, output_path: Path):
    rewards = []

    for line in input_path.read_text().splitlines():
        reward = json.loads(line)
        if reward is None:
            rewards.append(0)
        elif len(reward) != 1:
            raise ValueError(
                f"Expected exactly one key in reward dictionary, got {len(reward)}"
            )
        else:
            rewards.extend(reward.values())

    mean = sum(rewards) / len(rewards) if rewards else 0

    output_path.write_text(json.dumps({"mean": mean}))


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-i",
        "--input-path",
        type=Path,
        required=True,
        help="Path to a jsonl file containing rewards, one json object per line.",
    )
    parser.add_argument(
        "-o",
        "--output-path",
        type=Path,
        required=True,
        help="Path to a json file where the metric will be written as a json object.",
    )
    args = parser.parse_args()
    main(args.input_path, args.output_path)
```

When you publish your dataset, it automatically publishes the `metric.py` file with it:

```bash
harbor publish "<path/to/dataset>" --public
```

`metric.py` output should be a JSON object. Multiple metric values are allowed:

```json
{"mean": 0.81, "pass_rate": 0.5}
```


# Publishing a dataset



<Callout type="info" title="Agent skill available">
  Want your coding agent to walk you through publishing? Install the `publish` skill:

  ```bash
  npx skills add harbor-framework/harbor --skill publish
  ```
</Callout>

If you are new to Harbor, consider reading about [the composition of a Harbor dataset](/docs/datasets) first.

Benchmarks and datasets developed using Harbor can be easily shared with others inside and outside of your organization using the [Harbor registry](https://hub.harborframework.com/).

You can publish your dataset publicly or privately.

* **Public datasets** are visible and usable by everyone. This is a great way to increase the adoption of your dataset or benchmark.
* **Private datasets** are visible only to members of the publishing org.

You can run a published dataset with `--dataset` or `-d`:

```bash
harbor run -d "<org>/<dataset>" -a "<agent>" -m "<model>"
```

This guide will walk you through the process of publishing a dataset to the Harbor registry.

## Prerequisites

### Login to the Harbor registry

Before publishing, run:

```bash
harbor auth login
```

This opens a GitHub sign-in flow and creates your Harbor account. New accounts get an org using your GitHub username by default. You can publish to orgs you are an owner of. If you publish to an org that does not exist, it will be created for you.

You must be signed in to publish.

You can verify you are signed in by running:

```bash
harbor auth status
```

### Update old tasks

Task identifiers are specified in the `[task]` section of the `task.toml` file as `<org>/<name>`. This enables you to create globally unique identifiers that can be published to the registry.

If your tasks are missing the `[task]` section (any task created before Mar 30, 2026), you'll need to add it before publishing them or including them in a dataset. You can do this by running:

```bash
harbor task update "<path/to/task>" --org "<org>"
```

This will add the `[task]` section to the task with the name `<org>/<folder-name>`. For org name, we recommend using your company or benchmark name.

You can update an entire directory of tasks by including the `--scan` flag:

```bash
harbor task update "<path/to/tasks>" --org "<org>" --scan
```

## 1) Initialize a dataset manifest

A dataset is a collection of versioned tasks defined in a `dataset.toml` manifest.

Task pointers look like this:

```toml
[[tasks]]
name = "<org>/<name>"
digest = "sha256:<hash>"
```

The digest is the SHA256 hash of the task archive.

To initialize the dataset manifest run:

```bash
harbor dataset init "<org>/<dataset>" \
  --description "A short description" \
  --author "Your Name <your@email.com>"
```

Datasets can optionally include a metric script to define how rewards are aggregated across tasks.

If you need a custom metric script include the `--with-metric` flag:

```bash
harbor dataset init "<org>/<dataset>" --with-metric
```

This creates:

```
<dataset>/
├── dataset.toml
└── metric.py            # only if --with-metric
```

{/* The `dataset.toml` will look something like this:

  ```toml
  # Dataset manifest for <org>/<dataset>
  # Add tasks using: harbor add <org>/<task>
  # Publish using: harbor publish

  [dataset]
  name = "<org>/<dataset>"
  version = "1.0.0"
  description = "<some description>"
  authors = [{ name = "Your Name", email = "your@email.com" }]
  keywords = ["<some keyword>", "<some other keyword>"]

  tasks = []

  [[files]]
  path = "metric.py"
  digest = "sha256:<hash>"
  ``` */}

If you initialize a dataset in a directory with pre-existing tasks, the tasks will be auto-added to the dataset manifest. If you initialize a task in the same directory as `dataset.toml`, the task will be auto-added to the dataset manifest.

For example:

```bash
my-dataset/
├── dataset.toml
├── task-a/
├── task-b/
└── metric.py            # only if --with-metric
```

will create a `dataset.toml` with the tasks `task-a` and `task-b` auto-added.

## 2) Add tasks and files

Use `harbor add` to explicitly add local or published tasks.

```bash
cd "<path/to/dataset>"
```

### Add local tasks

```bash
harbor add "<path/to/task-a>" "<path/to/task-b>"
```

You can also add tasks from a local `dataset.toml` file by running:

```bash
harbor add "<path/to/dataset.toml>"
```

### Add registered tasks or datasets

```bash
harbor add org/name-of-task
```

By default, `harbor add` will add the latest version of a task. You can also add a specific version using `"<org>/<name>@<tag>"`, `"<org>/<name>@<revision>"`, or `"<org>/<name>@sha256:<hash>"`.

You can also add all of the tasks from a published dataset by running:

```bash
harbor add org/name-of-dataset
```

### Add all tasks from a local folder

Include the `--scan` flag to add all tasks from a local folder.

```bash
harbor add "<path/to/candidate-tasks>" --scan
```

### Optional metric file

If `dataset.toml` does not include `metric.py`:

```bash
harbor add metric.py
```

`metric.py` must be added by filename and must be in the same directory as `dataset.toml`.

### Removing tasks

You can remove tasks from a dataset by running:

```bash
harbor remove "<org>/<name-of-task>"
```

`harbor remove` accepts the same arguments as `harbor add`.

## 3) Sync task and file digests (when needed)

A `dataset.toml` references tasks and metrics by their digest. When you are developing a dataset, tasks and metrics are subject to change.

When you publish a dataset, it automatically refreshes the digests of the tasks and metrics in the same directory as `dataset.toml`.

If you need to refresh before publishing, you can use:

```bash
harbor sync
```

To also upgrade remote tasks to their latest published version, run:

```bash
harbor sync --upgrade
```

`harbor add` updates tasks if they are already present and can be used to refresh the digest for local tasks stored outside of the dataset folder.

## 4) Publish the dataset

Use `harbor publish` to publish a dataset:

```bash
harbor publish "<path/to/dataset>"
```

By default, `harbor publish` will also publish any tasks in the dataset directory.

### Publish options

* `-t / --tag`: add one or more tags (repeatable). `latest` is always included.
* `-c / --concurrency`: control upload concurrency.
* `--no-tasks`: don't publish the tasks in the dataset directory.
* `--public`: make the dataset public. Private is the default.

By default, datasets are published with the `latest` tag.

`harbor publish` refreshes the digests of local tasks in the dataset directory during upload. Use `harbor sync` when you need to refresh remote tasks before publishing. Re-add local tasks outside of the dataset directory to refresh their digests.

### Tagging and visibility

```bash
harbor publish "<path/to/dataset>" -t v1.0 --public
```

```bash
harbor publish "<path/to/dataset>" --no-tasks
```

## 5) Run your published dataset

After publishing, evaluate with:

```bash
harbor run -d "my-org/my-dataset@v1.0" -a "<agent>" -m "<model>"
```

## Sharing

If you published it publicly, anyone can run it. If you published it privately, only members of the publishing org can run it. You can toggle visibility at any time using `harbor dataset visibility` or through the UI on [the registry website](https://hub.harborframework.com/).

Publishing visibility and access is documented in [Sharing](/docs/sharing/sharing).

## Related docs

* [Datasets](/docs/datasets)
* [Custom Metrics](/docs/datasets/metrics)


# Sandboxes



import { Callout } from 'fumadocs-ui/components/callout';

Containerized agentic tasks can be slow when performing rollouts. This is due to container startup and teardown overhead, waiting for LLM API calls, and waiting for command execution. Horizontal scaling becomes the only viable way to accelerate experimentation, so we recommend using a cloud sandbox provider like Daytona.

Using a cloud sandbox provider shifts command execution to the cloud, making trials I/O bounded rather than compute bounded. This means you can typically parallelize far above your CPU count.

## Using a cloud sandbox provider

There are many cloud sandbox providers to choose from. Good options are [Daytona](https://www.daytona.io/), [Modal](https://modal.com/), [E2B](https://e2b.dev/), [Runloop](https://runloop.ai/), [Tensorlake](https://docs.tensorlake.ai/sandboxes/harbor), [Islo](https://islo.dev/rl), [CoreWeave Sandboxes](https://www.coreweave.com/products/coreweave-sandboxes), [W\&B Sandboxes](https://docs.wandb.ai/sandboxes), [LangSmith](https://docs.langchain.com/langsmith/harbor-integrations), [Blaxel](https://blaxel.ai/), [Novita Sandbox](https://novita.ai/), [Amazon EC2](https://aws.amazon.com/ec2/), [SkyPilot](https://docs.skypilot.co/en/latest/sandboxes.html) (limited early access; not generally available), [OpenSandbox](https://github.com/opensandbox-group/OpenSandbox), [Beam](https://beam.cloud/), [HF Sandbox](https://huggingface.co/docs/huggingface_hub/main/guides/sandbox) (Hugging Face Jobs), [Hyperbrowser](https://www.hyperbrowser.ai/), and [Vercel Sandbox](https://vercel.com/docs/sandbox).

```bash
harbor run -d "<org/name>" \
  -m "<model>" \
  -a "<agent>" \
  -e daytona \
  -n "<n-parallel-trials>"
```

We run up to 100 trials in parallel on a MacBook Pro with 14 cores.

<Callout title="Removing internet restrictions on Daytona">
  By default, Daytona accounts have internet access restrictions that can prevent many benchmarks from running correctly. Use the coupon code **HARBOR\_NETWORK** on your Daytona account to remove these restrictions.
</Callout>

## Multi-container deployments

Daytona, EC2, Islo, LangSmith, Blaxel, Novita Sandbox, Beam, Hyperbrowser, Vercel Sandbox, and Tensorlake support multi-container deployments. To use multi-container tasks, include an `environment/docker-compose.yaml` file in your task definition.

Other cloud sandbox providers (Modal, E2B, Runloop, CoreWeave Sandboxes, W\&B Sandboxes, SkyPilot, OpenSandbox, and HF Sandbox) do not currently support multi-container environments. For those providers, you will need to use single-container tasks or switch to Daytona, EC2, Islo, LangSmith, Blaxel, Novita Sandbox, Beam, Hyperbrowser, Vercel Sandbox, Tensorlake, or the local Docker environment.


# Trial Handoff



`harbor trial handoff` is the opposite side of [loading trajectories](/docs/run-jobs/load-trajectory): instead of seeding a past session into a container, it takes the session out of a finished trial so you can continue the conversation on your own machine, and ask the agent what it did, why it took an approach, or what it would try next.

```bash
# a local trial directory
harbor trial handoff jobs/<job>/<trial>

# or a trial ID, downloaded from the Harbor platform first
harbor trial handoff 594025f3-7d65-4655-8576-4bee95002eae
```

Harbor copies the trial's native session into the local agent CLI's session store for your current directory, then replaces itself with the resumed session. For `claude-code` that means you land directly in `claude --resume <session-id>`, talking to the agent with its full trial conversation as context.

## Requirements

* The trial was run by an agent with handoff support (`capabilities.handoff`); today that is `claude-code`.
* The agent CLI is installed locally (`claude` on PATH). If you are not logged in, the CLI's own login flow takes over on launch.
* The trial has exactly one session; multi-session trials (e.g. multi-step tasks with fresh per-step sessions) are rejected.

## What is restored

Only the conversation, the same rule as [loading](/docs/run-jobs/load-trajectory): the agent remembers everything it said and did, but files it created in the container do not exist on your machine, so it can discuss its work, not re-open it.


# Run Jobs



Use this section to run datasets, scale across cloud sandboxes, and inspect results and artifacts.

* [Run Evals](/docs/run-jobs/run-evals)
* [Skills](/docs/run-jobs/skills)
* [Results and Artifacts](/docs/run-jobs/results-and-artifacts)
* [Regrade](/docs/run-jobs/regrade)
* [Cloud Sandboxes](/docs/run-jobs/cloud-sandboxes)


# Loading Trajectories



import { Callout } from 'fumadocs-ui/components/callout';

Loading a trajectory seeds the agent's session from a previous run, so the agent resumes that conversation instead of starting fresh. Today `claude-code` and `codex` support both formats below; other agents fail fast before any environment starts.

## Native trajectories (often `.jsonl`)

A native trajectory is the agent's own session format. The resume is lossless, but the file can only be loaded by the same agent that recorded it. After a trial, the native file is stored in the trial's `agent/sessions/` directory:

| Agent         | Native trajectory file                            |
| ------------- | ------------------------------------------------- |
| `claude-code` | `agent/sessions/projects/-app/<session-id>.jsonl` |
| `codex`       | `agent/sessions/<YYYY>/<MM>/<DD>/rollout-*.jsonl` |

You can copy the file anywhere, but **keep its filename**: agents locate and validate sessions by name.

Native loading is run-level only: a task is agent-agnostic, while a native trajectory ties it to the one agent that recorded it.

```bash
harbor run -p path/to/task -a claude-code -m anthropic/claude-opus-5 \
  --load-trajectory path/to/<session-id>.jsonl
```

Example task: [examples/tasks/hello-load-native-trajectory/](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/hello-load-native-trajectory), loading [this claude-code session file](https://github.com/harbor-framework/harbor/blob/main/examples/tasks/hello-load-native-trajectory/environment/d7d4e19e-608d-44ef-b166-cd050ef274ba.jsonl).

## ATIF trajectories (always `.json`)

An [ATIF](/docs/agents/trajectory-format) trajectory (the `agent/trajectory.json` Harbor writes after every trial) is portable: Harbor converts it into the loading agent's native session format, so a trajectory recorded by one agent can seed another. The conversion keeps the conversation (messages, tool calls, tool results) but drops agent-specific session details, so it is not bit-for-bit lossless like a native load.

### Run-level

Pass the file with `--load-trajectory`, so the same task can be evaluated with and without prior context:

```bash
harbor run -p path/to/task -a codex -m openai/gpt-5.6-sol \
  --load-trajectory jobs/<job>/<trial>/agent/trajectory.json
```

Example task: [examples/tasks/hello-load-atif-trajectory-run-level/](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/hello-load-atif-trajectory-run-level), loading [this ATIF trajectory](https://github.com/harbor-framework/harbor/blob/main/examples/tasks/hello-load-atif-trajectory-run-level/environment/trajectory.json).

### Task-level

A task whose premise is a prior conversation ("continue from here") can ship the trajectory itself, so no flag is needed at run time: put the ATIF file beside the task's `instruction.md`, named `trajectory.json`. It is picked up by location, so nothing in `task.toml` refers to it.

```text title="Task directory with prior context"
my-task/
├── instruction.md
├── trajectory.json
├── task.toml
├── environment/
│   └── Dockerfile
└── tests/
    └── test.sh
```

```bash
harbor run -p path/to/task -a codex -m openai/gpt-5.6-sol
```

Task-level loading is ATIF-only, since a task is agent-agnostic. A run-level `--load-trajectory` overrides the task's `trajectory.json`. For [multi-step tasks](/docs/tasks/multi-step), the file belongs to the first step, at `steps/<first-step>/trajectory.json`.

Example task: [examples/tasks/hello-load-atif-trajectory-task-level/](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/hello-load-atif-trajectory-task-level), shipping [this ATIF trajectory](https://github.com/harbor-framework/harbor/blob/main/examples/tasks/hello-load-atif-trajectory-task-level/trajectory.json) beside its `instruction.md`.

## What is restored

Only the conversation. The agent sees the full prior message history as context, but files the original run created do not exist in the new environment.

For [multi-step tasks](/docs/tasks/multi-step), the trajectory is loaded before the first step only; combined with `--resume-trajectory`, the per-step sessions are (load, resume, resume, ...) instead of (load, fresh, fresh, ...).

## Harbor handoff

<Callout type="info" title="Coming soon">
  Handoff is the opposite side of `--load-trajectory`: instead of loading a past session into a container, you resume a finished Harbor run's session in the agent CLI on your own machine and ask the agent about what it did.
</Callout>


# Regrade



Treat regrade as a predictive feature: a regrade says, given this `lock.json` and this source trial, this is what we think the trial result would be in the new `<trial-name>/` directory. The recorded agent execution is held fixed; only the verdict is recomputed.

Regrading re-scores completed trials with a new (or updated) verifier. The agent phase is never re-run: Harbor seeds a fresh trial with the recorded agent logs and artifacts from the source trial, then runs the new verifier against them in a separate verifier environment. This makes it cheap to fix a broken grader, tighten test cases, or compare verifier variants across the same agent executions.

Source trials and jobs are never modified. Every regrade produces a new trial or job directory with its own results.

## Quick start

Regrade a job (can contain 1 trial as well):

```bash
harbor job regrade jobs/2026-07-16__15-44-21 -p ./my-task-v2 -e modal
```

Regrade a trial:

```bash
harbor trial regrade jobs/2026-07-16__15-44-21/my-task__abc1234 -p ./my-task-v2 -e modal
```

Both commands also accept a Harbor hub UUID instead of a local directory; the source is downloaded first (cached under `trials/.sources/` or `jobs/.sources/`) and then regraded the same way:

```bash
harbor trial regrade 8f3c2a9e-1b4d-4c6f-9e2a-7d5b8c1f0a3e -p ./my-task-v2
harbor job regrade deeac1d1-9588-422f-8d8a-d504ec9fae14 -p ./my-task-v2
```

`-p` points at the task providing the verifier to regrade with, typically a copy of the original task with an edited `tests/` directory. It accepts either a task directory or a directory of task directories (for example a downloaded dataset); tasks are matched to trials by task name.

At the end of a job regrade, Harbor prints a delta summary against the source rewards:

```
Regrade delta over 1 trial(s): 1 changed (0 up, 1 down), mean reward 1.000 → 0.500
```

## How it works

For each source trial, Harbor:

1. Creates a new trial directory and copies the source trial's `agent/` and `verifier` inputs (`artifacts/`) into it.
2. Builds the new task's verifier environment (from `tests/`) and uploads the recorded artifacts to their original container paths.
3. Runs the verifier and writes a fresh `verifier/` directory and `result.json`.

No agent environment is started and the recorded agent is never re-instantiated, so regrading needs no agent credentials or API keys. A regrade is best thought of as a fork of the source trial: the new trial's `config.json` and `lock.json` carry the source trial's original agent configuration (and skill locks, copied verbatim from the source lock rather than re-resolved), so downstream consumers such as leaderboards keep seeing the trial's real inputs. The source trial's agent identity and token/cost stats are likewise preserved on the new result so results tables group correctly; the recorded cost is reported, not re-incurred (avoid summing cost across a job and its regrades).

## Requirements

A trial is regradable when the new verifier's declared inputs are present in the record:

* The source is a completed single-step trial with a readable `result.json` and `artifacts/manifest.json`.
* The new task resolves to a separate-mode verifier (`[verifier] environment_mode = "separate"`, or a `[verifier.environment]` table). Shared-mode verifiers inspect the live agent environment, which no longer exists.
* Every artifact the new task declares (including the implicit `/logs/artifacts` convention directory) has a manifest entry in the source trial, and its collected bytes still exist on disk at the path the replay reads.

**Whether the source trial originally ran a shared or separate verifier does not matter; only the recorded artifacts do.** Separate verifier mode is the recommended way to author tasks and will soon become the default. Trials that fail these checks are refused with a specific reason (recorded as per-trial errors in a job regrade; other trials proceed). Entries recorded as `failed` or `skipped` (a host-path collision) make the declaration incompatible, since the record does not contain trustworthy bytes for them.

Multi-step tasks and trials are not supported yet.

## Options

Both commands share the core flags:

| Flag                             | Meaning                                                                                                                    |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `-p, --task-path`                | Task directory (or directory of task directories) providing the verifier. Repeatable on `job regrade`.                     |
| `-t, --task`                     | `job regrade` only: registry task providing a verifier (`org/name[@ref]`), as in `harbor run`. Repeatable.                 |
| `-d, --dataset`                  | `job regrade` only: dataset providing verifier tasks (`name@version` or `org/name[@ref]`), as in `harbor run`. Repeatable. |
| `-e, --env`                      | Environment provider for the verifier environment (default `docker`), independent of what the original job ran on.         |
| `--ve, --verifier-env`           | `KEY=VALUE` environment variable for the verifier process. Repeatable.                                                     |
| `--verifier`, `--verifier-kwarg` | Custom verifier import path and its kwargs, replacing the task's test-script verifier.                                     |
| `-o`                             | Output parent directory (`--jobs-dir` / `--trials-dir`).                                                                   |

`harbor job regrade` also accepts `-n/--n-concurrent` and `--job-name`; `harbor trial regrade` accepts `--trial-name`. New trials get independently generated names; the link back to the source lives in provenance, not the name.

## Provenance

Each regraded trial records where it came from:

* Trial level: `config.json` records a `source_trial` block, `{action, type, trial_id, path}`. `action` names the derivation (`regrade` today; other derivations may exist later) and `type` is `local` or `hub`: a hub source records only the trial UUID, a local source records the trial directory path plus its UUID when known. `lock.json` records the same block plus one extra field, `task`: the source trial's own task lock (name, type, and content digest) copied verbatim from the source `lock.json`; resolution also fills in the UUID for local sources that did not record it. The lock's `verifier` block records the resolved mode as `verifier.environment_mode`, and the result records it as `verifier_environment_mode`; source identity lives in the config and lock, not in `result.json`.
* Job level: the regrade job's `config.json` records `source_jobs`, a list of blocks with the same shape, `{action, type, job_id, path}` (the CLI passes one source; multiple source jobs are supported programmatically). Nothing extra is recorded in the job-level `lock.json`: the config pins the source identity and each derived trial's lock carries its own `source_trial`.
* A hub job regrade types every derived trial `hub`: both `config.json` and `lock.json` record only `{action: "regrade", type: "hub", trial_id}` per trial, and the job lock's trial entries match each trial's own `lock.json` exactly. The job archive is downloaded once and its trials are seeded into a scratch per-trial cache (`<job_dir>/.sources/`), which is removed when the regrade completes; an interrupted job keeps it so resume does not re-download, and a missing seed falls back to a per-trial hub download.
* Lock equality (used by `harbor job resume` to decide whether a rebuilt job is the same experiment) is anchored on the source trial's UUID and task digest, not the path: moving a source directory does not break resume, but swapping in a different trial at the same path is caught.

An interrupted job regrade resumes like any other job with `harbor job resume`.


# Artifact Collection



Harbor can automatically collect files from the sandbox environment after each trial completes. This is useful for preserving model outputs, logs, generated files, evidence held by sidecar services, or any other byproducts of the agent's work.

## Convention directory (zero configuration)

Any files written to `/logs/artifacts/` inside the sandbox are collected automatically with no configuration needed. For Docker environments, this directory is volume-mounted directly to the host. For remote environments (Daytona, Modal, E2B, Tensorlake, Blaxel, Novita Sandbox, Beam, Hyperbrowser, etc.), files are downloaded after the trial finishes.

For example, if your task's test script or agent writes files to `/logs/artifacts/`:

```bash
# Inside the sandbox
echo "result" > /logs/artifacts/output.txt
cp model.pt /logs/artifacts/model.pt
```

These files will appear in the trial output directory at `<trial_dir>/artifacts/logs/artifacts/`.

## Config-driven artifact collection

To collect files from arbitrary paths in the sandbox (not just `/logs/artifacts/`), add an `artifacts` field to your job configuration or task.toml.

### Simple form

List the paths you want to collect. Each path is mirrored under the trial's `artifacts/` base directory (the service is not part of the host path).

```yaml title="job.yaml"
artifacts:
  - /app/hello.txt
  - /workspace/output.csv
  - /data/results
```

This saves the files to `<trial_dir>/artifacts/app/hello.txt`, `.../artifacts/workspace/output.csv`, and `.../artifacts/data/results/`.

### Object form

Use the object form to control where files are saved within the `artifacts/` directory, or to collect files from a Docker Compose sidecar service.

```yaml title="job.yaml"
artifacts:
  # Place at an explicit (relative) destination on the host
  - source: /app/hello.txt
    destination: workspace/hello.txt
  # Collect from a compose sidecar service instead of the main container
  - source: /var/log/api/requests.log
    service: api
```

| Field         | Meaning                                                                                                                                                                                                |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `source`      | Absolute path inside the container to collect. Also where the file re-materializes inside a separate verifier environment ("no translation").                                                          |
| `destination` | Optional **host-side** relative path under `<trial_dir>/artifacts/`. Never affects verifier-side placement. Must be relative, must not contain `..`, and must not shadow the reserved `manifest.json`. |
| `service`     | Optional Docker Compose service to collect from. Defaults to `main` (the agent's container). Requires a compose-capable provider.                                                                      |

## Collecting evidence from sidecar services

Multi-container tasks often hold their score signal inside a sidecar — a database the agent wrote to, a server that logged the agent's requests. Sidecar artifacts let the verifier read that evidence even in `separate` verifier mode, where all containers are torn down before verification.

```toml title="task.toml"
artifacts = [
    { source = "/var/log/api/requests.log", service = "api" },
    { source = "/tmp/db-dump.sql", service = "postgres" },
]

# Snapshot runtime state into files before teardown
[[verifier.collect]]
service = "postgres"
command = "pg_dump -U postgres app > /tmp/db-dump.sql"
timeout_sec = 60.0
```

Because sidecar evidence is pulled directly from the sidecar's filesystem — over a channel the agent's container cannot write to — it is tamper-resistant: in separate verifier mode, Harbor stops the main container *before* collecting sidecar evidence, so leftover agent processes cannot interfere.

See the [task documentation](/docs/tasks#sidecar-artifacts-and-collect-hooks) for the full sidecar workflow, and [`examples/tasks/sidecar-artifacts`](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/sidecar-artifacts) for a working example.

## How collection works

Artifact collection runs after the agent finishes. It is **best-effort** -- failures to collect an artifact will never cause the trial to fail (the failure is recorded in the manifest instead).

The collection process:

1. **Main collect hooks**: `[[verifier.collect]]` entries targeting `main` run while the agent container is still up.
2. **Main artifacts**: The convention directory and main-targeted config paths are collected.
3. **Main stop** (separate verifier mode only): the main service is stopped so agent processes cannot interfere with sidecar evidence.
4. **Sidecar collect hooks**: `[[verifier.collect]]` entries targeting sidecars run.
5. **Sidecar artifacts**: Sidecar-targeted paths are pulled from each service's filesystem.
6. **Manifest**: A `manifest.json` file is written to the artifacts directory listing what was collected, from which service, and whether collection succeeded.

## Output structure

After collection, the trial directory contains:

```
<trial_dir>/
├── artifacts/                         # One flat base dir shared by all services
│   ├── manifest.json                  # Collection manifest
│   ├── logs/artifacts/                # The convention directory (from main)
│   │   └── output.txt
│   ├── app/hello.txt                  # Config-driven artifact (source-mirrored)
│   ├── var/log/api/requests.log       # Sidecar artifact (from the `api` service)
│   └── workspace/hello.txt            # Config-driven artifact with a destination
├── agent/
├── verifier/
├── config.json
└── result.json
```

All services' artifacts share this one base dir, keyed only by their source
path. If two services export the same path they collide on the host; collection
keeps the first claimant and logs a warning (it never overwrites), recording the
skipped entry with `status: "skipped"` in the manifest.

The manifest tracks each artifact's source, destination, originating service, type (file or directory), and whether collection succeeded:

```json title="manifest.json"
[
  {
    "source": "/logs/artifacts",
    "destination": "artifacts/logs/artifacts",
    "type": "directory",
    "status": "ok",
    "service": null
  },
  {
    "source": "/var/log/api/requests.log",
    "destination": "artifacts/var/log/api/requests.log",
    "type": "file",
    "status": "ok",
    "service": "api"
  }
]
```

## Viewing artifacts

Artifacts are viewable in the Harbor results viewer. Run `harbor view` and navigate to a trial to see collected artifacts under the Artifacts tab.

## Environment support

Artifact collection works across all environment types. Sidecar artifacts and collect hooks additionally require a Docker Compose-capable provider:

| Environment  | Convention directory                | Config-driven paths    | Sidecar artifacts & collect hooks |
| ------------ | ----------------------------------- | ---------------------- | --------------------------------- |
| Docker       | Volume-mounted (no download needed) | Downloaded after trial | Supported                         |
| Daytona      | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |
| Modal        | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |
| EC2          | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |
| E2B          | Downloaded after trial              | Downloaded after trial | Not supported (no compose)        |
| Tensorlake   | Downloaded after trial              | Downloaded after trial | Not supported (no compose)        |
| Blaxel       | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |
| Novita       | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |
| Beam         | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |
| Hyperbrowser | Downloaded after trial              | Downloaded after trial | Supported (compose tasks)         |

Tasks that declare sidecar artifacts or collect hooks on a provider without compose support fail at trial start with a clear error.


# Evals



## What is a dataset?

In Harbor, a dataset is a collection of [Harbor tasks](/docs/tasks). A task includes an instruction, environment, and test script. Use datasets to evaluate, train, or tune prompts.

## Viewing registered benchmarks

Harbor resolves published datasets from its package registry.
`harbor run -d "<org>/<dataset>"` uses the configured registry.

Use `--registry-path` or `--registry-url` only for legacy `registry.json` setups.

To list available datasets:

```bash
harbor dataset list
```

## Running a benchmark

> **Windows tasks.** Datasets containing tasks with `[environment].os = "windows"` require a Windows host with Docker Desktop in Windows container mode. If the daemon is in the wrong mode (or the task's image doesn't match), Harbor fails fast at start with a clear error. See [Windows tasks](/docs/tasks/windows-container-support).

Terminal-Bench:

```bash
harbor run -d terminal-bench/terminal-bench-2 -m "<model>" -a "<agent>"
```

Harbor resolves package metadata and downloads task artifacts as needed.

By default, omitted task resources use the provider's default sizing. When a task sets `cpus` or `memory_mb`, `--cpus` and `--memory` control how Harbor applies those values. See [Managing Resources](/docs/tasks/managing-resources) for enforcement policies, provider support, and override flags.

SWE-Bench Verified:

```bash
harbor run -d swe-bench/swe-bench-verified -m "<model>" -a "<agent>"
```

Omit the tag to use the latest dataset.

For dataset creation, sync, and publishing workflow:

* [Publishing a dataset](/docs/datasets/publishing)
* [Custom Metrics](/docs/datasets/metrics)

## Running a local dataset

Run a local dataset with:

```bash
harbor run -p "<path/to/dataset>" -m "<model>" -a "<agent>"
```

## Analyzing results

Running the `harbor run` command creates a [job](/docs/core-concepts#job) which by default is stored in the `jobs` directory.

The file structure looks something like this:

```bash
jobs/job-name
├── config.json               # Job config
├── result.json               # Job result
├── trial-name
│   ├── config.json           # Trial config
│   ├── result.json           # Trial result
│   ├── agent                 # Agent directory, contents depend on agent implementation
│   │   ├── recording.cast
│   │   └── trajectory.json
│   └── verifier              # Verifier directory, contents depend on test.sh implementation
│       ├── ctrf.json
│       ├── reward.txt
│       ├── test-stderr.txt
│       └── test-stdout.txt
└── ...                       # More trials
```

### Using the viewer

Harbor includes a web-based results viewer for browsing jobs, inspecting trials, and analyzing agent trajectories. To launch it, point it at your jobs directory:

```bash
harbor view jobs
```

This starts a local web server (default `http://127.0.0.1:8080`) where you can:

* **Browse jobs** — Filter and search by agent, model, dataset, and date range.
* **Inspect trials** — View trial results, rewards, durations, and errors for each task.
* **View trajectories** — Step through the agent's execution including tool calls, observations, and multimodal content (text and images).
* **Analyze performance** — See token usage breakdowns, timing metrics (environment setup, agent execution, verification), and verifier output.
* **Compare jobs** — Select multiple jobs to view a side-by-side comparison matrix of task performance across agent/model combinations.
* **View artifacts** — Browse files collected from the sandbox after each trial (see [Artifact Collection](/docs/run-jobs/results-and-artifacts)).
* **Generate summaries** — Use AI-powered summarization to analyze job failures.

The viewer supports keyboard navigation (`j`/`k` to move between rows, `Enter` to open, `Esc` to deselect).

| Option         | Description                                                            |
| -------------- | ---------------------------------------------------------------------- |
| `--port`, `-p` | Port or port range (e.g., `8080` or `8080-8089`). Default: `8080-8089` |
| `--host`       | Host to bind the server to. Default: `127.0.0.1`                       |
| `--dev`        | Run frontend in development mode with hot reloading                    |


# Simulated User





A simulated-user trial runs two agents in one environment. The **user agent** (`--user-agent`) holds the task privately and plays a human user. The **target agent** (`--agent`) is the coding agent under test; it starts with no instruction and learns everything through conversation. The verifier runs as usual.

<img alt="How a simulated-user trial works over ACP" src={__img0} placeholder="blur" />

## Bridge

A bridge is the protocol connecting the two agents. The only bridge today is `acp`: Harbor installs `acpx` in the environment, registers the target as an ACP server, and the user agent talks to it with `acpx prompt "<message>"`. Supported targets today are `claude-code` and `gemini-cli`; any agent can be the user agent. Any agent that speaks ACP (see [ACP Registry Agents](https://agentclientprotocol.com/get-started/registry)) could be a target, but the remaining ones are not wired up yet.

## Prompt

The user agent's prompt has three parts, each with one owner:

| Part                | Owner                 | Default           | Override               |
| ------------------- | --------------------- | ----------------- | ---------------------- |
| persona             | you, or the benchmark | built-in          | `--user-persona-path`  |
| bridge instructions | Harbor                | how to use `acpx` | `--bridge-prompt-path` |
| task instruction    | the task              | `instruction.md`  | none                   |

They are joined in that order. `--user-prompt-template-path` replaces the layout with a Jinja2 template using `{{ persona }}`, `{{ bridge_instructions }}` and `{{ instruction }}`.

## Examples

Hello world:

```bash
harbor run -t hello-world/hello-world \
  --agent claude-code --model anthropic/claude-sonnet-5 \
  --user-agent claude-code --user-model anthropic/claude-sonnet-5 \
  --bridge acp
```

A [SWE-Interact](https://github.com/scaleapi/SWE-Interact) task with its persona (use the `singleturn` task; the persona is the part of `persona.md` above the task block):

```bash
harbor run -p SWE-Interact/data/singleturn/deepswe_tomlkit-toml-table-converters \
  --agent claude-code --model anthropic/claude-sonnet-5 \
  --user-agent claude-code --user-model anthropic/claude-sonnet-5 \
  --user-persona-path swe-interact-persona.md \
  --bridge acp
```


# Skills



Skills let you inject context files (prompts, rules, reference docs) into agent trials. A skill is any directory containing a `SKILL.md` file. When passed via `--skill`, Harbor copies the skill directory into the agent's sandbox so the agent can read it during execution.

## Local skills

Pass a local directory path:

```bash
harbor run -d terminal-bench@2.0 -a claude-code \
  --skill ./my-skills/python-style
```

The directory must contain a `SKILL.md` at the top level or in subdirectories.

## Git skills

Instead of checking skills into every project, you can reference a git repository. Harbor clones the repo, resolves a commit SHA, and caches the result locally.

### Shorthand syntax

```bash
# Default branch, reading skills from repo/skills/
harbor run --skill org/repo -a claude-code -d my-dataset

# Tag or branch, reading skills from repo/skills/
harbor run --skill org/repo@v1.2.0 -a claude-code -d my-dataset
harbor run --skill org/repo@main -a claude-code -d my-dataset
```

Shorthand always resolves to `github.com` and uses the repository's `skills/` directory.

### Full URL syntax

```bash
# Default branch, reading skills from repo/skills/
harbor run --skill https://github.com/org/repo -a claude-code -d my-dataset

# Subdirectory within a repo (uses /tree/ref/path format)
harbor run --skill https://github.com/org/repo/tree/main/skills/python -a claude-code -d my-dataset
```

Plain repository URLs use the repository's `skills/` directory. The `/tree/<ref>/<subdir>` format lets you point at a specific subdirectory of a monorepo.

### Multiple skills

Use `--skill` multiple times to inject several skills. If two skills share the same directory name, the last one wins:

```bash
harbor run \
  --skill org/common-skills \
  --skill ./local-overrides \
  -a claude-code -d my-dataset
```

## Caching

Git skills are cached locally at `~/.cache/harbor/skills/<host>/<org>/<name>/<sha>/`. Once a commit is cached, subsequent runs skip the clone. Different subdirectories from the same repository and commit are handled automatically.

To clear the cache:

```bash
rm -rf ~/.cache/harbor/skills
```

## Reproducibility

When a job runs, Harbor records each skill's provenance in the job lock file:

* **`name`** -- the skill directory name
* **`source`** -- local path used during the run
* **`digest`** -- SHA-256 content hash of all files in the skill
* **`git_url`** -- the source repository (for git skills)
* **`git_commit_id`** -- the resolved commit SHA (for git skills)

This ensures that job results are fully traceable back to the exact skill content that was used, even if the branch has since moved.


# API Overview



The launch functionality available through the Web UI and CLI is largely available through an API.

The base URL for all endpoints is:

```text
https://ofhuhcpkvzjlejydnvyd.supabase.co/functions/v1
```

## Authentication

Every request needs a Harbor API key sent as a bearer token. See
[Harbor Hub API Key](/docs/hub/api-key) for how to create one.

```http
Authorization: Bearer sk-harbor-...
Content-Type: application/json
```

## Errors

All errors share one shape:

```json
{
  "error": {
    "code": "validation_failed",
    "message": "config must contain at least one agent"
  }
}
```

| Code                              | Status | Meaning                                                                      |
| --------------------------------- | ------ | ---------------------------------------------------------------------------- |
| `bad_request`                     | 400    | Malformed request, or a missing/oversized `Idempotency-Key`                  |
| `validation_failed`               | 400    | The body parsed but broke a schema or server-side rule                       |
| `unauthorized`                    | 401    | Missing or invalid API key                                                   |
| `forbidden`                       | 403    | Not approved for remote rollouts, or not a member of the chosen organization |
| `not_found`                       | 404    | The job does not exist or is not visible to you                              |
| `method_not_allowed`              | 405    | HTTP method not supported on that path                                       |
| `replacement_required`            | 409    | A credential with this identity already exists; confirm the replacement      |
| `replacement_stale`               | 409    | The credential changed while you were confirming; re-read and retry          |
| `quota_exceeded`                  | 429    | You are over your hosted quota                                               |
| `server_error`                    | 500    | Internal failure                                                             |
| `agent_version_resolution_failed` | 503    | Could not resolve the latest agent version; retry                            |


# CLI: Leaderboards



When creating a leaderboard, optionally restrict it to specific dataset-version
UUIDs in the create config:

```yaml
dataset_version_ids:
  - 11111111-1111-4111-8111-111111111111
dataset_version_refs:
  - latest
```

IDs and refs may be combined; the API resolves refs and stores their UUIDs.
Omitting both fields associates every version that exists when the leaderboard
is created. Setting both to empty lists associates none. Versions published
later are never added automatically.

Use `harbor hub leaderboard show BOARD` to display a curated leaderboard, or
add `--json` to print the complete read API response. `BOARD` may be a UUID or
an `org/package/name` slug.

Export an update-ready YAML or JSON definition, edit it, and apply it:

```bash
harbor hub leaderboard export BOARD --output board.yaml
harbor hub leaderboard update BOARD --config board.yaml
```

Simple definition fields can be updated directly; these flags override values
from `--config` when both are provided:

```bash
harbor hub leaderboard update BOARD \
  --title "New title" \
  --description "New description" \
  --visibility private
```

Rows have the same round-trip workflow. Use `--all` to export every row:

```bash
harbor hub leaderboard row list BOARD
harbor hub leaderboard row create BOARD --config new-rows.yaml
harbor hub leaderboard row export ROW_ID --output row.yaml
harbor hub leaderboard row update ROW_ID --config row.yaml
harbor hub leaderboard row update ROW_ID --status hide

harbor hub leaderboard row export BOARD --all --output rows.yaml
harbor hub leaderboard update BOARD --config board.yaml --rows rows.yaml
```

`row list` shows canonical ranks, configured leaderboard columns, status, trial
count, timestamps, and row IDs. It supports `--limit`, `--page`, `--json`, and
`--quiet` using the same paging behavior as `hub job list`.

The create config contains a `rows` list. Each row accepts `metadata`, `metrics`,
`status`, and optional `trial_ids`. You can also create a leaderboard and its
initial rows atomically with `harbor hub leaderboard create --rows new-rows.yaml`.

Definition and batch row changes commit atomically. A schema change is rejected
when any resulting row is invalid. Delete incompatible rows explicitly with
`harbor hub leaderboard row delete`, or update them in the same command with
`--rows`. Use `--dry-run` with the combined `--config` and `--rows` migration
form to validate it without committing.

The API mirrors these resource boundaries: `leaderboard-update` changes only
the definition; row create, read, update, delete, and trial changes use their
dedicated endpoints. Only `leaderboard-migrate` accepts definition and row
changes together.

Manage row provenance separately:

```bash
harbor hub leaderboard row trial list ROW_ID
harbor hub leaderboard row trial set ROW_ID --trial-id TRIAL_ID
harbor hub leaderboard row trial add ROW_ID --trial-id TRIAL_ID
harbor hub leaderboard row trial remove ROW_ID --trial-id TRIAL_ID
```

Leaderboard reads and mutation responses return `n_trials` rather than every
association.
`row trial list` pages through the associations; use `--json` for one page or
`--quiet` to stream every trial ID.

`set` can also read the complete replacement list from YAML or JSON:

```yaml
trial_ids:
  - 11111111-1111-1111-1111-111111111111
  - 22222222-2222-2222-2222-222222222222
```

```bash
harbor hub leaderboard row trial set ROW_ID --trial-ids-file trials.yaml
```

`set` replaces every association; use `set --clear` to remove them all. Trial
changes do not recompute row metadata or metrics.


# CLI



Please ensure you have either logged in with `harbor auth login` or minted an API key
(see [Creating a Harbor Hub API Key](/docs/hub/api-key))

## Launching a Job

The command `harbor run --launch` will attempt to launch a job under your personal organization

The `--launch` flag can be combined with a singular agent/model combination (e.g. `harbor run --launch -a claude-code -m anthropic/fable-5 -d harbor/hello-world`).
Agent/model sweeps must be launched using a config file (e.g. `harbor run --launch -c config.yaml`)

```yaml
# config.yaml
job_name: opus-vs-codex-sweep
organization: my-org          # omit to use your personal org
credential_mode: gateway      # or "direct"

n_attempts: 3                 # 1-10
n_concurrent_trials: 32       # 1-1000

# Every agent runs against every task. Trials = n_attempts x tasks x agents.
agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

  - name: codex
    model_name: openai/gpt-5
    secrets: [OPENAI_API_KEY, HF_TOKEN]
    n_concurrent: 8           # per-agent sub-limit under n_concurrent_trials
    env:
      RUST_LOG: info          # nonsensitive only - secrets go in `secrets`

  - name: oracle              # needs no inference credential
    secrets: []

datasets:
  - name: terminal-bench/terminal-bench-2-1
    ref: "6"                  # `ref` or `version`, never both
    n_tasks: 25
    exclude_task_names:
      - "flaky-*"

tasks:
  - name: harbor/hello-world
    ref: latest

# Supplied for this job only, read from your local environment at launch.
job_secrets:
  HF_TOKEN:
    from_env: HF_TOKEN

retry:
  max_retries: 2
  include_exceptions:
    - EnvironmentStartError
```

### Injecting Secrets

There are two methods of injecting secrets into the hosted job: flags `--stored-secret <ENV_VAR_NAME>` and `--one-off-secret ENV_VAR_NAME=secret_value`,
and config fields `job_secrets` and `secrets`.

When using flags, `--stored-secret` will pull `<ENV_VAR_NAME>` from the organization's stored credentials on the hub.
`--one-off-secret` will encrypt the secret value and ensure that the value is not persisted
in any job config files. The secret will be decrypted only when necessary and marked revoked
after the job finishes. Flags apply secrets to every agent/model combination in the job config.

In other words, if I launch with the following command: `harbor run --launch -c simple_config.yaml --stored-secret TEST_VAR`
where `simple_config.yaml` is

```yaml
job_name: simple-config
organization: my-org          # omit to use your personal org

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

tasks:
  - name: harbor/hello-world
    ref: latest
```

The final resolved config would be as follows:

```yaml
job_name: simple-config
organization: my-org

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY, TEST_VAR] # notice TEST_VAR is used in both agents

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY, TEST_VAR] # notice TEST_VAR is used in both agents

tasks:
  - name: harbor/hello-world
    ref: latest
```

| Flag                | Value                 | Effect                                                                                                               |
| ------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `--dry-run`         | —                     | Validate without queuing. Resolves tasks, agents, and the owning org, checks selections, and reports the trial count |
| `--org`             | `TEXT`                | Organization that should own the hosted job. Defaults to your personal org                                           |
| `--credential-mode` | `gateway` \| `direct` | `gateway` proxies the key through the Hub, `direct` hands the real key to the agent                                  |
| `--stored-secret`   | `NAME`                | Select a secret already stored in the owning org. Repeatable                                                         |
| `--one-off-secret`  | `NAME[=VALUE]`        | Supply a secret for this job only. Bare `NAME` reads it from your environment. Repeatable                            |
| `--env-file`        | `PATH`                | Load a `.env`. Every name in it is selected, as with `--one-off-secret`                                              |
| `--registry-secret` | `HOST=NAME_OR_ID`     | Pin a stored pull secret for a private image host. Repeatable                                                        |
| `--no-secrets`      | —                     | Launch with no credentials at all                                                                                    |

`--launch` and `--upload` are mutually exclusive, so the upload-side flags do not apply to a
hosted launch: `--upload`, `--public` / `--private`, `--share-org`, and `--share-user`.

## Listing Jobs

Using the command `harbor hub job list` you can print out a list of all jobs visible to your user on the hub.

| Flag         | Value                     | Effect                                                                  |
| ------------ | ------------------------- | ----------------------------------------------------------------------- |
| `--scope`    | `my` \| `shared` \| `all` | Visibility scope. Defaults to `my`, the jobs your user owns             |
| `--search`   | `TEXT`                    | Only jobs whose names contain the substring                             |
| `--agent`    | `NAME`                    | Filter by agent name. Repeatable, and multiple values match any of them |
| `--provider` | `NAME`                    | Filter by model provider. Repeatable                                    |
| `--model`    | `NAME`                    | Filter by model. Repeatable                                             |

## Job Overview

Use the command `harbor hub job show JOB_ID JOB_ID_2 ...` to print an overview similar to the one provided on the hub.

The overview reports the number of trials, errors, and retries, the average return on each
metric, the cost in USD, and token usage.

## Per-Task Job Breakdown

Use the command `harbor hub job tasks JOB_ID` to see a per-task breakdown of a job.

| Flag                               | Value  | Effect                                       |
| ---------------------------------- | ------ | -------------------------------------------- |
| `--search`                         | `TEXT` | Only tasks whose names contain the substring |
| `--agent`, `--provider`, `--model` | `NAME` | As for [`job list`](#listing-jobs)           |

## Trials Overview

Use the command `harbor hub job trials JOB_ID JOB_ID_2 ...` to list trials across one or more jobs.
In interactive terminals, you can switch pages to audit all trials without running a second command.

| Flag                               | Value                                                 | Effect                                               |
| ---------------------------------- | ----------------------------------------------------- | ---------------------------------------------------- |
| `--search`                         | `TEXT`                                                | Only trials whose names contain the substring        |
| `--agent`, `--provider`, `--model` | `NAME`                                                | As for [`job list`](#listing-jobs)                   |
| `--limit`                          | `N`                                                   | Page size. Defaults to 100 here                      |
| `--failed-only`                    | —                                                     | Only trials that errored or failed                   |
| `--include-retries`                | —                                                     | Include retry history, not just the latest execution |
| `--sort-by`                        | `started_at` \| `task_name` \| `name` \| `error_type` | Sort column                                          |
| `--sort-order`                     | `asc` \| `desc`                                       | Sort direction                                       |

## Comparing 2 or More Jobs

Use command `harbor hub job compare JOB_ID JOB_ID_2 ...` to get the side-by-side grid for performance on tasks

## Job Visibility

Use command `harbor hub job shares JOB_ID` to see who a job is shared with. This returns orgs and users

## Deleting Jobs

Use command `harbor hub job delete JOB_ID JOB_ID_2 ...` to permanently delete jobs you own from the hub, including all of their trials and shares.

The command prompts for confirmation before deleting anything; pass `--yes` / `-y` to skip the prompt (required when scripting or piping).

Only the job's owner can delete a job. Jobs linked to a leaderboard submission and hosted jobs that are still running cannot be deleted.

## Hosted Job Status

Use command `harbor hub job status JOB_ID` to get your job status. Returns counts of pending, running, failed, and completed trials.

## Trial Show

Use command `harbor hub trial show TRIAL_ID` to show a single trial's metadata

## Downloading a Trial

Use command `harbor hub trial download TRIAL_ID` to download a particular trial

| Flag                 | Value  | Effect                                                                        |
| -------------------- | ------ | ----------------------------------------------------------------------------- |
| `--output-dir`, `-o` | `PATH` | Directory to materialize the trial into. Defaults to `./trials`               |
| `--overwrite`        | —      | Replace an existing trial directory                                           |
| `--trajectory`       | —      | Download only `trajectory.json`. Errors if the trial has no stored trajectory |

## Retrying a Hosted Trial

For a trial in a job that was launched remotely, you can use the command `harbor hub trial retry TRIAL_ID` to retry those trials.

These flags are cumulative filters. `harbor hub trial retry --job JOB_ID --failed-only`
selects every trial in the job, then narrows to the ones that failed.

| Flag            | Value    | Selects                                  |
| --------------- | -------- | ---------------------------------------- |
| `--job`         | `JOB_ID` | Every trial in that job                  |
| `--search`      | `TEXT`   | Trials whose names contain the substring |
| `--agent`       | `TEXT`   | Trials run with that agent               |
| `--provider`    | `TEXT`   | Trials run with that provider            |
| `--model`       | `TEXT`   | Trials run with that model               |
| `--task`        | `TEXT`   | Trials run against that task             |
| `--exception`   | `TEXT`   | Trials that failed with that exception   |
| `--failed-only` | —        | Only trials that failed                  |
| `--yes`         | —        | Skip confirmation                        |

## Canceling a Hosted Trial

To cancel a hosted trial, use command `harbor hub trial cancel TRIAL_ID`

| Flag       | Value    | Effect                                                                |
| ---------- | -------- | --------------------------------------------------------------------- |
| `--job`    | `JOB_ID` | Cancel every trial in the job. Same as `harbor hub job cancel JOB_ID` |
| `--all`    | —        | Cancel every currently running trial                                  |
| `--reason` | `TEXT`   | Record a reason for the cancellation                                  |
| `--yes`    | —        | Skip confirmation                                                     |

The selection filters from [`trial retry`](#retrying-a-hosted-trial) apply here too.

## Adding Secrets

Use command `harbor hub secrets add NAME` to upload a secret to the Hub.

| Flag          | Value    | Effect                                                              |
| ------------- | -------- | ------------------------------------------------------------------- |
| `--org`       | `ORG_ID` | Organization to store the secret on. Defaults to your personal org  |
| `--job`       | `JOB_ID` | Scope the secret to a single job instead of account-wide            |
| `--from-env`  | —        | Read the value from the local environment variable of the same name |
| `--yes`, `-y` | —        | Supersede an existing secret without prompting                      |

## Listing Secrets

Use command `harbor hub secrets list` to list the names and metadata of your uploaded secrets.

| Flag                | Value    | Effect                                 |
| ------------------- | -------- | -------------------------------------- |
| `--org`             | `ORG_ID` | Organization to list secrets for       |
| `--job`             | `JOB_ID` | List secrets scoped to that job        |
| `--include-revoked` | —        | Include secrets that have been revoked |

## Deleting Secrets

Use command `harbor hub secrets delete` to revoke a secret.

| Flag      | Value    | Effect                                                                             |
| --------- | -------- | ---------------------------------------------------------------------------------- |
| `--org`   | `ORG_ID` | Revoke a secret in that organization                                               |
| `--job`   | `JOB_ID` | Revoke a secret scoped to that job                                                 |
| `--purge` | —        | Delete the record outright, so it no longer appears under `list --include-revoked` |

## Adding an Image Registry Secret

Use command `harbor hub secrets registry add` to add an image registry secret. Also supports `secrets registry list` and `secrets registry delete`.

| Flag          | Value  | Effect                                                              |
| ------------- | ------ | ------------------------------------------------------------------- |
| `--name`      | `TEXT` | Display name used to select the credential later                    |
| `--from-file` | `PATH` | Read the credential from a file, such as a GAR service account JSON |
| `--yes`, `-y` | —      | Supersede an existing credential without prompting                  |

## Shared Flags

The listing commands above also accept these:

| Flag            | Value | Effect                                                 |
| --------------- | ----- | ------------------------------------------------------ |
| `--quiet`, `-q` | —     | Print only IDs, for piping into `xargs`                |
| `--no-trunc`    | —     | Show full cell content, wrapping instead of truncating |
| `--no-headers`  | —     | Omit the header row                                    |
| `--page`        | `N`   | Fetch one specific page, disabling interactive paging  |
| `--json`        | —     | Return the raw API response as JSON                    |


# Custom Agents



Custom agents run over ACP from a source repository containing a `harbor-agent.json` manifest.
Connect the repository from your profile settings before using a private one.

The template below intentionally omits `source.ref`, so Harbor starts from the repository's default
branch and records the resolved commit SHA in the stored job config:

```json
{
  "config": {
    "retry": {
      "exclude_exceptions": [
        "AgentTimeoutError",
        "VerifierTimeoutError",
        "RewardFileNotFoundError",
        "RewardFileEmptyError",
        "VerifierOutputParseError",
        "ApiUsageLimitError",
        "AgentSafetyRefusalError"
      ],
      "include_exceptions": []
    },
    "agents": [
      {
        "name": "acp",
        "source": {
          "path": ".",
          "repo": "<github-owner>/<custom-agent-repo>",
          "type": "github",
          "manifest": "harbor-agent.json"
        },
        "model_name": "<model-provider>/<model-name>",
        "secrets": ["<MODEL_API_KEY_ENV_VAR>"]
      }
    ],
    "datasets": [
      {
        "ref": "<dataset-ref>",
        "name": "<dataset-org>/<dataset-name>",
        "n_tasks": 3
      }
    ],
    "job_name": "<job-name>"
  },
  "job_secrets": {
    "<MODEL_API_KEY_ENV_VAR>": "<model-api-key>"
  },
  "dry_run": false
}
```

Replace every `<...>` value before submitting. To select an explicit revision, add
`"ref": "<branch-tag-or-commit>"` inside `source`.

The source rules are:

* `name` must be `acp` when `source` is present.
* `source.type` must be `github`, and `source.repo` must use `owner/repo` form.
* `source.ref` is optional and accepts a branch, tag, or commit SHA. When omitted, Harbor resolves
  GitHub `HEAD`, which is the repository's configured default branch and is normally `main`, then
  records the exact commit SHA in the stored config so retries stay reproducible.
* `source.path` defaults to `.` and selects the repository directory holding the agent project.
* `source.manifest` defaults to `harbor-agent.json` and is resolved relative to `source.path`.
* Custom agents bring their own model credentials. Supply the key in `job_secrets` or store it
  as a hosted secret, and name it in the agent's [`secrets`](/docs/hosted-harbor/submitting-jobs#agent-secrets) either way. A custom
  agent reads `HOSTED_INFERENCE_TOKEN` and `HOSTED_INFERENCE_URL` rather than the provider's own
  variables, so the selection is what decides which credential backs that token.

The source directory needs a locked Python project and a manifest:

```json
{
  "schema_version": 1,
  "id": "<agent-id>",
  "version": "<agent-version>",
  "protocol": "acp",
  "runtime": {
    "kind": "python-uv",
    "python": "3.12",
    "project": ".",
    "lockfile": "uv.lock",
    "entrypoint": ["python", "-m", "<agent-module>"]
  }
}
```

`schema_version` must be `1`, `protocol` must be `acp`, `runtime.kind` must be `python-uv`, and
`runtime.python` must be `"3.12"`. The entrypoint must be an executable that speaks ACP over standard
input and output, and its first element is a command name rather than a path. Commit the manifest,
the project files, and `uv.lock` to the repository.

To check repository access, the ref, and the manifest without launching anything, send the same
request with `"dry_run": true`. A successful validation returns the resolved repository, the pinned
commit, the manifest identity, and the access mode.


# Remote Rollouts



The [Harbor Hub](https://hub.harborframework.com) can now be used to run remote harbor rollouts.
This guide provides instructions for launching remote jobs through the CLI, web UI, and API.

First, visit [the job launcher](https://hub.harborframework.com/jobs/launch).
Click the link in the banner to fill out a google form for access to remote rollouts.

Then follow the [API key creation guide](/docs/hub/api-key) to create an API key.

## Interfaces

| Page                                    | Use it for                                                          |
| --------------------------------------- | ------------------------------------------------------------------- |
| [Web UI](/docs/hosted-harbor/web-ui)    | Adding secrets and launching a job from the browser                 |
| [CLI](/docs/hosted-harbor/cli)          | Launching with `harbor run --launch`, then browsing jobs and trials |
| [API Overview](/docs/hosted-harbor/api) | Base URL, authentication, and error shapes                          |

## API Reference

| Page                                                             | Endpoints                                     |
| ---------------------------------------------------------------- | --------------------------------------------- |
| [Submitting Jobs](/docs/hosted-harbor/submitting-jobs)           | `POST /job-submit`, `GET /job-status`         |
| [Custom Agents](/docs/hosted-harbor/custom-agents)               | Running an ACP agent from a GitHub repository |
| [Managing Secrets](/docs/hosted-harbor/secrets)                  | `/secrets`, `/secrets/preflight`              |
| [Registry Credentials](/docs/hosted-harbor/registry-credentials) | `/registry-credentials`                       |
| [CLI: Leaderboards](/docs/hosted-harbor/cli-leaderboards)        | Curated leaderboards from the CLI             |


# Registry Credentials



`/registry-credentials` stores encrypted credentials for pulling private task images. It supports
Google Artifact Registry (GAR), GitHub Container Registry (GHCR), and Amazon Elastic Container
Registry (ECR).

The examples below use `$BASE` and `$KEY`. Get a key from
[Harbor Hub API Key](/docs/hub/api-key).

```bash
export BASE=https://ofhuhcpkvzjlejydnvyd.supabase.co/functions/v1
export KEY=sk-harbor-...
```

## Storing a Credential

`POST /registry-credentials` validates the credential, encrypts it, and returns metadata without ever
returning the secret. Every request takes a `registry_host` and a `display_name` of at most 100
characters, which is how jobs later select the credential. The provider is inferred from the host, so
`provider` is optional; if you do send it, it has to agree with the host.

| Provider | Host                                          | Secret fields                        |
| -------- | --------------------------------------------- | ------------------------------------ |
| `gar`    | `<region>-docker.pkg.dev`                     | `service_account_json`               |
| `ghcr`   | `ghcr.io`                                     | `username`, `token`                  |
| `ecr`    | `<account-id>.dkr.ecr.<region>.amazonaws.com` | `access_key_id`, `secret_access_key` |

A GAR credential:

```json
{
  "registry_host": "us-east1-docker.pkg.dev",
  "display_name": "prod-puller",
  "service_account_json": "{...service account JSON...}"
}
```

Use a service account with read access to the repository you need, typically
`roles/artifactregistry.reader` scoped to that repository.

A GHCR credential:

```json
{
  "registry_host": "ghcr.io",
  "display_name": "github-packages-puller",
  "username": "octocat",
  "token": "ghp_..."
}
```

Use a classic GitHub personal access token limited to `read:packages`.

An ECR credential:

```json
{
  "registry_host": "123456789012.dkr.ecr.us-east-1.amazonaws.com",
  "display_name": "production-ecr-puller",
  "access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "secret_access_key": "..."
}
```

Use a long-lived IAM access key limited to the ECR pull permissions your task images need. Temporary
session credentials, the ones beginning `ASIA`, are rejected because they need a session token that
Harbor does not store.

A successful response looks like:

```json
{
  "id": "00000000-0000-0000-0000-000000000003",
  "org_id": "00000000-0000-0000-0000-00000000000a",
  "provider": "gar",
  "registry_host": "us-east1-docker.pkg.dev",
  "display_name": "prod-puller",
  "fingerprint": "harbor-puller@my-project.iam.gserviceaccount.com",
  "status": "active",
  "created_at": "2026-08-13T14:30:00Z"
}
```

Reusing an existing `display_name` returns `409 replacement_required` with the existing record's
fingerprint; confirm it with `supersede_credential_id` exactly as with secrets. Each organization can
hold at most 20 active registry credentials.

For GAR, build the request from the service account file to avoid shell quoting problems:

```bash
curl -sS -X POST "$BASE/registry-credentials" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile sa ./gar-puller.json \
    '{registry_host:"us-east1-docker.pkg.dev", display_name:"prod-puller", service_account_json:$sa}')"
```

## Listing and Revoking Registry Credentials

`GET /registry-credentials` lists active credentials and returns metadata only. Use `status=revoked`
or `status=all` to include other states.

```bash
curl -sS "$BASE/registry-credentials?status=active" \
  -H "Authorization: Bearer $KEY"
```

`DELETE /registry-credentials` revokes a credential by default. Set `purge` to `true` only when the
stored record should be permanently deleted.

```bash
curl -sS -X DELETE "$BASE/registry-credentials" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"credential_id":"00000000-0000-0000-0000-000000000003","purge":false}'
```

## Using a Credential In a Job

Pin a stored credential by display name or ID in `registry_credentials`, as a sibling of `config`:

```json
{
  "config": {
    "job_name": "private-image-job",
    "agents": [
      {
        "name": "terminus-2",
        "model_name": "openai/gpt-4o",
        "secrets": ["OPENAI_API_KEY"]
      }
    ],
    "datasets": [
      {
        "name": "my-org/private-image-dataset",
        "ref": "latest"
      }
    ]
  },
  "registry_credentials": {
    "us-east1-docker.pkg.dev": "prod-puller"
  },
  "job_secrets": {
    "OPENAI_API_KEY": "sk-..."
  }
}
```

That mapping reads: for images on `us-east1-docker.pkg.dev`, use the stored credential named
`prod-puller`. Keys have to be supported registry hosts, and you can send at most 20 entries.

If exactly one active credential is available for a host you can leave `registry_credentials` out
entirely. When several match, the launch has to pick one by display name, credential ID, or the
owner-qualified display name Harbor shows.


# Managing Secrets



`POST /secrets` stores a reusable hosted secret, such as a model API key or an environment variable a
task requires. This is the API equivalent of [adding secrets](/docs/hosted-harbor/web-ui#adding-secrets) through the Web UI.

The examples below use `$BASE` and `$KEY`. Get a key from
[Harbor Hub API Key](/docs/hub/api-key).

```bash
export BASE=https://ofhuhcpkvzjlejydnvyd.supabase.co/functions/v1
export KEY=sk-harbor-...
```

```json
{
  "scope": "user",
  "env_var": "OPENAI_API_KEY",
  "value": "sk-...",
  "provider": "openai"
}
```

`scope` is `user` by default. Use `job` instead to scope a secret to a single job, which then
requires a `job_id`; `job_id` is rejected when the scope is `user`. You can pass an `org_id` to place
the secret in a specific organization, and an optional `provider` label of up to 64 characters.

The response returns metadata only, never the value:

```json
{
  "id": "00000000-0000-0000-0000-000000000002",
  "scope": "user",
  "job_id": null,
  "org_id": "00000000-0000-0000-0000-00000000000a",
  "env_var": "OPENAI_API_KEY",
  "provider": "openai",
  "value_last4": "abcd",
  "status": "active",
  "created_at": "2026-08-13T14:30:00Z"
}
```

## Replacing a Secret

Storing a secret whose `env_var` already has an active secret does not silently overwrite it.
The request comes back as `409 replacement_required` along with the record you are about to replace,
so you can see exactly what is being replaced:

```json
{
  "error": {
    "code": "replacement_required",
    "message": "An active secret ending in abcd already exists. Confirm that you want to supersede it.",
    "existing": {
      "id": "00000000-0000-0000-0000-000000000002",
      "scope": "user",
      "env_var": "OPENAI_API_KEY",
      "value_last4": "abcd",
      "created_at": "2026-08-13T14:30:00Z"
    }
  }
}
```

Send the same request again with `"supersede_credential_id": "<existing.id>"` to confirm. If that ID
no longer matches the active secret, because something changed in between, you get
`409 replacement_stale` instead and should re-read before retrying. Superseding
requires organization owner rights.

The same pattern applies to registry secrets below.

## Listing and Revoking Secrets

`GET /secrets` lists your secrets as metadata only. Filter with `scope`, `job_id`, and `status`,
where `status` accepts `active` (the default), `revoked`, or `all`.

```bash
curl -sS "$BASE/secrets?scope=user&status=active" \
  -H "Authorization: Bearer $KEY"
```

```json
{
  "secrets": [
    {
      "id": "...",
      "scope": "user",
      "job_id": null,
      "env_var": "OPENAI_API_KEY",
      "provider": "openai",
      "value_last4": "abcd",
      "status": "active",
      "created_at": "...",
      "last_used_at": null
    }
  ]
}
```

`DELETE /secrets` revokes a secret by environment variable name. Set `purge` to `true` only when the
stored record should be permanently deleted rather than marked revoked.

```json
{
  "scope": "user",
  "env_var": "OPENAI_API_KEY",
  "purge": false
}
```

## Checking Secrets Before Launching

`POST /secrets/preflight` is an advisory check that reports whether each agent in a config will
actually receive a credential its model can authenticate with. It never blocks a launch on its own.

```json
{
  "config": {
    "agents": [
      {
        "name": "terminus-2",
        "model_name": "openai/gpt-5-mini",
        "secrets": ["OPENAI_API_KEY"]
      }
    ]
  },
  "organization": "my-org",
  "declared_env_vars": ["OPENAI_API_KEY"]
}
```

Because secrets are scoped per agent, the check is run against each agent's own selection rather than
everything the organization stores: a name counts only when that agent's `secrets` lists it **and**
it resolves, either to an active secret in the owning organization or to a name in
`declared_env_vars`. Send `declared_env_vars` for the one-off keys you plan to pass as
`job_secrets`, since preflight never sees their values.

The response reports an overall `ok`, per-provider results under `providers`, per-agent detail under
`agent_requirements` — each with its `model`, `provider`, `missing_env_vars`, and `configured` — and
`task_requirements` listing the environment variables the resolved tasks hard-require. The older
`agents` field is still returned for existing clients. A task requirement counts as met
only when the secret is configured **and** `supplyable`. Reserved infrastructure names — `PATH`,
`LD_PRELOAD`, anything starting with `MODAL_`, `SUPABASE_`, `HOSTED_HARBOR_`, or `GCP_`, and anything
ending in `_PROXY` — are never exported into a trial's environment, so a task requiring one cannot be
satisfied by supplying your own key.


# Submitting Jobs



`POST /job-submit` launches a remote rollout.

The examples below use `$BASE` and `$KEY`. Get a key from
[Harbor Hub API Key](/docs/hub/api-key).

```bash
export BASE=https://ofhuhcpkvzjlejydnvyd.supabase.co/functions/v1
export KEY=sk-harbor-...
```

Request headers:

```http
Authorization: Bearer sk-harbor-...
Content-Type: application/json
Idempotency-Key: <unique-retry-key>
```

The `Idempotency-Key` header is **required**. Requests without it are rejected, and the key may be
at most 200 characters. Reusing a key returns the job that key already created instead of launching
a second one, so generate a fresh key per launch and reuse it only when retrying that same launch.
If `Idempotency-Key` is reused, the server will return the previous launch URL even if the config changes.

The top level of the request body takes these fields:

| Field                  | Required | Notes                                                                                                                                       |
| ---------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `config`               | yes      | The job configuration                                                                                                                       |
| `organization`         | no       | Name of the organization that will own the job. Defaults to your personal org                                                               |
| `job_secrets`          | no       | One-off environment variable name to secret value. Added just for this job, and injected only into agents whose `secrets` sub-field name it |
| `registry_credentials` | no       | Registry host to stored credential name or ID `"us-east1-docker.pkg.dev": "registry-cred-stored-name"`                                      |
| `dry_run`              | no       | Validate and resolve everything without creating a job. Defaults to `false`                                                                 |

`job_secrets` and `registry_credentials` are siblings of `config`, not fields inside it. This is
deliberate: it keeps plaintext secrets out of the job config that gets stored and replayed. Placing
either one inside `config` is rejected. Upon submission these credentials are routed through an encryptor and decrypted only at runtime.
Plaintext secrets are never stored on our platform.

Here is a launch using a registry dataset and a built-in agent:

```json
{
  "config": {
    "job_name": "I-love-harbor",
    "agents": [
      {
        "name": "terminus-2",
        "model_name": "openai/gpt-5.6-luna",
        "secrets": ["OPENAI_API_KEY"]
      }
    ],
    "datasets": [
      {
        "name": "harbor/hello-world",
        "ref": "latest",
        "n_tasks": 5
      }
    ],
    "n_attempts": 1,
    "n_concurrent_trials": 20
  },
  "job_secrets": {
    "OPENAI_API_KEY": "sk-..."
  },
  "dry_run": false
}
```

A successful launch returns the job and its viewer URL:

```json
{
  "job_id": "00000000-0000-0000-0000-000000000001",
  "job_name": "I-love-harbor",
  "viewer_url": "https://hub.harborframework.com/jobs/00000000-0000-0000-0000-000000000001",
  "status_url": "https://hub.harborframework.com/jobs/00000000-0000-0000-0000-000000000001",
  "n_trials": 1,
  "owner_org": { "id": "...", "name": "..." },
  "agent_sources": []
}
```

`owner_org` reports which organization ended up owning the job, and `agent_sources` echoes any
resolved custom agent sources including their pinned commits. On a dry run no job is created, so
`job_id` comes back `null`, but every source is still resolved and validated.

To submit a request stored in `launch.json`:

```bash
BASE="https://ofhuhcpkvzjlejydnvyd.supabase.co/functions/v1"
KEY="sk-harbor-..."

curl -sS -X POST "$BASE/job-submit" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d @launch.json
```

## Job Config

`config` mirrors the `JobConfig` the CLI sends, so unrecognized keys pass through untouched.
The fields the API validates directly are:

| Field                 | Required | Notes                                                                                                                                                         |
| --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agents`              | yes      | At least one agent                                                                                                                                            |
| `tasks`               | no       | Individual tasks                                                                                                                                              |
| `datasets`            | no       | Registry or git repo datasets                                                                                                                                 |
| `job_name`            | no       | Defaults to a UTC timestamp such as `2026-08-13__14-30-00` (this timestamp can be super annoying to look through later. So seriously consider setting a name) |
| `n_attempts`          | no       | Between 1 and 10. Defaults to 1                                                                                                                               |
| `n_concurrent_trials` | no       | Between 1 and 1000. Defaults to 4                                                                                                                             |
| `credential_mode`     | no       | `gateway` (default) or `direct`. `gateway` uses credential proxying. `direct` directly injects decrypted secrets into the task container                      |
| `retry`               | no       | `include_exceptions` and `exclude_exceptions` lists                                                                                                           |

You need at least one of `tasks` or `datasets`, and together they must resolve to at least one task.
The total number of trials is `n_attempts × tasks × agents`, and must land between 1 and 50,000.

On each agent, `name` is the Harbor agent type, `model_name` is the model it uses, and `secrets` is
the list of credentials that agent receives. Agent versions installed from a package manager are
pinned when the job is submitted, so every trial in a job installs the same release even if the job
sits queued across an upstream publish.

## Agent Secrets

Credentials are scoped per agent, not per job. Every agent carries its own `secrets` list naming the
environment variables it receives, so two agents in the same launch can be given different keys.

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "secrets": ["ANTHROPIC_API_KEY", "HF_TOKEN"]
}
```

Each name must look like an environment variable, matching `^[A-Z][A-Z0-9_]{0,63}$`, and
at most 64 of them may be selected per-agent. Secrets are matched to their values either
from stored organization secrets, or supplied through the launch's `job_secrets`. When there is
a secret supplied through `job_secrets` and stored in the organization secrets,
the `job_secrets` submission wins. A selected name backed by neither carries no value of its
own.

For example, if I launch a job with the following config snippet:

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "secrets": ["ANTHROPIC_API_KEY"]
}
"job_secrets": {}
```

and `ANTHROPIC_API_KEY` is not set on my Harbor Hub organization, the variable `ANTHROPIC_API_KEY=None`
would be injected into the agent's task environment. If `ANTHROPIC_API_KEY` is set on my Harbor Hub
organization, it will be appropriately injected.

If I instead launch with the following config snippet:

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "secrets": ["ANTHROPIC_API_KEY"]
}
"job_secrets": {"ANTHROPIC_API_KEY": "sk-ant-job-secret"}
```

Then the value provided in `job_secrets` (or a proxy) will be injected into the agent env
regardless of whether or not the same credential name is stored on my Harbor Hub organization.

You should send the `secrets` field for every agent.

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "secrets": ["ANTHROPIC_API_KEY"]
}
```

| `secrets`              | What the agent receives                              |
| ---------------------- | ---------------------------------------------------- |
| `[A, list, of, names]` | Exactly those names, as far as each one resolves     |
| `[]`                   | Nothing. The agent starts with no credentials at all |

The `oracle` and `nop` agents need no inference credential and can be launched with `"secrets": []`.

Under the default `gateway` credential mode, a selected credential for `openai`, `anthropic`,
`openrouter`, `xai`, or `gemini` arrives as a proxy capability rather than your real
key, injected under its own name alongside the provider's standard variables so an agent's usual
lookup finds it. Everything else you select is injected
under its own name with its real value. Credentials for `vercel_ai_gateway` and `devin`
require `"credential_mode": "direct"`; in gateway mode they are dropped.

Secrets should not be placed in an agent's `env`. That is only for nonsensitive environment variables.

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "env": [], // secrets do NOT go here
  "secrets": ["ANTHROPIC_API_KEY"] // secret names go HERE
}
"job_secrets": {"ANTHROPIC_API_KEY": "sk-ant-job-secret"} // secret {"key": value} go HERE
```

## Tasks

Use `tasks` for individual tasks. Each entry is either a registry task or a git task, never both.

A registry task takes a `name` in `org/name` form and an optional `ref`, which is a version ref like
`"latest"`, or a semantic version string.

A git task takes a `git_url` and a `path`, where `path` is the task directory relative to the repository
root and is **required**. Pin the revision with either `git_ref` (a branch, tag, or SHA) or
`git_commit_id` (a resolved 40-character SHA), but not both. Git tasks use those fields rather than
`ref`, and local filesystem paths cannot be used for remote rollouts.

```json
{
  "config": {
    "job_name": "individual-tasks",
    "agents": [
      {
        "name": "terminus-2",
        "model_name": "openai/gpt-5-mini",
        "secrets": ["OPENAI_API_KEY"]
      }
    ],
    "tasks": [
      {
        "git_url": "https://github.com/owner/private-tasks.git",
        "git_ref": "main",
        "path": "tasks/my-task"
      },
      {
        "name": "terminal-bench/torch-tensor-parallelism",
        "ref": "latest"
      }
    ]
  },
  "job_secrets": {
    "OPENAI_API_KEY": "sk-..."
  },
  "dry_run": false
}
```

Private GitHub repositories have to be connected or shared through the profile settings flow before
they can be used as a task or agent source. TODO: add pics here.

## Datasets

Use `datasets` for registry or git repo datasets.

A registry dataset takes a `name` in `org/name` form and either `ref` or `version`, but never both.
You can narrow it with `n_tasks` to cap how many tasks are drawn, and with the `task_names` and
`exclude_task_names` glob patterns.

A git repo dataset uses Harbor's `repo` source string, the same grammar as `harbor run --repo`. That
accepts `org/name`, `github.com/org/name`, a full https or ssh URL, or a GitHub `/tree/<ref>/<subdir>`
URL, each with an optional trailing `@<branch|tag|sha>`. The optional `path` is the tasks directory
relative to the repository root, which Harbor defaults to `tasks`.

```json
{
  "datasets": [
    {
      "repo": "my-org/my-tasks@main",
      "path": "tasks",
      "n_tasks": 5
    }
  ]
}
```

Git repo datasets pin their revision inside the repo string, so they do not take `ref` or `version`.
They also do not accept `git_url`, `git_ref`, or `git_commit_id` — those belong to `tasks`.

Whatever a `latest` or a tag resolves to is pinned into the stored config, so a saved job records the
concrete version that actually ran rather than a moving reference.

## Job Secrets

`job_secrets` is a flat map of environment variable name to value:

```json
{ "OPENAI_API_KEY": "sk-..." }
```

Names must be uppercase and start with a letter, values must be non-empty and at most 16,384
characters, and you can send at most 50 entries. Values are encrypted before storage and never
returned.

Secret-looking environment variables found inside `config` are automatically moved into this
encrypted channel before anything is saved, so plaintext never lands in a stored config. If the same
name appears in both places, the explicit `job_secrets` entry wins.

Sending a value here makes it available to the job; it does not decide which agent gets it. Name it
in that agent's [`secrets`](/docs/hosted-harbor/submitting-jobs#agent-secrets) too, unless you are relying on the omitted-`secrets`
fallback for the model's canonical key.

You can also store a key once as a hosted secret and leave it out of the launch entirely. Stored
secrets are still selected the same way, by naming them in `secrets`, and are read from the
organization that owns the job.

`GET /job-status` returns trial counts for one or more jobs.

```bash
curl -sS "$BASE/job-status?job_id=<uuid>" \
  -H "Authorization: Bearer $KEY"
```

Pass `job_id` once per job to check several at a time, or a comma-separated `job_ids` list. Add
`force_combined=true` to get the combined overview shape even for a single job. The same request
works as a `POST` with a `{"job_ids": [...], "force_combined": false}` body.

A single-job response reports `pending`, `running`, `completed`, `failed`, `canceled`, and `total`.


# Web UI























































## Adding Secrets

Unless you are running the `oracle` or `nop` agents which do not require API model inference, the first step will be adding your API secrets.
On the Harbor Hub, everything is owned by organizations. This includes jobs, trials, packages, and secrets.
When running remote rollouts, secrets will be selected from the organization chosen to own the job.

To add secrets to your personal org on the Web UI, first go to [The Hub](https://hub.harborframework.com)

Click on your profile in the top right corner:

<div className="dark:hidden">
    <img alt="The Harbor Hub home page, listing published datasets" src={__img0} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="The Harbor Hub home page, listing published datasets" src={__img1} placeholder="blur" />
</div>

Then click on your settings tab:

<div className="dark:hidden">
    <img alt="Profile Page" src={__img2} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Profile Page" src={__img3} placeholder="blur" />
</div>

Scroll down to the secrets section, and click "Add secret":

<div className="dark:hidden">
    <img alt="Secrets Tab" src={__img4} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Secrets Tab" src={__img5} placeholder="blur" />
</div>

Add a registry secret or an environment variable secret:

<div className="max-w-sm dark:hidden">
    <img alt="Secrets Modal" src={__img6} placeholder="blur" />
</div>

<div className="max-w-sm hidden dark:block">
    <img alt="Secrets Modal" src={__img7} placeholder="blur" />
</div>

Environment secrets can be injected
into the agent runtime during a rollout, registry secrets are never injected, and only used to resolve private image repositories referenced by tasks.

## Adding Secrets To Organizations

To add a secret to an organization you own, click the "Organization" button on the top of the page:

<div className="dark:hidden">
    <img alt="Org Button" src={__img8} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Org Button" src={__img9} placeholder="blur" />
</div>

Select an organization that you own, or create a new one.

Then select the settings tab of the organization to view the saved secrets there. Note that only organization owners may modify secrets.

<div className="dark:hidden">
    <img alt="Add Org Secrets" src={__img10} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Add Org Secrets" src={__img11} placeholder="blur" />
</div>

Finally, add a secret:

<div className="dark:hidden">
    <img alt="Add Org Secrets" src={__img12} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Add Org Secrets" src={__img13} placeholder="blur" />
</div>

## Launching a Job

After adding your secrets, go to the [job launcher](https://hub.harborframework.com/jobs/launch).

First, select the organization that should own this job.

<div className="dark:hidden">
    <img alt="Select Organization" src={__img14} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Select Organization" src={__img15} placeholder="blur" />
</div>

Then, select the dataset or task that you would like to evaluate. You can select tasks and
datasets that have been uploaded to the hub, or to GitHub. For private GitHub tasks, you must
first link your private GitHub repositories following [this guide](insert-link-here)

<div className="dark:hidden">
    <img alt="Select Sources" src={__img16} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Select Sources" src={__img17} placeholder="blur" />
</div>

### Adding Agents

Next, you will select the agent/model combinations that you would like to evaluate. By default,
the Harbor Hub will use credential proxying, ensuring that your provider secrets are never
injected plaintext into the agent container. For greater compatibility with agents and inference providers,
you may elect to use direct credential mode, which will inject your provider secrets directly into the agent
environment.

<div className="dark:hidden">
    <img alt="Select Agent" src={__img18} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Select Agent" src={__img19} placeholder="blur" />
</div>

Custom agents must be stored on GitHub, compatible with ACP, and read from `HOSTED_INFERENCE_TOKEN`/`HOSTED_INFERENCE_URL` instead of canonical provider tokens/base urls.

After selecting your agent, you may add additional environment variables that you would like set.
THIS FIELD IS NOT FOR SECRETS. Environment variables placed in these fields are persisted plaintext
in the job config. Additional kwargs can be added in the "Advanced agent options" section. To inspect
an agent's accepted kwargs, types, and defaults, run `harbor agent schema <name>` locally.

### Choosing Secrets to Inject

Once you have finished configuring the agents, choose which credentials each
one receives. The `Secrets` section lists the names of every secret uploaded to the
owning organization. You can select any number of them per
agent, including more than one inference provider key.

<div className="dark:hidden">
    <img alt="Select Secrets" src={__img20} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Select Secrets" src={__img21} placeholder="blur" />
</div>

Every selected secret is injected. One of them is also renamed: the key whose
provider matches that agent's selected model arrives under the name the agent
expects, so you do not have to store a second copy of it under a different
name. Everything else you selected is injected under its own name.

For example, running `claude-code` against `openrouter/zai/glm-5.2`, you would
select your organization's `OPENROUTER_API_KEY`. It matches the model's
`openrouter/` provider, and `claude-code` reads its credential from
`ANTHROPIC_API_KEY`, so that is the name it is injected under. Any other
secrets you selected, an `OPENAI_API_KEY`, a token a task needs, arrive
unchanged.

If none of the selected secrets match the model's provider, nothing is renamed
and the agent starts without a credential, so the trial fails with a
credential error. Selecting only `OPENAI_API_KEY` for that OpenRouter model
does exactly this: the key is injected under its own name, but it is not the
one the model needs.

### Overall Job Settings

The overall job settings include the name, retries, attempts, concurrency, and timeout multiplier.

<div className="dark:hidden">
    <img alt="Select Job Settings" src={__img22} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Select Job Settings" src={__img23} placeholder="blur" />
</div>

**Name**

Naming the job is recommended as the default timestamp can be difficult to identify later.

**Retries**

Retries change the number of times a trial will retry on infrastructure or other failures (which can be configured in advanced job settings below).

**Attempts**

Attempts per task refer to the number of trials that will be scored against a particular task.
e.g. 4 attempts with 3 retries can have a maximum of 12 rollouts per task, but only 4 of the rollouts will be scored.

**Concurrent Trials**

Concurrency can be set based on your api key rate limits. Harbor Hub will monitor concurrency
and back off if rate limits are causing failures, but it is recommended to set concurrency
such that rate limit failures do not occur.

**Timeout Multiplier**

Finally, the timeout multiplier refers to the time limit an agent has to complete a task.
The time limit is declared in the task, but if you would like to modify this timeout without
modifying the task, you can set the timeout multiplier to something other than `1.0`.
e.g. `0.5` gives half of the time limit, `2.0` gives twice as much time.

### Advanced Job Settings

<div className="dark:hidden">
    <img alt="Select Advanced Settings" src={__img24} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Select Advanced Settings" src={__img25} placeholder="blur" />
</div>


# LLM-as-a-Judge



import { Callout } from 'fumadocs-ui/components/callout';

<Callout type="info" title="Recommended: Rewardkit">
  For a declarative approach to llm-as-a-judge, see [Rewardkit](/docs/rewardkit/judge-criteria). The tutorial below shows how to do it manually with a custom Python script.
</Callout>

This example shows how to evaluate agent outputs with an LLM instead of pass/fail unit tests. The full source is in [`examples/tasks/llm-judge-example`](https://github.com/laude-institute/harbor/tree/main/examples/tasks/llm-judge-example).

In general, a judge is a script, just like any other verifier defined using `test.sh` and is executed in the environment after the agent runs. The only difference is that judges need API keys to access the LLM provider.

## Task overview

The agent is asked to write a funny poem. A verifier script sends the poem to Claude, which returns a `funny_score` between 0 and 1. That score is written to `reward.json`.

## Directory structure

```
llm-judge-example/
├── instruction.md
├── task.toml
├── environment/
│   └── Dockerfile
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    └── llm_judge.py
```

## Passing secrets to the verifier

The `[verifier.env]` section in `task.toml` lets you inject environment variables into the container during verification *after the agent runs*. Use `${VAR}` syntax to read from the host environment (for secrets), or pass literal values directly:

```toml title="task.toml"
[verifier]
timeout_sec = 900.0

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"   # from host env
MODEL_NAME = "claude-haiku-4-5"              # literal value
```

This keeps API keys out of your task source while making them available during verification.

## Judge script

This judge uses the Anthropic API with [structured outputs](https://docs.anthropic.com/en/docs/build-with-claude/structured-outputs) to get a validated score.

```python title="tests/llm_judge.py"
# /// script
# dependencies = [
#   "anthropic>=0.75.0",
#   "pydantic==2.12.5",
# ]
# ///

import json
import os
from pathlib import Path

from anthropic import Anthropic, transform_schema
from pydantic import BaseModel, Field


class FunnyScoreResponse(BaseModel):
    funny_score: float = Field(..., ge=0.0, le=1.0)


def main():
    poem = Path("/app/poem.txt").read_text()
    client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
    schema = transform_schema(FunnyScoreResponse.model_json_schema())

    response = client.messages.create(
        model=os.getenv("MODEL_NAME"),
        max_tokens=1024,
        output_config={"format": {"type": "json_schema", "schema": schema}},
        messages=[
            {"role": "user", "content": f"Rate how funny this poem is from 0.0 to 1.0.\n\nPoem:\n{poem}"}
        ],
    )

    result = FunnyScoreResponse.model_validate_json(response.content[0].text)
    Path("/logs/verifier/reward.json").write_text(
        json.dumps({"funny": result.funny_score}, indent=2)
    )


if __name__ == "__main__":
    main()
```

The test script simply runs the judge:

```bash title="tests/test.sh"
#!/bin/bash
uv run /tests/llm_judge.py
```

## Writing `reward.json`

Instead of a binary `reward.txt`, this task writes a JSON object with named metrics. Harbor reads both formats — `reward.txt` first, then `reward.json` as a fallback.

```json
{ "funny": 0.75 }
```

You can return multiple scores in a single JSON object:

```json
{ "creativity": 0.9, "humor": 0.7, "grammar": 1.0 }
```

## Running it

```bash
# Prerequisites
export ANTHROPIC_API_KEY="sk-ant-..."

# Run with oracle
harbor run -p examples/tasks/llm-judge-example -a oracle

# Run with a real agent
harbor run -p examples/tasks/llm-judge-example -a claude-code -m anthropic/claude-sonnet-4-5
```

## Adapting for your tasks

You are free to define your judge however you like, just be sure to put the corresponding API keys in the `[verifier.env]` section of `task.toml`.

If you want to copy this boilerplate, you can do the following:

1. Define a Pydantic model for the score(s) you need
2. Write a prompt that instructs the LLM how to evaluate the output
3. Write the scores to `/logs/verifier/reward.json`
4. Pass any required API keys via `[verifier.env]`

This pattern works with any LLM provider — swap the Anthropic client for OpenAI, Google, etc.

<Callout type="info" title="Cost">
  Each verification call costs API tokens.
</Callout>


# MCP Server Task



Multi-container tasks are useful for simulating external services. This example shows how to build a task where the agent must interact with an [MCP](https://modelcontextprotocol.io/) server running as a sidecar service. The full source is in [`examples/tasks/hello-mcp`](https://github.com/laude-institute/harbor/tree/main/examples/tasks/hello-mcp).

## Multi-container environments

Harbor tasks define their environments in the [`environment/` directory](/docs/tasks#environment). Every implementation of the `BaseEnvironment` class defines which files are required in that environment directory. Most environments expect a single `Dockerfile`, which is insufficient for multi-container tasks.

The `--env docker` environment supports multi-container tasks by preferring a `environment/docker-compose.yaml` file if present.

Note that the `DockerEnvironment` class is currently the only environment that supports multi-container tasks. We are actively working on adding cloud support for multi-container tasks.

## Task overview

In this example task, the agent is asked to connect to an MCP server, call a tool, and write the result to a file. The key concepts demonstrated are:

* Using **Docker Compose** to run additional services alongside the agent container
* Declaring MCP servers in `task.toml` so compatible agents can register them automatically
* Health checks to ensure the sidecar is ready before the agent starts
* Verifying the agent's output with a simple pytest check

## Directory structure

```
hello-mcp/
├── instruction.md
├── task.toml
├── environment/
│   ├── Dockerfile                 # Agent container
│   ├── docker-compose.yaml        # Compose override (adds MCP server)
│   └── mcp-server/
│       ├── Dockerfile             # MCP server container
│       └── server.py              # FastMCP server
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    └── test_outputs.py
```

## Docker Compose sidecar

When a task needs additional services (databases, APIs, MCP servers, etc.), place a `docker-compose.yaml` in the `environment/` directory. Harbor merges it with its base compose config automatically.

```yaml title="environment/docker-compose.yaml"
services:
  main:
    depends_on:
      mcp-server:
        condition: service_healthy

  mcp-server:
    build:
      context: ./mcp-server
    expose:
      - "8000"
    healthcheck:
      test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('localhost',8000),timeout=2); s.close()"]
      interval: 2s
      timeout: 5s
      retries: 15
      start_period: 5s
```

* `main` is the agent's container — Harbor configures it automatically. You only add overrides like `depends_on`.
* Additional services (here `mcp-server`) are defined normally. They share a Docker network with `main`, so the agent can reach them by service name.

## Declaring MCP servers in `task.toml`

You can list MCP servers in the task config so that compatible agents (e.g., Claude Code, Codex) register them automatically. Each agent decides how to consume the config — for example, Claude Code writes it to `~/.claude.json`.

```toml title="task.toml"
[[environment.mcp_servers]]
name = "mcp-server"
transport = "streamable-http"
url = "http://mcp-server:8000/mcp"
```

Supported transports are `"sse"`, `"streamable-http"`, and `"stdio"`. Network-based transports require a `url`; `stdio` requires `command` and optionally `args`.

## MCP server

The server uses [FastMCP](https://github.com/jlowin/fastmcp) and exposes a single tool over streamable HTTP:

```python title="environment/mcp-server/server.py"
from fastmcp import FastMCP

mcp = FastMCP("hello-mcp")

SECRET_VALUE = "harbor-mcp-secret-12345"

@mcp.tool()
def get_secret() -> str:
    """Returns a secret value."""
    return SECRET_VALUE

if __name__ == "__main__":
    mcp.run(transport="streamable-http", host="0.0.0.0", port=8000)
```

## Instruction

```markdown title="instruction.md"
There is an MCP server running at `http://mcp-server:8000/sse`
that exposes a tool called `get_secret`.

1. Connect to the MCP server
2. Call the `get_secret` tool
3. Write the returned secret value to `/app/secret.txt`
```

## Verification

The test script runs a pytest check that compares the file contents to the expected secret:

```python title="tests/test_outputs.py"
SECRET_VALUE = "harbor-mcp-secret-12345"

def test_secret_file_exists():
    with open("/app/secret.txt") as f:
        content = f.read()
    assert content.strip() == SECRET_VALUE
```

## Running it

```bash
# Verify with the oracle agent (runs solution/solve.sh)
harbor run -p examples/tasks/hello-mcp -a oracle

# Test with a real agent
harbor run -p examples/tasks/hello-mcp -a claude-code -m anthropic/claude-sonnet-4-5
```

import { Callout } from 'fumadocs-ui/components/callout';

<Callout type="info" title="Docker Compose & cloud providers">
  Docker Compose tasks currently only work with the local Docker environment (`--env docker`). Most cloud sandbox providers only support single-Dockerfile environments. We are actively working on multi-container support for cloud sandbox providers.
</Callout>


# Running Terminal-Bench



[Terminal-Bench](https://tbench.ai) is a benchmark for evaluating the performance of agents on terminal-based tasks. Harbor is the official harness for running Terminal-Bench 2.0.

To run Terminal-Bench you will first need to install [Harbor](/docs/getting-started). You'll know that it's installed correctly if you're able to run the oracle solutions for Terminal-Bench 2.0. Note that you will first need to install Docker and have it running on your machine:

```bash
harbor run -d terminal-bench/terminal-bench-2 -a oracle
```

You should then be able to try any of the more advanced features offered by Harbor, such as running Terminal-Bench with Claude Code on Daytona:

```bash
export DAYTONA_API_KEY="<your-daytona-api-key>"
export ANTHROPIC_API_KEY="<your-anthropic-api-key>"
harbor run \
  -d terminal-bench/terminal-bench-2 \
  -m anthropic/claude-haiku-4-5 \
  -a claude-code \
  --env daytona \
  -n 32
```

## Testing your own agent

See our docs on [agents](/docs/agents) for more information on how to test your own agent on Terminal-Bench.

## Submitting to the Terminal-Bench leaderboard

Leaderboard logs are stored in [this HuggingFace repo](https://huggingface.co/datasets/alexgshaw/terminal-bench-2-leaderboard). To submit your results, open a PR there following the instructions in the README.

## Viewing the Terminal-Bench leaderboard

You can view the leaderboard [here](https://tbench.ai/leaderboard).


# Task Structure



The Harbor task format is designed to be maximally flexible while still being intuitive to implement. The differences between the Harbor task format and the Terminal-Bench task format are [documented here](/docs/tasks/task-difference).

<Callout type="info" title="Agent skill available">
  Want your coding agent to guide you through task creation? Install the `create-task` skill:

  ```bash
  npx skills add harbor-framework/harbor --skill create-task
  ```
</Callout>

## Creating a task

To create a task, you can use the following command:

```bash
harbor init --task "<org>/<name>"
```

This will create a new task directory with the following structure:

import { File, Folder, Files } from 'fumadocs-ui/components/files';

<Files>
  <File name="instruction.md" />

  <File name="task.toml" />

  <Folder name="environment" defaultOpen>
    <File name="Dockerfile" />

    <File name="..." />
  </Folder>

  <Folder name="solution" defaultOpen>
    <File name="solve.sh" />

    <File name="..." />
  </Folder>

  <Folder name="tests" defaultOpen>
    <File name="test.sh" />

    <File name="..." />
  </Folder>
</Files>

You can then populate the files with your task's content.

<Callout type="info" title="Multi-step tasks">
  Harbor also supports tasks split into sequential steps, each with its own
  instruction, tests, and optional pre-agent setup. See
  [Multi-step tasks](/docs/tasks/multi-step).
</Callout>

<Callout type="info" title="Resource requirements">
  Tasks can declare CPU, memory, storage, GPU, and TPU needs under `[environment]`.
  Harbor applies them differently per provider using enforcement policies. See
  [Managing Resources](/docs/tasks/managing-resources).
</Callout>

To evaluate an agent on your task, you can use the following command:

```bash
harbor run -p "<path/to/task>" -a "<agent>" -m "<model>"
```

## Task files

### Instruction

The instruction is a markdown file that contains the task's instruction.

### Configuration & Metadata

The `task.toml` file contains the task's configuration and metadata. Metadata is arbitrary and can consist of any information a task developer wants. Config params are nested into their respective sections rather than flat.

An example is shown below:

```toml
schema_version = "1.4"

[task]
name = "<org>/<name>"
version = "1.0.0"
description = "A short description of the task"
authors = [{ name = "Steve Jobs", email = "steve@apple.com" }]
keywords = ["trivial", "programming"]

[metadata]
difficulty_explanation = "Trivial task for demonstration"
category = "programming"

[verifier]
timeout_sec = 120.0
env = { API_KEY = "sk-test-123" }
user = "root"  # optional: run the verifier as this OS user

[agent]
timeout_sec = 120.0
user = "agent"  # optional: run the agent as this OS user

[solution]
env = { API_KEY = "sk-test-123" }

[environment]
network_mode = "no-network"  # baseline; defaults to "public" when omitted
build_timeout_sec = 600.0
docker_image = "some-org/some-name:some-tag"
os = "linux"  # or "windows" to target Windows containers
cpus = 1
memory_mb = 2048
storage_mb = 10240
gpus = 0
gpu_types = ["H100", "A100"]
env = { SOME_ENV_VAR = "${SOME_ENV_VAR}" } # harbor run requests approval from the user for these env vars

[environment.tpu]                   # optional; omit the table if you don't need TPUs
type = "v6e"                        # alias (v3, v4, v5e, v5p, v6e, v7, trillium, ironwood) or canonical GKE label
topology = "2x4"                    # required; per-pod chip count = product of dimensions (here, 8)
# A task allocates one TPU slice per pod; specify a single spec rather than a list.
# Currently only the GKE environment honors this field.

[[environment.mcp_servers]]
name = "mcp-server"
transport = "streamable-http"
url = "http://mcp-server:8000/mcp"

[environment.healthcheck]
command = "curl -f http://localhost:8080/health"
interval_sec = 5.0
timeout_sec = 30.0
retries = 3
```

### Network policy

Network access uses **baselines** (set at env start, restored between phases), **phase overrides** (optional; only during `agent.run()` or `verifier.verify()`), and **run-time merges** (`--allow-environment-host`, `--allow-agent-host`). Multi-step tasks: `[steps.*]` overrides task-level fields.

| Field                                                      | Layer    | Applied                                                                  |
| ---------------------------------------------------------- | -------- | ------------------------------------------------------------------------ |
| `[environment].network_mode`                               | Baseline | Agent env start; shared verifier baseline                                |
| `[verifier.environment].network_mode`                      | Baseline | Separate verifier env start                                              |
| `[steps.verifier.environment].network_mode`                | Baseline | Per-step separate verifier env start                                     |
| `[agent].network_mode`, `[steps.agent].network_mode`       | Override | During matching `agent.run()`                                            |
| `[verifier].network_mode`, `[steps.verifier].network_mode` | Override | During matching `verify()`                                               |
| `--allow-environment-host`                                 | Run-time | Merged into `environment.extra_allowed_hosts` → `[environment]` baseline |
| `--allow-agent-host`                                       | Run-time | Merged into `agent.extra_allowed_hosts` → agent phase allowlist          |

Verifier baseline: **shared** → `[environment]`; **separate** → `[verifier.environment]` if set, else a copy of `[environment]`.

`[environment].network_mode` defaults to `"public"`. `[agent]` / `[verifier]` (and step equivalents) are optional overrides applied only when set **and** different from the phase baseline; matching the baseline is a no-op. Modes: `public`, `no-network`, or `allowlist` with optional `allowed_hosts` (exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostnames such as `*.example.com`, when supported by the selected environment; not URLs, bracketed IPv6 addresses, ports, or paths). In `allowlist` mode, empty or omitted `allowed_hosts` denies all egress. Hostnames are exact: for example, `ubuntu.com` does not allow `ask.ubuntu.com`; use a leading wildcard pattern such as `*.ubuntu.com` to allow subdomains. Wildcard apex behavior is provider-dependent. For portable tasks, include both `example.com` and `*.example.com` when both apex and subdomains are needed. Legacy `allow_internet = false` on a baseline section maps to `no-network`.

If a phase override differs from its baseline, the provider must support `dynamic_network_policy` or Harbor rejects the task. Use `verifier.environment_mode = "separate"` for a different verifier baseline without runtime switching. Pass `--allow-environment-host` for deps needed at env start; `--allow-agent-host` for deps needed only during `agent.run()` (e.g. `pypi.org`). On a `public` baseline, run-time host flags emit a warning and are ignored.

Examples: `examples/tasks/network-policy-matrix/`.

The configuration parameters are shown below:

import { TypeTable } from 'fumadocs-ui/components/type-table';

<TypeTable
  type={{
  "schema_version": {
    description: "Version of the task configuration format.",
    type: "string",
    default: '"1.4"',
    path: "schema_version",
  },
  "multi_step_reward_strategy": {
    description: "How to derive the trial-level reward from per-step verifier results in a multi-step task. \"mean\" computes per-key means across steps (missing keys treated as 0; steps without a verifier_result excluded). \"final\" uses the last step's verifier_result verbatim. Only applies to multi-step tasks; leave unset for single-step tasks. Defaults to \"mean\" when unset on a multi-step task.",
    type: '"mean" | "final" | null',
    default: "null",
    path: "multi_step_reward_strategy",
  },
  "task.name": {
    description: 'Package name in org/name format (e.g., "harbor/hello-world").',
    type: "string",
    path: "task.name"
  },
  "task.version": {
    description: "Task package version. Semantic versions are recommended, but any non-empty string is accepted.",
    type: "string | null",
    default: "null (new tasks are initialized to \"1.0.0\")",
    path: "task.version"
  },
  "task.description": {
    description: "Human-readable description of the task.",
    type: "string",
    default: '""',
    path: "task.description"
  },
  "task.authors": {
    description: 'List of task authors. Each entry has a name and optional email (e.g., [{ name = "Jane", email = "jane@example.com" }]).',
    type: "list[Author]",
    default: "[]",
    path: "task.authors"
  },
  "task.keywords": {
    description: "Keywords for search and categorization.",
    type: "list[string]",
    default: "[]",
    path: "task.keywords"
  },
  "metadata": {
    description: "Arbitrary metadata provided by the task author.",
    type: "object",
    path: "metadata"
  },
  "verifier.timeout_sec": {
    description: "Timeout in seconds for the verifier.",
    type: "number",
    default: 600.0,
    path: "verifier.timeout_sec"
  },
  "verifier.network_mode": {
    description: "Optional override during verify(). Applied only when set and different from the verifier baseline. See Network policy.",
    type: '"no-network" | "public" | "allowlist" | null',
    default: "null (no override)",
    path: "verifier.network_mode"
  },
  "verifier.allowed_hosts": {
    description: "Allowlist exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostname patterns when verifier.network_mode is allowlist. Bare hostnames are exact; wildcard apex behavior and IP target support are provider-dependent.",
    type: "list[string] | null",
    default: "null",
    path: "verifier.allowed_hosts"
  },
  "verifier.env": {
    description: "Environment variables to set when running the verifier.",
    type: "object",
    default: "{}",
    path: "verifier.env"
  },
  "verifier.user": {
    description: "Username or UID to run the verifier as. When set, the environment's default user is configured accordingly before verification. If not set, the environment's container default (typically root) is used.",
    type: "string | int | null",
    default: null,
    path: "verifier.user"
  },
  "verifier.environment_mode": {
    description: 'Where the verifier runs. "shared" (default when no [verifier.environment] is declared) reuses the agent\'s container; "separate" launches a dedicated verifier container with its own image. Omitting this field while declaring [verifier.environment] implies "separate". Declaring "shared" alongside [verifier.environment] is a validation error.',
    type: '"shared" | "separate" | null',
    default: null,
    path: "verifier.environment_mode"
  },
  "verifier.environment": {
    description: 'Optional [verifier.environment] section (same schema as the top-level [environment]). Defines a dedicated container the verifier runs in. When set, the task\'s tests/ directory becomes the verifier image\'s build context — the image must provide /tests/test.sh (Linux) or /tests/test.bat (Windows) at runtime; Harbor does not upload tests for separate envs. When environment_mode="separate" is declared without this section, a fresh copy of the top-level [environment] is used.',
    type: "EnvironmentConfig | null",
    default: null,
    path: "verifier.environment"
  },
  "verifier.environment.network_mode": {
    description: "Separate verifier env baseline. Same modes as environment.network_mode.",
    type: '"no-network" | "public" | "allowlist" | null',
    default: "null (effective default: public)",
    path: "verifier.environment.network_mode"
  },
  "verifier.environment.allowed_hosts": {
    description: "Allowlist exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostname patterns when verifier.environment.network_mode is allowlist. Bare hostnames are exact; wildcard apex behavior and IP target support are provider-dependent.",
    type: "list[string] | null",
    default: "null",
    path: "verifier.environment.allowed_hosts"
  },
  "agent.timeout_sec": {
    description: "Timeout in seconds for the agent. If not set, no timeout is enforced.",
    type: "number | null",
    default: null,
    path: "agent.timeout_sec"
  },
  "agent.network_mode": {
    description: "Optional override during agent.run(). Applied only when set and different from [environment]. See Network policy.",
    type: '"no-network" | "public" | "allowlist" | null',
    default: "null (no override)",
    path: "agent.network_mode"
  },
  "agent.allowed_hosts": {
    description: "Allowlist exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostname patterns when agent.network_mode is allowlist. Bare hostnames are exact; wildcard apex behavior and IP target support are provider-dependent.",
    type: "list[string] | null",
    default: "null",
    path: "agent.allowed_hosts"
  },
  "agent.user": {
    description: "Username or UID to run the agent as. When set, the environment's default user is configured accordingly before agent setup and execution. If not set, the environment's container default (typically root) is used.",
    type: "string | int | null",
    default: null,
    path: "agent.user"
  },
  "solution.env": {
    description: "Environment variables to set when running the solution.",
    type: "object",
    default: "{}",
    path: "solution.env"
  },
  "environment.build_timeout_sec": {
    description: "Timeout in seconds for environment build.",
    type: "number",
    default: 600.0,
    path: "environment.build_timeout_sec"
  },
  "environment.network_mode": {
    description: "Agent env baseline (defaults to public). Also the shared verifier baseline. See Network policy.",
    type: '"no-network" | "public" | "allowlist" | null',
    default: "null (effective default: public)",
    path: "environment.network_mode"
  },
  "environment.allowed_hosts": {
    description: "Allowlist exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostname patterns when environment.network_mode is allowlist. Bare hostnames are exact; wildcard apex behavior and IP target support are provider-dependent.",
    type: "list[string] | null",
    default: "null",
    path: "environment.allowed_hosts"
  },
  "environment.docker_image": {
    description: "A pre-built Docker image to use for the environment. When set, environment/Dockerfile is optional for supported environment types.",
    type: 'string | null',
    default: null,
    path: "environment.docker_image"
  },
  "environment.os": {
    description: 'Target operating system for the task container. "linux" (default) or "windows". When "windows", Harbor uses Windows-style paths, cmd.exe execution, tar-over-exec file transfer, and filters script discovery to .bat only. The Docker daemon mode and image OS are validated against this value at start; mismatches fail fast.',
    type: '"linux" | "windows"',
    default: '"linux"',
    path: "environment.os"
  },
  "environment.cpus": {
    description: "Number of CPUs requested by the task. When omitted, Harbor leaves CPU sizing to the selected provider.",
    type: "integer | null",
    default: null,
    path: "environment.cpus"
  },
  "environment.memory_mb": {
    description: "Amount of RAM requested by the task in megabytes. When omitted, Harbor leaves memory sizing to the selected provider.",
    type: "integer | null",
    default: null,
    path: "environment.memory_mb"
  },
  "environment.storage_mb": {
    description: "Amount of storage requested by the task in megabytes. When omitted, Harbor leaves storage sizing to the selected provider.",
    type: "integer | null",
    default: null,
    path: "environment.storage_mb"
  },
  "environment.gpus": {
    description: "Number of GPUs requested by the task. When omitted, Harbor does not request GPUs.",
    type: "integer | null",
    default: null,
    path: "environment.gpus"
  },
  "environment.gpu_types": {
    description: "List of acceptable GPU types (e.g., ['H100', 'A100', 'T4']). None means any GPU type is acceptable.",
    type: 'list[string] | null',
    default: null,
    path: "environment.gpu_types"
  },
  "environment.tpu": {
    description: "TPU slice specification (type + topology). When set, the environment requests a TPU node matching this spec; per-pod chip count is derived from the topology. Singular because a task allocates exactly one TPU slice per pod. Only supported on TPU-capable environments (currently GKE).",
    type: "TpuSpec | null",
    default: null,
    path: "environment.tpu"
  },
  "environment.tpu.type": {
    description: "TPU accelerator type. Accepts either a user-friendly alias (e.g., 'v6e', 'trillium', 'v4') or a canonical GKE label (e.g., 'tpu-v6e-slice', 'tpu7x').",
    type: "string",
    path: "environment.tpu.type"
  },
  "environment.tpu.topology": {
    description: "TPU topology as 'NxM' or 'NxMxK' (e.g., '2x4', '2x2x1'). Required — GKE's implicit default topology is not part of a stable contract, so omitting it would make Harbor runs non-reproducible across GKE versions. Per-pod TPU chip count is computed as the product of dimensions (e.g. '2x2x1' → 4 chips, '2x4' → 8 chips). Each dimension must be a positive integer (no leading zeros).",
    type: "string",
    path: "environment.tpu.topology"
  },
  "environment.allow_internet": {
    description: "Deprecated compatibility field. Prefer [environment].network_mode. When set and [environment].network_mode is omitted, false maps to no-network and true maps to public.",
    type: "boolean | null",
    default: null,
    path: "environment.allow_internet"
  },
  "environment.env": {
    description: 'Environment variables required for the task, resolved from the host at runtime. Supports ${VAR} and ${VAR:-default} template syntax.',
    type: "object",
    default: "{}",
    path: "environment.env"
  },
  "environment.mcp_servers": {
    description: 'List of MCP servers to make available to the agent. Each entry has a name, transport ("sse", "streamable-http", or "stdio"), and transport-specific fields (url for sse/streamable-http, command/args for stdio). Compatible agents will register these servers automatically.',
    type: "list[MCPServerConfig]",
    default: "[]",
    path: "environment.mcp_servers"
  },
  "environment.skills_dir": {
    description: "Path to a skills directory in the environment. Contents are copied to the agent's skills config directory.",
    type: "string | null",
    default: null,
    path: "environment.skills_dir"
  },
  "environment.healthcheck.command": {
    description: "Shell command to run as a healthcheck after environment start. Exit 0 means healthy.",
    type: "string",
    path: "environment.healthcheck.command"
  },
  "environment.healthcheck.interval_sec": {
    description: "Time in seconds between healthcheck attempts.",
    type: "number",
    default: 5.0,
    path: "environment.healthcheck.interval_sec"
  },
  "environment.healthcheck.timeout_sec": {
    description: "Maximum time in seconds for a single healthcheck command.",
    type: "number",
    default: 30.0,
    path: "environment.healthcheck.timeout_sec"
  },
  "environment.healthcheck.start_period_sec": {
    description: "Grace period in seconds after environment start during which failures don't count.",
    type: "number",
    default: 0.0,
    path: "environment.healthcheck.start_period_sec"
  },
  "environment.healthcheck.start_interval_sec": {
    description: "Interval in seconds between checks during the start period.",
    type: "number",
    default: 5.0,
    path: "environment.healthcheck.start_interval_sec"
  },
  "environment.healthcheck.retries": {
    description: "Number of consecutive failures before the healthcheck is considered failed.",
    type: "integer",
    default: 3,
    path: "environment.healthcheck.retries"
  },
  "source": {
    description: "Optional source string for task provenance.",
    type: 'string | null',
    default: null,
    path: "source"
  }
}}
/>

During task creation, you can pass a `--metadata-template` flag with a path to a TOML file to pre-populate `task.toml` with metadata fields and config defaults:

```bash
harbor tasks init "<task-name>" --metadata-template task-template.toml
```

Sections in the template override Harbor's built-in defaults. Anything not specified falls back to the defaults listed above.

### Environment

The environment definition is placed in an `environment/` folder. **Harbor does not require any specific file to exist in that directory**. Which file is required depends on the environment type being used for the evaluation. For example, to use `--env docker`, the `DockerEnvironment` class accepts any of: `[environment].docker_image`, an `environment/Dockerfile`, or `environment/docker-compose.yaml`. Setting `docker_image` lets you omit the Dockerfile when using a pre-built image. If you omit both `environment/Dockerfile` and `environment/docker-compose.yaml`, any other files in `environment/` are uploaded into the container workdir when the environment starts. Use `--force-build` only when you have a Dockerfile and want to rebuild from source instead of pulling the pre-built image. Different environment types could require other files to be present (e.g. an Apptainer environment could check for an `image.def` file). Most cloud sandbox providers only support `Dockerfile` defined environments and not docker compose.

The target container OS is declared via `[environment].os` in `task.toml`. It defaults to `"linux"`; set it to `"windows"` to target Windows containers (see [Windows tasks](/docs/tasks/windows-container-support) for details). Container-side paths, file transfer, command execution, and script discovery all adapt to this value automatically.

There are a few special paths in the environment's filesystem (Linux paths shown; Windows containers use the `C:` drive equivalents — `C:/logs`, `C:/tests`, `C:/solution`):

| Path              | Description                                                                                                |
| ----------------- | ---------------------------------------------------------------------------------------------------------- |
| `/logs/verifier/` | Contains the reward file and other verifier logs.                                                          |
| `/logs/agent/`    | A directory agents can use to store logs from their runs.                                                  |
| `/solution/`      | The solution folder is copied here by the Oracle agent at runtime and executed from the working directory. |
| `/tests/`         | The tests folder is copied here by the Harbor harness at runtime and executed from the working directory.  |

The `/logs/` directory is downloaded to the host after the agent/verifier run and are often useful for debugging and analysis.

### Solution (Optional)

The solution folder must contain a `solution/solve.sh` script (or `solve.bat` for Windows tasks; the right extension is selected based on `[environment].os`). Other dependencies are allowed. This folder is copied to `/solution` by the Oracle agent at runtime and executed from the working directory.

If no solution is provided, the Oracle agent cannot be used to sanity check the task.

### Tests

The tests folder must contain a `tests/test.sh` script (or `test.bat` for Windows tasks). The test script should install test dependencies and verify the agent completed the instruction. In Terminal-Bench, this was done by running a `pytest` command, but this is now left to the task developer.

Other dependencies are allowed in the `tests/` folder. This folder is copied to `/tests` by the Harbor harness at runtime and executed from the working directory. E.g. `bash /tests/test.sh` is executed from `/app` in many cases.

**We recommend using absolute paths in your test script to avoid relative path issues.**

Importantly, the test script must produce a reward file in the `/logs/verifier/` directory. This is the file that the verifier will read to determine if the task was successful.

There are two ways to produce a reward file:

| Reward File                  | Format                                                       | Description                                                                                                 |
| ---------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `/logs/verifier/reward.txt`  | Plain text (e.g. `1`)                                        | A plain text file containing a single integer or float value, typically `1` for success or `0` for failure. |
| `/logs/verifier/reward.json` | JSON (e.g. `{ "runtime_sec": 1.23, "accuracy": 0.95, ... }`) | A JSON file that can define multiple metrics as rewards, but they must be floats or integers.               |

You may use either `reward.txt` or `reward.json` as the output of your test script. Harbor will read `reward.json` by default and fall back to `reward.txt`.

For verifiers with multiple criteria, score aggregation, and LLM judging, see [Rewardkit](/docs/rewardkit).

Often, a reward can be determined by the exit code or a unit test command.

```bash title="tests/test.sh"
#!/bin/bash

uvx pytest /tests/test.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```

### Verifier environment (Shared vs Separate)

By default, the verifier runs inside the **same container as the agent** — it can see the workdir, any tools the agent installed, the agent's environment variables, etc. For tasks that need an isolated grading environment — proprietary grading code that the agent must not see, a different OS for grading, a clean image with verifier-only dependencies — declare a dedicated **separate** verifier environment.

You opt in by adding `[verifier.environment]` to `task.toml`:

```toml
[verifier]
environment_mode = "separate"  # optional when [verifier.environment] is present

[verifier.environment]
docker_image = "my-org/grading-image:latest"
cpus = 2
memory_mb = 1024
```

The two new fields under `[verifier]`:

* `environment_mode`: `"shared"` (default) or `"separate"`.
* `environment`: same schema as `[environment]` (including `network_mode` baseline). Network overrides: [Network policy](#network-policy).

Resolution rules when fields are omitted:

| `environment_mode` | `[verifier.environment]` | Result                                                            |
| ------------------ | ------------------------ | ----------------------------------------------------------------- |
| omitted            | omitted                  | `"shared"`                                                        |
| omitted            | present                  | `"separate"`                                                      |
| `"shared"`         | omitted                  | `"shared"`                                                        |
| `"shared"`         | present                  | **validation error**                                              |
| `"separate"`       | omitted                  | `"separate"` (uses a fresh copy of the top-level `[environment]`) |
| `"separate"`       | present                  | `"separate"` (uses the verifier-specific environment)             |

**For separate verifier environments**, Harbor builds the verifier image from one of the task's `tests/` directories — the step's own `steps/<name>/tests/` when it provides an OS-appropriate test script, otherwise the task's top-level `tests/`. The image must provide `/tests/test.sh` (Linux) or `/tests/test.bat` (Windows) on its own; Harbor does **not** upload `tests/` at runtime. A typical layout:

```text title="Task directory with separate verifier env"
my-task/
├── task.toml
├── instruction.md
├── environment/
│   └── Dockerfile        # agent's environment
└── tests/
    ├── Dockerfile        # verifier's environment (builds /tests/test.sh into the image)
    ├── test.sh
    └── grader.py
```

#### What gets transferred from the agent env to the verifier env

When a separate verifier env runs, Harbor copies these inputs from the agent env into the verifier env at the same paths:

* `/logs/artifacts/` (the agent's "publish" directory — files the agent intentionally produced for grading).
* Every artifact listed in the task-level `artifacts =` field, the trial-level artifacts, and the current step's `artifacts =` field.

Every artifact re-materializes at its **original absolute source path** in the verifier container ("no translation"): if you declared `/app/output.json`, read it at `/app/output.json`. Harbor creates parent directories during upload, so the verifier image does not need to pre-create them.

`/logs/agent/` and `/logs/verifier/` are **not** transferred implicitly. However, if you explicitly declare them as configured artifacts, they will be transferred — this is the canonical pattern for a **trajectory-grading verifier**:

```toml
artifacts = ["/logs/agent/trajectory.json"]
```

That single line makes the agent's trajectory file available to the separate verifier container at the same path.

#### Sidecar artifacts and collect hooks

Multi-container tasks (declared via `environment/docker-compose.yaml`) often hold their score signal inside a **sidecar service**: a database the agent wrote rows into, an API server that logged the agent's requests, a load generator with in-memory counters. In separate verifier mode all containers are torn down before verification, so that evidence must be captured first. Two mechanisms work together:

**1. Sidecar artifact entries** — add `service` to an artifact entry to pull a file from that compose service's filesystem instead of the agent's container:

```toml
artifacts = [
    "/app/output.json",                                          # from main (the agent), as usual
    { source = "/var/log/api/requests.log", service = "api" },   # from the api sidecar
]
```

**2. Collect hooks** — run a snapshot command inside a service after the agent finishes, to dump runtime state (database contents, in-memory counters) into files that artifact entries then collect:

```toml
[[verifier.collect]]
service = "postgres"
command = "pg_dump -U postgres app > /tmp/dump.sql"
timeout_sec = 60.0

artifacts = [{ source = "/tmp/dump.sql", service = "postgres" }]
```

The verifier reads sidecar evidence at the same original paths (`/var/log/api/requests.log`, `/tmp/dump.sql`).

**Shell.** Commands targeting `main` run under `bash` (the agent image is harbor-built and always provides it). Commands targeting a **sidecar** run under POSIX `sh -c`, because sidecars are arbitrary third-party images and `bash` is frequently absent from minimal ones (for example the `*-alpine` variants of `postgres`, `redis`, and `nginx`). Keep sidecar collect commands POSIX-compatible. If you need bash-specific syntax (`[[ ... ]]`, arrays, `source`, process substitution) on a sidecar whose image ships bash — the default `postgres`, `mysql`, `redis`, and `kafka` tags all do — invoke it explicitly, e.g. `command = "bash -c '[[ -f /data/ready ]] && pg_dump ...'"`; for anything elaborate, drop a script into the image and run `bash /path/snapshot.sh` to avoid nested quoting.

**Anti-cheat properties.** Sidecar evidence is pulled directly from each service's filesystem — a channel the agent's container cannot write to. In separate verifier mode, Harbor stops the `main` service *before* running sidecar collect hooks and pulling sidecar artifacts, so leftover agent processes cannot interfere. Corollaries for task authors:

* Evidence collected from **sidecars** is trustworthy as long as the agent could not gain code execution on the sidecar (network access to a service ≠ filesystem access). Keeping sidecars unexploitable is the task author's responsibility.
* Collect hooks targeting **`main`** run in an environment the agent fully controlled — treat their output with the same suspicion as any agent deliverable. They are fine for "did the agent's own service work" checks, never for tamper-sensitive signals.
* The stop-`main`-first guarantee applies to single-step trials and the **final step** of a multi-step trial. Earlier steps keep the `main` container running (later steps need it), so their sidecar evidence is collected with the agent's container still live and any processes it left behind able to reach the sidecar over the network. Put tamper-sensitive sidecar evidence on the final step (or in a single-step separate-verifier task); for intermediate steps, treat sidecar evidence as agent-influenceable.

Harbor validates artifact sets at task load. Because all services share one flat `artifacts/` base dir, entries from different services whose source paths are equal or nested would collide on the same host path; Harbor emits a load-time warning and, at collection time, keeps the first claimant and skips the rest (recorded in `manifest.json`). Avoid overlapping sidecar sources: on collision only the first-collected service's content survives, so an unintended overlap can silently drop the evidence you meant to score. The one hard error is a sidecar entry whose source is not an absolute path.

Sidecar artifacts and collect hooks require a compose-capable environment provider (docker, daytona, modal, ec2, islo, gke, novita, langsmith, blaxel, beam, vercel). See [`examples/tasks/sidecar-artifacts`](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/sidecar-artifacts) for a complete working task.

#### Per-step verifier environments (multi-step tasks)

Each step can override the trial-level verifier mode under `[steps.verifier]`. Mixed shared/separate is supported — for example, an early "build" step that uses the agent env for fast feedback, plus a final "grade" step that uses a separate locked-down grading image:

```toml title="task.toml (multi-step mixed mode)"
[[steps]]
name = "build"
# Inherits the trial-level mode (shared by default).

[[steps]]
name = "grade"
[steps.verifier.environment]
docker_image = "my-org/grading-image:latest"
```

Step-level resolution:

* `[steps.verifier].environment_mode` (when set) wins.
* `[steps.verifier.environment]` present + mode omitted → implies `"separate"`.
* Otherwise the step inherits the trial-level resolution.

Multi-step network fields follow the same baseline/override rules; see [Network policy](#network-policy).

Tests for each step are validated against the OS of that step's *effective* verifier environment, not always the top-level `[environment].os`. So a Linux agent can be graded by a Windows verifier env (and vice versa) — Harbor checks that the corresponding `test.bat` / `test.sh` exists in the step's `tests/` dir at task-load time.


# Managing Resources



Tasks declare resources in `task.toml`. Harbor applies CPU and memory using **enforcement policies**; storage, GPU, and TPU requests are passed through when the provider supports them.

## Task fields

```toml
[environment]
cpus = 2
memory_mb = 4096
storage_mb = 10240
gpus = 1
gpu_types = ["H100", "A100"]

[environment.tpu]   # optional; GKE only
type = "v6e"
topology = "2x4"
```

| Field          | Description                                                                           |
| -------------- | ------------------------------------------------------------------------------------- |
| `cpus`         | CPU count                                                                             |
| `memory_mb`    | RAM in MB                                                                             |
| `storage_mb`   | Ephemeral disk in MB                                                                  |
| `gpus`         | GPU count                                                                             |
| `gpu_types`    | Acceptable GPU types (optional)                                                       |
| `tpu.type`     | TPU accelerator type — alias (`v6e`, `trillium`, `v4`) or GKE label (`tpu-v6e-slice`) |
| `tpu.topology` | TPU topology as `NxM` or `NxMxK` (required; chip count = product of dimensions)       |

All fields are optional. Omitted fields use the provider's default sizing — Harbor does not inject defaults.

Separate verifier sandboxes can set their own values under `[verifier.environment]`. See [Separate verifier environments](/news/separate-verifier-sandboxes).

## Enforcement policies

CPU and memory each get an independent policy. Set them via `--cpus` / `--memory`, or `cpu_enforcement_policy` / `memory_enforcement_policy` in job or trial config.

| Policy      | Meaning                               | Requires `cpus` / `memory_mb`? |
| ----------- | ------------------------------------- | ------------------------------ |
| `auto`      | Use the provider's default mode       | No                             |
| `limit`     | Hard ceiling only                     | Yes                            |
| `request`   | Reservation only, no ceiling          | Yes                            |
| `guarantee` | Both reservation and hard ceiling     | Yes                            |
| `ignore`    | Do not pass the value to the provider | No                             |

```bash
harbor run -p "<path/to/dataset>" -m "<model>" -a "<agent>" \
  -e docker --cpus limit --memory guarantee
```

```yaml
environment:
  type: docker
  cpu_enforcement_policy: limit
  memory_enforcement_policy: auto
```

Use `--override-cpus`, `--override-memory-mb`, `--override-storage-mb`, `--override-gpus`, and `--override-tpu` (e.g. `v6e=2x4`) to replace task values at run time.

## Provider support

Harbor validates policies at job start. Unsupported combinations fail before trials run. `limit` and `guarantee` require limit support; `request` and `guarantee` require request support.

## Storage, GPUs, and TPUs

No enforcement policies. Harbor passes declared values to providers that support them:

| Resource | Providers                      |
| -------- | ------------------------------ |
| Storage  | Daytona, Islo, Runloop, GKE, … |
| GPUs     | Modal, GKE, Daytona            |
| TPUs     | GKE                            |

## Validation

| Check                                        | When              |
| -------------------------------------------- | ----------------- |
| Policy vs provider                           | Job creation      |
| Missing value for non-`auto`/`ignore` policy | Environment start |
| GPU / TPU / internet requirements            | Environment start |
| GPU and TPU both set (GKE)                   | Environment start |
| GPU on docker-compose task (Daytona)         | Environment start |


# Multi-step Tasks



import { File, Folder, Files } from 'fumadocs-ui/components/files';
import { Callout } from 'fumadocs-ui/components/callout';
import { TypeTable } from 'fumadocs-ui/components/type-table';

A multi-step task runs an agent through a sequence of ordered steps against a single, shared environment. Each step has its own instruction, tests, and optional setup hook. Steps share an environment, run sequentially, and produce per-step verifier results that roll up into a single trial-level reward.

Multi-step tasks are helpful when implementing long-horizon tasks with early stopping conditions, testing continual learning methods like memory, and observing an agent's ability to build on its prior work.

<Callout type="info" title="Agent skill available">
  Want your coding agent to guide you through creating a multi-step task? Install the `create-task` skill:

  ```bash
  npx skills add harbor-framework/harbor --skill create-task
  ```
</Callout>

## Directory layout

A multi-step task replaces the single-step `instruction.md`, `tests/`, and `solution/` at the task root with a `steps/` directory containing one sub-directory per step:

<Files>
  <File name="task.toml" />

  <Folder name="environment" defaultOpen>
    <File name="Dockerfile" />
  </Folder>

  <Folder name="steps" defaultOpen>
    <Folder name="step-one" defaultOpen>
      <File name="instruction.md" />

      <Folder name="workdir">
        <File name="setup.sh" />

        <File name="..." />
      </Folder>

      <Folder name="tests">
        <File name="test.sh" />
      </Folder>

      <Folder name="solution">
        <File name="solve.sh" />
      </Folder>
    </Folder>

    <Folder name="step-two">
      <File name="instruction.md" />

      <File name="..." />
    </Folder>
  </Folder>

  <Folder name="tests">
    <File name="test.sh" />
  </Folder>
</Files>

The task-level `environment/` directory (with the Dockerfile and shared environment assets) still lives at the task root — the environment is built once and shared across all steps. An optional task-level `tests/` directory provides shared test helpers (and optionally a fallback `test.sh`) that every step can use. It's uploaded to `/tests` for each step's verification; any step-level `tests/` files are uploaded afterward and override same-named files, so the step's own `test.sh` wins when present.

## Configuration

Declare steps in `task.toml` using `[[steps]]` array-of-tables entries. Order determines execution order.

```toml
schema_version = "1.4"

[task]
name = "harbor/example-multi-step"
version = "1.0.0"
description = "A three-step example task"

[environment]
build_timeout_sec = 600.0
workdir = "/app"

[[steps]]
name = "scaffold"
# Abort the trial if this step's reward falls below 1.0 —
# later steps depend on a working scaffold.
min_reward = 1.0

[steps.agent]
timeout_sec = 60.0

[steps.verifier]
timeout_sec = 30.0

[[steps]]
name = "implement"
min_reward = 0.5

[steps.agent]
timeout_sec = 120.0

[steps.verifier]
timeout_sec = 30.0

[[steps]]
name = "document"

[steps.agent]
timeout_sec = 60.0

[steps.verifier]
timeout_sec = 30.0
```

### Step fields

<TypeTable
  type={{
  "name": {
    description: "Step identifier. Must match the corresponding sub-directory under steps/.",
    type: "string",
    path: "steps[].name",
  },
  "agent.timeout_sec": {
    description: "Per-step agent execution timeout, in seconds. Overrides task-level [agent].timeout_sec; unset falls back to the task-level value.",
    type: "float | null",
    default: "null",
    path: "steps[].agent.timeout_sec",
  },
  "agent.user": {
    description: "User to run the agent as for this step. Overrides task-level [agent].user; unset falls back to the task-level value.",
    type: "string | int | null",
    default: "null",
    path: "steps[].agent.user",
  },
  "verifier.timeout_sec": {
    description: "Per-step verifier execution timeout, in seconds. Overrides task-level [verifier].timeout_sec; unset falls back to the task-level value.",
    type: "float | null",
    default: "null",
    path: "steps[].verifier.timeout_sec",
  },
  "verifier.env": {
    description: "Environment variables scoped to this step's verifier run.",
    type: "dict[str, str]",
    default: "{}",
    path: "steps[].verifier.env",
  },
  "verifier.user": {
    description: "User to run the verifier as for this step. Overrides task-level [verifier].user; unset falls back to the task-level value.",
    type: "string | int | null",
    default: "null",
    path: "steps[].verifier.user",
  },
  "verifier.environment_mode": {
    description: 'Whether this step\'s verifier runs in the agent\'s container ("shared") or a dedicated one ("separate"). Overrides the trial-level setting. Steps may mix modes within one task. See [Verifier environment](/docs/tasks#verifier-environment-shared-vs-separate) for resolution rules.',
    type: '"shared" | "separate" | null',
    default: "null",
    path: "steps[].verifier.environment_mode",
  },
  "verifier.environment": {
    description: 'Optional EnvironmentConfig for the step\'s separate verifier container. Same schema as the top-level [environment] section. When set, the task\'s tests/ directory is used as the verifier image\'s build context. See [Verifier environment](/docs/tasks#verifier-environment-shared-vs-separate).',
    type: "EnvironmentConfig | null",
    default: "null",
    path: "steps[].verifier.environment",
  },
  "min_reward": {
    description: "If set, abort the trial after this step when its rewards do not meet the threshold(s). A float gates on the \"reward\" key (1D convention); a dict (e.g., { correctness: 0.8, style: 0.5 }) gates on each declared key and aborts if any key is below its threshold or missing. A missing verifier_result (verifier crash) or missing gated key is treated as -inf. Ignored when verification is globally disabled.",
    type: "float | dict[str, float] | null",
    default: "null",
    path: "steps[].min_reward",
  },
  "healthcheck.command": {
    description: "Shell command to run as the healthcheck. Exit 0 means healthy.",
    type: "string",
    path: "steps[].healthcheck.command",
  },
  "healthcheck.interval_sec": {
    description: "Seconds between healthcheck attempts.",
    type: "float",
    default: "5.0",
    path: "steps[].healthcheck.interval_sec",
  },
  "healthcheck.timeout_sec": {
    description: "Maximum seconds for a single healthcheck command to run.",
    type: "float",
    default: "30.0",
    path: "steps[].healthcheck.timeout_sec",
  },
  "healthcheck.start_period_sec": {
    description: "Grace period in seconds after the step's setup during which failures do not count toward retries.",
    type: "float",
    default: "0.0",
    path: "steps[].healthcheck.start_period_sec",
  },
  "healthcheck.start_interval_sec": {
    description: "Interval in seconds between checks during the start period.",
    type: "float",
    default: "5.0",
    path: "steps[].healthcheck.start_interval_sec",
  },
  "healthcheck.retries": {
    description: "Number of consecutive failures before the healthcheck is considered failed, causing the step to abort.",
    type: "int",
    default: "3",
    path: "steps[].healthcheck.retries",
  },
  "artifacts": {
    description: "Artifacts to collect after this step's verification into steps/{name}/artifacts/. Appended to task-level and trial-level artifacts during this step's collection pass.",
    type: "list[str | ArtifactConfig]",
    default: "[]",
    path: "steps[].artifacts",
  },
}}
/>

The per-step healthcheck runs after the step's `workdir/setup.sh` (if any) completes and before the agent starts. It supplements the top-level environment healthcheck rather than replacing it; a failure aborts the step and the trial.

## Resuming agent context

By default each step starts the agent in a fresh conversation: the environment persists across steps, but the agent's context does not. Pass `--resume-trajectory` at run time when the agent should instead continue its native session from the previous step, receiving each step's instruction as a follow-up turn in the same conversation:

```bash
harbor run -t path/to/multi-step-task -a claude-code -m anthropic/claude-sonnet-5 --resume-trajectory
```

The agent's session per step, illustrated:

|                       | step 1 | step 2 | step 3 | ...    |
| --------------------- | ------ | ------ | ------ | ------ |
| default               | fresh  | fresh  | fresh  | fresh  |
| `--resume-trajectory` | fresh  | resume | resume | resume |

This is a run-level setting (`agent.resume_trajectory` in a job config), not a task field, so the same task can be evaluated with and without context carry-over. Harbor preserves the previous step's live agent state for resumed steps and calls the agent's native resume mode.

Only agents with native resume support (`capabilities.resume`) can run with `--resume-trajectory`; for other agents the trial fails before the first step rather than silently starting new sessions. The first step always starts fresh because there is no previous session.

A related run-level flag, [`--load-trajectory`](/docs/run-jobs/load-trajectory), seeds the agent's session from a previous run's native trajectory before the first step. It composes with `--resume-trajectory`: (load, resume, resume, ...) instead of (load, fresh, fresh, ...).

## The `workdir/` directory

Anything you place under `steps/{name}/workdir/` is uploaded to the container's WORKDIR before the agent runs for that step. This is the mechanism for staging step-specific files — fixtures, configs, seed data — into the location the agent will work from.

<Callout type="info" title="Files persist across steps">
  The container filesystem is shared across all steps in a trial. Files left in
  WORKDIR by step N are visible to step N+1 (and to its agent and verifier).
  This is intentional — it's how multi-step tasks compose work — but means step
  N's `workdir/` uploads can clobber files that an earlier step's agent created.
  If that matters, either pick non-colliding filenames or have an earlier
  step's `setup.sh` preserve/rename state.
</Callout>

### `workdir/setup.sh` (reserved)

If `steps/{name}/workdir/setup.sh` exists, it is executed after the step's `workdir/` contents are uploaded and before the agent runs. Use it for per-step prep that the agent shouldn't have to do itself: seeding a database, installing a one-off dependency, resetting file permissions, or running a pre-flight check.

Execution contract:

* **cwd** is WORKDIR (so relative paths to sibling `workdir/` files work naturally).
* **User** is the step's agent user (same as the agent will run under).
* **Non-zero exit** aborts the step: `exception_info` is recorded on the step result, the agent and verifier do not run for this step, and remaining steps are skipped.
* **The script stays in WORKDIR** after execution — it's uploaded like any other `workdir/` file. If the agent shouldn't see it, have the script remove itself on its last line:

```bash
#!/usr/bin/env bash
set -euo pipefail

# ... your setup work ...

rm -- "$0"
```

## Early stopping with `min_reward`

`min_reward` gates whether remaining steps run based on the current step's verifier result. Two shapes are supported:

* **Scalar**: `min_reward = 1.0` — gates on `rewards["reward"]` (the 1D convention written by `reward.txt`). Abort if below.
* **Dict**: `min_reward = { correctness = 0.8, style = 0.5 }` — gates on each declared key. Abort if any key falls below its threshold or is missing from the rewards dict. Use this for multi-dim rewards or when you want to gate on a non-default key.

The trial-level reward is computed from whatever steps did run.

* Missing reward key or no `verifier_result` at all → treated as `-inf` (aborts).
* When `config.verifier.disable` is true at the trial level, `min_reward` is ignored (there's nothing to compare against) — a debug log notes the skip.

Exceptions during a step (agent crash, setup failure) abort the trial independently of `min_reward`; the threshold check is in addition to, not in place of, the exception path.

## Trial-level reward: `multi_step_reward_strategy`

After all steps that will run have completed, Harbor derives a single trial-level `verifier_result` from the per-step results. The optional `multi_step_reward_strategy` field at the task root selects how (defaults to `"mean"` when unset):

<TypeTable
  type={{
  '"mean"': {
    description: "Per-key means across steps. For each key that appears in any step's rewards, the trial-level value is the mean across steps that produced a verifier_result (missing keys contribute 0; steps with no verifier_result are excluded from the denominator). Default.",
    type: "strategy",
  },
  '"final"': {
    description: "The last step's verifier_result is used verbatim. Preserves the full rewards dict from the final step (multi-dim rewards flow through unchanged) but discards every earlier step's reward signal.",
    type: "strategy",
  },
}}
/>

`"final"` is the right choice when the last step is an end-to-end verifier whose reward dict already represents the full-task signal.

<Callout type="info" title="Early stops and `final`">
  If a step's `min_reward` triggers an abort, `"final"` uses the aborted
  step's `verifier_result`, not the step the task author thought of as "final."
  Keep this in mind when designing thresholds alongside `"final"` strategy.
</Callout>

## Artifacts per step

Artifacts snapshot paths out of the environment into the trial directory. For a multi-step trial, collection runs **once per step** into `steps/{name}/artifacts/` after that step's verification. The list of paths collected at each pass is the concatenation, in order, of:

1. Task-level `artifacts` (`task.toml` root)
2. Trial-level `artifacts` (passed via `TrialConfig`)
3. Step-level `steps[].artifacts`

```toml
# Snapshotted after every step (e.g., the script that evolves across steps).
artifacts = ["/app/greet.sh"]

[[steps]]
name = "document"
# Only the document step writes README.md, so only collect it here.
artifacts = ["/app/README.md"]
```

After this trial, `steps/document/artifacts/` contains both `greet.sh` (task-level) and `README.md` (step-level); `steps/scaffold/artifacts/` contains only `greet.sh`.

## Example

A comprehensive worked example lives at [`examples/tasks/hello-multi-step-advanced/`](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/hello-multi-step-advanced) in the Harbor repo. It exercises per-step instructions, per-step `workdir/` uploads, per-step verifier environment variables, per-step healthchecks, `min_reward` early-stopping, and artifact collection per step (including a step-level artifact on the final step).


# Network Policy



Often, task authors will want to constrain the network access of the agent or verifier during a rollout for security or reward hacking mitigation.

Network policies can be specified in the `task.toml` file at varying degrees of granularity. How you choose to specify depends on your environment affordances and use case.

## Example

```toml title="task.toml"
[environment]
network_mode = "public"

[agent] # applies during agent.run() -- but *not* during agent.setup()!
network_mode = "allowlist"
allowed_hosts = ["pypi.org", "api.openai.com"]

[verifier] # applies during verifier.verify()
network_mode = "no-network"
```

## Network modes

Harbor supports three network modes: `public`, `no-network`, and `allowlist`.

| Network mode | Description                                                                                       | Supported environments                                                                                                                                                          |
| ------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `public`     | Full network access.                                                                              | All                                                                                                                                                                             |
| `no-network` | No network access.                                                                                | `docker`³, `daytona`, `e2b`, `langsmith`, `tensorlake`, `cwsandbox`, `wandb`, `runloop`, `modal`¹, `gke`², `ec2`, `novita`, `islo`, `blaxel`, `beam`, `hyperbrowser`, `vercel`⁷ |
| `allowlist`  | Network access only to targets listed in `allowed_hosts`; empty or omitted hosts deny all egress. | `docker`⁴, `daytona`¹, `e2b`, `islo`, `runloop`, `modal`¹, `novita`¹, `blaxel`¹, `beam`⁵, `tensorlake`⁶, `hyperbrowser`, `vercel`⁷                                              |

¹ Single-container tasks only (not in Docker Compose mode).
² Docker Compose (multi-container) tasks only.
³ Docker support requires Linux containers; Docker Windows containers do not support this network policy mode.
⁴ Docker support requires Linux containers and local Docker runtime support for the nftables kernel features used by Harbor's egress-control sidecar.
⁵ Beam resolves concrete hostnames to IP CIDRs before applying the policy; wildcard host entries are not supported.
⁶ TensorLake allowlists accept exact hostnames, leading-wildcard hostnames (a wildcard matches subdomains at any depth but not the apex domain), IPv4 literals, and IPv4 CIDR ranges, but not IPv6 targets.
⁷ Vercel allowlists are available for single-container tasks only. They match domain rules on the TLS SNI, accept exact and leading-wildcard hostnames but not IP literals or CIDR ranges, and deny plain-text HTTP to an allowlisted host. The policy applies from sandbox creation and can be updated on a running sandbox. Compose tasks support `public` and `no-network` only.

### Local Docker runtime compatibility

Docker `no-network`, `allowlist`, and dynamic network policy support require
Docker Linux containers. Docker Windows containers do not expose the Linux
networking primitives Harbor uses for these modes.

Docker `allowlist` and dynamic network policy support are implemented with an
egress-control sidecar that applies nftables rules. The ruleset uses nftables
`fib` expressions, including `fib daddr type local`. In kernel config terms,
the required capability is `CONFIG_NFT_FIB_INET=y` or
`CONFIG_NFT_FIB_INET=m`.

On Linux hosts, Harbor assumes the local Docker runtime supports the required
kernel features and lets the Docker environment start normally. On non-Linux
hosts, Harbor probes Docker's Linux VM before enabling Docker egress control. If
the VM exposes `/proc/config.gz`, Harbor requires `CONFIG_NFT_FIB_INET=y` or
`CONFIG_NFT_FIB_INET=m`; if `/proc/config.gz` is not present, Harbor assumes the
capability exists and lets the runtime report any later nftables failure.

Docker Desktop for macOS runs containers inside a LinuxKit VM whose kernel may
not enable `CONFIG_NFT_FIB_INET`. When Harbor can detect that missing support,
Docker `no-network`, `allowlist`, and dynamic network policy modes are rejected
during environment validation. If a runtime hides its kernel config but still
lacks the feature, `nft` rejects the ruleset and the sidecar fails during
startup. On macOS, use a Docker runtime whose Linux VM enables
`CONFIG_NFT_FIB_INET`, such as OrbStack, or run the task on a Linux Docker host.

`allowed_hosts` entries can be exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostnames such as `*.example.com`, when supported by the selected environment. They are not URLs, bracketed IPv6 addresses, ports, or paths. Bare hostnames are exact: `example.com` does not grant access to `www.example.com`. Wildcard apex behavior is provider-dependent. For portable tasks, include both `example.com` and `*.example.com` when both apex and subdomains are needed.

## Phases

Network policies can be specified for the following phases:

| Phase                    | Description                                                                                                                                                                 | Supported environments                                                                                       |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `[environment]`          | The baseline network policy configured at environment start time.                                                                                                           | Any environment that supports the requested network mode                                                     |
| `[agent]`                | Network access during `agent.run()` phase. This requires the environment provider to support dynamic network policy switching. This overrides the `[environment]` baseline. | `docker`⁴, `daytona`¹, `e2b`, `islo`, `modal`¹, `novita`¹, `beam`⁵, `hyperbrowser`, `vercel`⁷, `tensorlake`⁶ |
| `[verifier]`             | Network access during `verify()` phase. This requires the environment provider to support dynamic network policy switching. This overrides the `[environment]` baseline.    | `docker`⁴, `daytona`¹, `e2b`, `islo`, `modal`¹, `novita`¹, `beam`⁵, `hyperbrowser`, `vercel`⁷, `tensorlake`⁶ |
| `[verifier.environment]` | The baseline network policy configured at verifier environment start time, when using a separate verifier environment.                                                      | Any environment that supports the requested network mode                                                     |

Baseline phases are subject to the environment supporting the requested network mode (see the table above).

These phases can also be specified at the step-level, when using [multi-step tasks](/docs/tasks/multi-step).

## Capabilities

Each `BaseEnvironment` implementation declares an `EnvironmentCapabilities` model describing what it can do. The following capabilities govern network policy enforcement:

| Capability                             | Description                                                                                                            | Environments                                                                                                                                                                    |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `disable_internet`                     | The environment can run containers without internet access (`no-network`).                                             | `docker`³, `daytona`, `e2b`, `langsmith`, `tensorlake`, `cwsandbox`, `wandb`, `runloop`, `modal`¹, `gke`², `ec2`, `novita`, `islo`, `blaxel`, `beam`, `hyperbrowser`, `vercel`⁷ |
| `network_allowlist`                    | The environment can restrict egress to configured allowlist entries (`allowlist`).                                     | `docker`⁴, `daytona`¹, `e2b`, `islo`, `runloop`, `modal`¹, `novita`¹, `blaxel`¹, `beam`⁵, `tensorlake`⁶, `hyperbrowser`, `vercel`⁷                                              |
| `network_allowlist_hostnames`          | The environment can enforce exact hostname entries in `allowed_hosts`.                                                 | `docker`⁴, `daytona`¹, `e2b`, `islo`, `runloop`, `modal`¹, `novita`¹, `blaxel`¹, `beam`⁵, `tensorlake`⁶, `hyperbrowser`, `vercel`⁷                                              |
| `network_allowlist_wildcard_hostnames` | The environment can enforce leading-wildcard hostname entries in `allowed_hosts`.                                      | `docker`⁴, `daytona`¹, `e2b`, `runloop`, `modal`¹, `novita`¹, `blaxel`¹, `hyperbrowser`, `vercel`⁷, `tensorlake`⁶                                                               |
| `network_allowlist_ipv4_addresses`     | The environment can enforce IPv4 address literal entries in `allowed_hosts`.                                           | `docker`⁴, `daytona`¹, `e2b`, `modal`¹, `novita`¹, `beam`⁵, `tensorlake`⁶, `hyperbrowser`                                                                                       |
| `network_allowlist_ipv6_addresses`     | The environment can enforce IPv6 address literal entries in `allowed_hosts`.                                           | `docker`⁴, `beam`⁵                                                                                                                                                              |
| `network_allowlist_ipv4_cidrs`         | The environment can enforce IPv4 CIDR range entries in `allowed_hosts`.                                                | `docker`⁴, `daytona`¹, `modal`¹, `novita`¹, `beam`⁵, `tensorlake`⁶, `hyperbrowser`                                                                                              |
| `network_allowlist_ipv6_cidrs`         | The environment can enforce IPv6 CIDR range entries in `allowed_hosts`.                                                | `docker`⁴, `beam`⁵                                                                                                                                                              |
| `dynamic_network_policy`               | The environment can switch the active network policy after start, enabling `[agent]` and `[verifier]` phase overrides. | `docker`⁴, `daytona`¹, `e2b`, `islo`, `modal`¹, `novita`¹, `beam`⁵, `hyperbrowser`, `vercel`⁷, `tensorlake`⁶                                                                    |

¹ Single-container tasks only (not in Docker Compose mode).
² Docker Compose (multi-container) tasks only.
³ Docker support requires Linux containers; Docker Windows containers do not support this network policy mode.
⁴ Docker support requires Linux containers and local Docker runtime support for the nftables kernel features used by Harbor's egress-control sidecar.
⁵ Beam resolves concrete hostnames to IP CIDRs before applying the policy; wildcard host entries are not supported.
⁶ TensorLake allowlists accept exact hostnames, leading-wildcard hostnames (SDK 0.5.110+; a wildcard matches subdomains at any depth but not the apex domain), IPv4 literals, and IPv4 CIDR ranges, but not IPv6 targets.
⁷ Vercel allowlists are available for single-container tasks only. They match domain rules on the TLS SNI, accept exact and leading-wildcard hostnames but not IP literals or CIDR ranges, and deny plain-text HTTP to an allowlisted host. The policy applies from sandbox creation and can be updated on a running sandbox. Compose tasks support `public` and `no-network` only.

Daytona supports hostname/domain allowlists and IPv4 literal/CIDR allowlists, but a single allowlist policy cannot combine hostname/wildcard entries with IPv4 targets, and IPv6 targets are not supported.

TensorLake enforces egress rules host-side. The baseline policy is applied when the sandbox is created, and `[agent]` / `[verifier]` phase overrides are applied to the running sandbox as an atomic firewall swap (established connections are not interrupted). Under `allowlist`, DNS resolution stays available so hostname entries can resolve; under `no-network` (including an `allowlist` policy with no `allowed_hosts`), DNS is blocked along with all other egress. Hostname entries are resolved when the policy is applied, so an `allowed_hosts` entry that does not resolve fails environment startup — or, for a phase override, the switch is rejected and the previous policy stays enforced — rather than being skipped.

If a task requests a network mode the environment does not declare a capability for, Harbor rejects the trial at validation time rather than running with a weaker policy.

`EnvironmentCapabilities` also declares non-network capabilities (`gpus`, `tpus`, `windows`, `mounted`, and `docker_compose`) which are unrelated to network policy.

## Why phases?

Often, task authors will want to enforce different network policies for the agent (or verifier) phase, but do not want those policies enforced when setting up an agent in the environment. For example, you may want to be able to install Claude Code into the environment but then turn off all network access except for Anthropic's API during the rollout.

## Support

Environment implementations declare whether they support network policy configuration at start time and during runtime. Configuration at start time is more widely supported than during runtime, however, support is growing for both.


# Publishing a task



<Callout type="info" title="Agent skill available">
  Want your coding agent to walk you through publishing? Install the `publish` skill:

  ```bash
  npx skills add harbor-framework/harbor --skill publish
  ```
</Callout>

If you are new to Harbor, consider reading about [the composition of a Harbor task](/docs/tasks) first.

Tasks developed using Harbor can be easily shared with others inside and outside of your organization using the [Harbor registry](https://hub.harborframework.com/).

You can publish your task publicly or privately.

* **Public tasks** are visible and usable by everyone.
* **Private tasks** are visible only to members of the publishing org.

This guide will walk you through the process of publishing one or more tasks to the Harbor registry.

## Prerequisites

### Login to the Harbor registry

Before publishing, run:

```bash
harbor auth login
```

This opens a GitHub sign-in flow and creates your Harbor account. New accounts get an org using your GitHub username by default. You can publish to orgs you are an owner of. If you publish to an org that does not exist, it will be created for you.

You must be signed in to publish.

You can verify you are signed in by running:

```bash
harbor auth status
```

### Update old tasks

Task identifiers are specified in the `[task]` section of the `task.toml` file as `<org>/<name>`. This enables you to create globally unique identifiers that can be published to the registry.

If your task is missing the `[task]` section (any task created before Mar 30, 2026), you'll need to add it before publishing. You can do this by running:

```bash
harbor task update "<path/to/task>" --org "<org>"
```

This will add the `[task]` section to the task with the name `<org>/<folder-name>`. For org name, we recommend using your company or benchmark name.

You can update an entire directory of tasks by including the `--scan` flag:

```bash
harbor task update "<path/to/tasks>" --org "<org>" --scan
```

## 1) Publish the task

Use `harbor publish` to publish a task:

```bash
harbor publish "<path/to/task>"
```

You can publish multiple tasks in one command:

```bash
harbor publish "<path/to/task-a>" "<path/to/task-b>"
```

You can also publish all tasks in a directory:

```bash
harbor publish "<path/to/tasks>"
```

## 2) Publish options

* `-t / --tag`: add one or more tags (repeatable). `latest` is always included.
* `-c / --concurrency`: control upload concurrency.
* `--public`: make the task public. Private is the default.

By default, tasks are published with the `latest` tag.

### Tagging and visibility

```bash
harbor publish "<path/to/task>" -t benchmark-baseline --public
```

## 3) What publish does

* Computes and uploads the task archive.
* Resolves task metadata and task digest from `task.toml`.
* Stores the validated canonical task configuration, including defaults and multi-step configuration.
* Registers the version in the Harbor registry.

The output includes a registry package page link.

## Sharing

If you published it publicly, anyone can use it. If you published it privately, only members of the publishing org can use it. You can change visibility later using `harbor task visibility` or through the UI on [the registry website](https://hub.harborframework.com/).

Publishing visibility and access is documented in [Sharing](/docs/sharing/sharing).


# Differences from Terminal-Bench



The Harbor iteration of the Terminal-Bench task framework addresses many of the shortcomings we observed in the original framework, including structurally preventing common errors like copying tests into the container image, supporting many different environment definitions, allowing for non-binary rewards, supporting multi-file solutions, improving the intuitiveness of the task structure, and reducing the clutter of the task directory.

## How each component differs from Terminal-Bench

### Instruction

In Terminal-Bench, the instruction was nested in the `task.yaml`. This made viewing the instruction more difficult, made writing `task.yaml` files complicated (due to multiline strings) and often led to formatting issues. Reading task instructions required loading yaml files and indexing into dictionaries.

Harbor addresses this by giving the instruction its own `instruction.md` file. This improves formatting, readability, and loading of instructions.

### Configuration & Metadata

We move `task.yaml` to `task.toml` to improve readability and writability. Metadata is now arbitrary and can consist of any information a task developer wants. Config params are now nested into their respective sections rather than flat. An example is shown below.

```toml
version = "1.0"

[metadata]
author_name = "Steve Jobs"
author_email = "steve@apple.com"
difficulty = "easy"
category = "programming"
tags = ["trivial"]

[verifier]
timeout_sec = 120.0

[agent]
timeout_sec = 120.0

[environment]
build_timeout_sec = 600.0
docker_image = "some-org/some-name:some-tag"
cpus = 1
memory = "2G"
storage = "10G"
```

The `[environment].os` field controls Windows vs Linux behavior; it defaults to `"linux"` so existing Terminal-Bench-derived tasks keep working unchanged. Set `os = "windows"` to opt into Windows containers (see [Windows tasks](/docs/tasks/windows-container-support)).

### Environment

In Terminal-Bench, the only required environment-related file was `docker-compose.yaml`. Typically, this also means a `Dockerfile` would be present in the task, although the exact location was not enforced and it was at times nested in subdirectories. Docker compose introduced issues when trying to support other container runtimes.

Additionally, build context was also placed directly in the task directory, cluttering the folder and reducing readability. Sometimes, this also led to task developers copying `task.yaml`'s or even the `tests/` directory into the image on accident.

In Harbor, we require the environment definition to be placed in an `environment/` folder. **Harbor does not require any specific file to exist in that directory**. Which file is required depends on the environment type being used for the evaluation. For example, to use `--env docker`, the `DockerEnvironment` class accepts any of: `[environment].docker_image`, an `environment/Dockerfile`, or `environment/docker-compose.yaml`. Setting `docker_image` lets you omit the Dockerfile when using a pre-built image. Different environment types could require other files to be present (e.g. an Apptainer environment could check for an `image.def` file). Most cloud sandbox providers only support `Dockerfile` defined environments and not docker compose.

### Solution

The solution folder must contain a `solution/solve.sh` script. Other dependencies are allowed. This folder is copied to `/solution` by the Oracle agent at runtime and executed from the working directory.

The main difference between Harbor here is that we've moved from a single file solution, `solution.sh` to enabling dependencies.

### Tests

This is the most similar to the original Terminal-Bench setup. The main difference here is that the `run-tests.sh` script from Terminal-Bench is now `tests/test.sh` to reflect the fact that it is part of the test dependencies and is copied to the same location (`/tests`) at runtime by the Harbor harness.

Terminal-Bench tasks specified a parser, implemented as part of the Terminal-Bench repo – NOT as part of the task. The parser was in charge of mapping the test command's stdout to a binary reward. This created an awkward dependency between task logic and Terminal-Bench logic. We've gotten rid of this in Harbor. Tasks are now in charge of producing a reward. Currently, this is done by producing a `/logs/verifier/reward.txt` file.

Most Terminal-Bench tasks can produce a reward by checking the exit code of the `pytest` command:

```bash
# ... test script above

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```


# Tutorial



<Callout type="info" title="Agent skill available">
  Prefer having your coding agent walk you through it interactively? Install the `create-task` skill:

  ```bash
  npx skills add harbor-framework/harbor --skill create-task
  ```
</Callout>

## Step 0: Install Harbor

Follow our [installation instructions](/docs/getting-started) to install Harbor, which involves installing the package and its dependencies.

## Step 1: Create your task

Now that Harbor is installed, run the following command to create a new task directory with the required files:

```bash
harbor task init ssh-key-pair
```

This will generate a task directory with the following structure:

```
ssh-key-pair/
├── instruction.md         # Task instructions
├── task.toml              # Configuration and metadata
├── environment/
│   └── Dockerfile         # Container definition
├── solution/
│   └── solve.sh           # Solution script
└── tests/
    │── test_outputs.py    # Pytest unit tests
    └── test.sh            # Test verification script
```

## Step 2: Write the task instructions

Open the `instruction.md` file in your task directory and add the task description:

```markdown title="ssh-key-pair/instruction.md"
# SSH Key Pair Generation

Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`.

Don't make them password protected.
```

## Step 3: Configure task metadata

Open the `task.toml` file and configure your task metadata:

```toml title="ssh-key-pair/task.toml"
version = "1.0"

[metadata]
author_name = "Your Name"
author_email = "your.email@example.com"
difficulty_explanation = "Simple SSH key generation command"
category = "system-administration"
tags = ["ssh", "cryptography", "linux"]

[verifier]
timeout_sec = 120.0

[agent]
timeout_sec = 120.0

[environment]
build_timeout_sec = 600.0
```

Add `os = "windows"` here to target Windows containers; the default is `"linux"`. Add `cpus`, `memory_mb`, `storage_mb`, or `gpus` when the task needs explicit resources.

## Step 4: Create the task environment

Open the Dockerfile in the `environment/` directory that was generated:

```dockerfile title="ssh-key-pair/environment/Dockerfile"
FROM ubuntu:24.04

# Create working directory
WORKDIR /app

# Install openssh-client for the task
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*
```

This Dockerfile defines the environment an agent will interact with through the terminal. Add any dependencies your task requires here.

## Step 5: Test your solution idea

Before writing the automated solution, you'll want to manually verify your approach works. Build and run the container interactively:

```bash
harbor task start-env -p ssh-key-pair -e docker -a -i # or use daytona or modal
```

Inside the container, test that the following command solves the task without requiring interactive input:

```bash
ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""
```

Verify the keys were created correctly:

```bash
ls -l ~/.ssh/id_rsa*
```

You should see:

```
~/.ssh/
├── id_rsa      (-rw-------  600  private key)
└── id_rsa.pub  (-rw-r--r--  644  public key)
```

Exit the container with `exit` or `Ctrl+D`.

## Step 6: Write the solution script

Take the command you verified in the previous step and create the solution script. This file will be used by the Oracle agent to ensure the task is solvable.

Update the `solution/solve.sh` file:

```bash title="ssh-key-pair/solution/solve.sh"
#!/bin/bash

ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""
```

Make sure the script is executable:

```bash
chmod +x ssh-key-pair/solution/solve.sh
```

## Step 7: Create the test script

The test script verifies whether the agent successfully completed the task. It must produce a reward file in `/logs/verifier/`. This tutorial uses pytest for simplicity, but for multi-criterion verifiers, weighted scoring, or LLM-as-a-judge rubrics, consider using [Reward Kit](/docs/rewardkit) instead.

Update the `tests/test.sh` file:

```bash title="ssh-key-pair/tests/test.sh"
#!/bin/bash

apt-get update
apt-get install -y curl

curl -LsSf https://astral.sh/uv/0.9.5/install.sh | sh

source $HOME/.local/bin/env

# Run pytest tests
uvx \
  --python 3.12 \
  --with pytest==8.4.1 \
  pytest /tests/test_outputs.py

# Check exit code and write reward
if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```

Now create the Python test file:

```python title="ssh-key-pair/tests/test_outputs.py"
import os
from pathlib import Path


def test_key_files_exist() -> None:
    """Test that both private and public key files exist."""
    private_key = Path.home() / ".ssh" / "id_rsa"
    public_key = Path.home() / ".ssh" / "id_rsa.pub"

    assert private_key.exists(), "Private key file does not exist"
    assert public_key.exists(), "Public key file does not exist"


def test_key_file_permissions() -> None:
    """Test that the key files have correct permissions."""
    private_key = Path.home() / ".ssh" / "id_rsa"
    public_key = Path.home() / ".ssh" / "id_rsa.pub"

    private_perms = oct(os.stat(private_key).st_mode)[-3:]
    public_perms = oct(os.stat(public_key).st_mode)[-3:]

    assert private_perms == "600", (
        f"Private key has incorrect permissions: {private_perms}"
    )
    assert public_perms == "644", (
        f"Public key has incorrect permissions: {public_perms}"
    )


def test_key_format() -> None:
    """Test that the public key has the correct RSA format."""
    public_key = Path.home() / ".ssh" / "id_rsa.pub"

    with open(public_key, 'r') as f:
        content = f.read()

    assert content.startswith("ssh-rsa "), "Public key does not start with 'ssh-rsa'"
    assert len(content.split()) >= 2, "Public key format is invalid"
```

## Step 8: Test your task with the Oracle agent

Run the following command to verify your task is solved by the solution script:

```bash
harbor run -p ssh-key-pair -a oracle
```

If successful, you should see output indicating the task was completed and the reward was 1.

<Callout type="info" title="Troubleshooting">
  If the Oracle agent fails, check:

  * The solution script has execute permissions
  * The Dockerfile installs all required dependencies
  * The test script correctly writes to `/logs/verifier/reward.txt`
  * The paths in your tests match the paths in your solution
</Callout>

## Step 9 (Optional): Test with a real agent

Test your task with an actual AI agent to see if it can solve the task. For example, using Terminus with Claude:

```bash
harbor run \
  -p ssh-key-pair \
  -a terminus-2 \
  -m anthropic/claude-haiku-4-5
```

## Step 10 (Optional): Inspect the output in the viewer

Launch the viewer to browse the run's trajectory, verifier logs, and artifacts:

```bash
harbor view ./jobs
```

Open the latest job, pick the `ssh-key-pair` trial, and check the **Verifier Logs** tab to see your pytest output and the final reward. The **Trajectory** tab walks you through each step the agent took — invaluable when debugging why a real agent fails the task.

## Step 11: Contribute your task!

Congratulations! You've created your first Harbor task. Your task is now ready to be used for benchmarking AI agents!


# Windows tasks



Harbor supports running tasks inside Windows containers (e.g., `windowsservercore`, `nanoserver`) on a Windows host with Docker Desktop in Windows container mode.

## Prerequisites

* **Windows 10/11 Pro, Enterprise, or Education** (or **Windows Server 2019+**) — Windows Home does not support Hyper-V, which is required for Windows containers
* **Docker Desktop** installed and set to **Windows containers** mode (right-click system tray icon → "Switch to Windows containers...")
* **`tar.exe`** available on the host (ships with Windows 10 1803+ at `C:\Windows\System32\tar.exe`)

## How It Works

### Declaring the target OS

Tasks declare their target container OS in `task.toml` under `[environment]`:

```toml
[environment]
os = "windows"   # or "linux" (default)
```

The framework reads this field at startup and adapts every OS-conditional
behavior accordingly:

* container-side paths (POSIX vs `C:` drive),
* script discovery (`.sh` vs `.bat`),
* file transfer (native `docker cp` on Linux vs tar-over-exec on Windows —
  `docker cp` does not work with running Hyper-V isolated Windows containers,
  so files are streamed through `tar` piped over `docker exec` instead),
* command execution (`bash -c` vs `cmd /S /C`),
* container keepalive command,
* environment variables (`LOGS_DIR`).

Omitting the field is equivalent to `os = "linux"` and preserves the
behavior of every existing Linux task.

### Daemon-mode and image-OS validation

Two preflight checks run when the environment starts:

1. **Daemon mode**: Harbor calls `docker info --format "{{.OSType}}"` and
   compares the result to `[environment].os`. Mismatch → fail fast with a
   message instructing you to switch Docker Desktop to the matching
   container mode. A Windows task on a non-Windows host is also rejected.
2. **Image OS**: After build/pull but before `docker compose up`, Harbor
   runs `docker inspect --format "{{.Os}}" <image>` and compares it to
   `[environment].os`. Mismatch → fail fast indicating the actual image OS.

These checks replace the previous host-only auto-detection and surface
configuration mistakes immediately rather than mid-run.

### Container-Side Paths

Linux containers use POSIX paths (`/logs`, `/tests`, `/solution`). Windows containers use drive-prefixed paths:

| Purpose       | Linux             | Windows             |
| ------------- | ----------------- | ------------------- |
| Logs root     | `/logs`           | `C:/logs`           |
| Agent logs    | `/logs/agent`     | `C:/logs/agent`     |
| Verifier logs | `/logs/verifier`  | `C:/logs/verifier`  |
| Artifacts     | `/logs/artifacts` | `C:/logs/artifacts` |
| Tests         | `/tests`          | `C:/tests`          |
| Solution      | `/solution`       | `C:/solution`       |

These are provided by `EnvironmentPaths.for_os(...)`.

### Volume Mounts

Docker Compose volume mounts traditionally use a short-form colon-separated syntax:

```yaml
volumes:
  - C:/host/path:/container/path
```

This works fine on Linux where paths look like `/host/path:/container/path` — Docker splits on the first colon to separate source from target. But on Windows, paths include a drive letter with a colon (e.g., `C:/host/path`), so Docker sees `C` as the source, `/host/path` as the target, and `/container/path` as an invalid extra field. This causes errors like:

```
invalid volume specification: 'C:/git/harbor/jobs/.../verifier:/logs/verifier:rw'
```

To avoid this, Harbor's runtime-generated mounts override file (a per-trial `docker-compose-mounts.json` written next to the trial directory and layered on top of the base compose stack) declares every bind mount in **long-form syntax**, which avoids colon-based parsing entirely:

```yaml
services:
  main:
    volumes:
      - type: bind
        source: C:/git/harbor/jobs/<job>/trials/<trial>/verifier
        target: C:/logs/verifier
```

Each part is a separate YAML key, so colons in Windows paths are never ambiguous. This syntax works identically on both Linux and Windows.

Additionally, host paths are converted to forward-slash format (`C:/git/harbor/...` instead of `C:\git\harbor\...`) since Docker expects forward slashes regardless of the host OS.

### File Transfer (tar-over-exec)

Standard `docker cp` does not work with running Hyper-V isolated Windows containers. Instead, file transfers use `tar` piped through `docker exec`:

* **Upload**: `tar cf - -C <source> . | docker exec -i <container> tar xf - -C <target>`
* **Download**: `docker exec <container> tar cf - -C <source> . | tar xf - -C <target>`

This works with both Hyper-V (default) and process isolation modes.

The container is targeted by an explicit name (a random GUID assigned via
`container_name` in the Windows keepalive compose file), avoiding the need
to query `docker compose ps` for a container ID at runtime.

### Command Execution

Commands inside Windows containers are executed via `cmd /S /C <command>` instead of `bash -c <command>`. This correctly handles all script types.

When a task has multiple script formats, they are discovered in priority order
filtered by `[environment].os`:

| `os`              | Extensions tried (in order) |
| ----------------- | --------------------------- |
| `linux` (default) | `.sh`                       |
| `windows`         | `.bat`                      |

Within each list the first matching script wins. Because filtering is
OS-driven, a task may safely ship cross-platform scripts (e.g. both
`solve.sh` and `solve.bat`) — the right one is selected automatically.

For `.sh` scripts, `chmod +x` is executed as a separate step with `user="root"` before the script runs. This ensures scripts are executable even when the task configures a non-root agent or verifier user.

The execution shells used per extension are:

1. `.sh` → direct execution (callers run `chmod +x` as root separately)
2. `.bat` → `cmd /c <script>`

Users who need PowerShell can call it from within a `.bat` file (e.g. `powershell -ExecutionPolicy Bypass -File script.ps1`).

### Container Keepalive

Linux containers use `sh -c "sleep infinity"`. Windows containers use `cmd /C "ping -t localhost > nul"` to keep the container running. While `ping -t` has minor CPU overhead from ICMP packets, alternatives like `timeout /t -1` do not work in non-interactive Docker containers.

### Environment Variables

The `LOGS_DIR` environment variable is automatically set to `C:\logs` inside Windows containers, allowing test scripts to locate the verifier output directory via `%LOGS_DIR%` (batch).

## Writing Windows Tasks

A minimal Windows task structure:

```
my-task/
├── environment/
│   └── Dockerfile
├── solution/
│   └── solve.bat
├── tests/
│   └── test.bat
├── instruction.md
└── task.toml
```

### Dockerfile

Any Windows-based Docker image should work. Tested base images:

* `mcr.microsoft.com/windows/servercore:ltsc2022`
* `python:3.13-windowsservercore-ltsc2022`

Example:

```dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2022
WORKDIR C:\\app
```

> **Note**: Windows base images are several GB in size. The first pull will take significantly longer than typical Linux images.

### Test Scripts

Test scripts must write a reward value to `%LOGS_DIR%\verifier\reward.txt`:

**Batch (.bat):**

```batch
@echo off
echo 1 > "%LOGS_DIR%\verifier\reward.txt"
```

### Multi-Step Tasks

Windows multi-step tasks use the same per-step layout as Linux tasks, with
Batch entrypoints:

```
my-task/
├── steps/
│   └── grade/
│       ├── instruction.md
│       ├── solution/
│       │   └── solve.bat
│       └── tests/
│           └── test.bat
├── environment/
│   └── Dockerfile
└── task.toml
```

For each step, Harbor first looks for `steps/<step>/tests/test.bat`. If a
step does not provide its own test script, Harbor falls back to the shared
top-level `tests/test.bat`. When both shared and step-specific test directories
exist, shared tests are uploaded first and step-specific tests are uploaded
second, so step files can override shared helpers.

Oracle solutions follow the same rule: a step-local
`steps/<step>/solution/solve.bat` is preferred when that solution directory
exists. Otherwise, the oracle falls back to top-level `solution/solve.bat`.

### task.toml

Set `[environment].os = "windows"` to declare the task as Windows-targeted.
Everything else is standard:

```toml
version = "1.0"

[verifier]
timeout_sec = 300.0

[agent]
timeout_sec = 300.0

[environment]
os = "windows"
build_timeout_sec = 600.0
cpus = 1
memory_mb = 2048
storage_mb = 10240
```

Omitting `os` (or setting it to `"linux"`) keeps the task on Linux containers
— this is the default for back-compat.

## Sample Tasks

A sample hello-world task is included in `examples/tasks/`, demonstrating Windows batch script support. It creates a `greet` script at `C:\app\` and verifies its output:

| Sample               | Script Type                | Path                                   |
| -------------------- | -------------------------- | -------------------------------------- |
| hello-world-bat      | `.bat` (Batch)             | `examples/tasks/hello-world-bat/`      |
| hello-multi-step-bat | `.bat` (Batch, multi-step) | `examples/tasks/hello-multi-step-bat/` |

Run it with the oracle agent:

```bash
uv run harbor run -p examples/tasks/hello-world-bat -a oracle --debug
```

It has been tested and produces **Mean: 1.000** (full score).

## Limitations

* **Windows host only**: Windows containers require a Windows host with Docker Desktop. They cannot run on Linux hosts.
* **Hyper-V isolation**: The default isolation mode is Hyper-V. Process isolation is possible but may trigger Windows Defender Controlled Folder Access blocks.
* **No GPU support**: Windows containers in Harbor do not support GPU allocation.
* **No internet isolation**: The `network_mode: none` compose override has not been tested with Windows containers.
* **Performance**: Windows base images are several GB (vs. tens of MB for Linux), resulting in longer initial pulls and slower first-time builds due to image size and Windows-specific tooling installation.
* **Only Docker environment**: Windows tasks are supported only in the local Docker environment, not in cloud providers (Daytona, Modal, E2B, etc.).
* **Agent compatibility**: Only agents with `capabilities.windows = True` can run Windows tasks. Currently this is limited to `oracle` and `nop`. All other installed agents (terminus, claude-code, codex, aider, etc.) rely on Linux-only tools (`bash`, `apt-get`, `tmux`) and will be rejected with a clear error before setup. Custom agents must declare `capabilities = AgentCapabilities(windows=True)` to opt in.
* **CRLF handling**: No automatic CRLF→LF conversion is performed for Windows containers (it is only needed for Linux containers running on Windows hosts).


# Built-in Criteria



import { Callout } from "fumadocs-ui/components/callout";

Rewardkit ships with built-in criteria for common rubrics. Use them from any Python file in your tests directory:

```python
import rewardkit as rk

rk.file_exists("output.txt")
rk.command_succeeds("python main.py", weight=2.0)
```

All criteria accept optional `weight` (default 1.0) and `isolated` (default false) parameters in addition to the ones listed below.

## File criteria

| Criterion             | Parameters       | Description                                                               |
| --------------------- | ---------------- | ------------------------------------------------------------------------- |
| `file_exists`         | `path`           | File exists in workspace                                                  |
| `file_not_exists`     | `path`           | File does not exist                                                       |
| `file_contains`       | `path, text`     | File contains a substring                                                 |
| `file_contains_regex` | `path, pattern`  | File content matches a regex pattern                                      |
| `file_matches`        | `path, expected` | File content equals expected text (whitespace-stripped)                   |
| `files_equal`         | `path1, path2`   | Two files have identical content                                          |
| `diff_ratio`          | `path, expected` | Similarity ratio between file content and expected text (returns 0.0–1.0) |

## Command criteria

| Criterion                      | Parameters                      | Description                               |
| ------------------------------ | ------------------------------- | ----------------------------------------- |
| `command_succeeds`             | `cmd, cwd?, timeout?`           | Command exits with code 0                 |
| `command_output_contains`      | `cmd, text, cwd?, timeout?`     | Command stdout contains text              |
| `command_output_matches`       | `cmd, expected, cwd?, timeout?` | Command stdout equals expected (stripped) |
| `command_output_matches_regex` | `cmd, pattern, cwd?, timeout?`  | Command stdout matches a regex            |

Default timeout is 30 seconds. The `cwd` parameter is relative to the workspace.

## Data format criteria

| Criterion             | Parameters                     | Description                                 |
| --------------------- | ------------------------------ | ------------------------------------------- |
| `json_key_equals`     | `path, key, expected`          | Top-level JSON key equals a value           |
| `json_path_equals`    | `path, json_path, expected`    | Dot-separated path into JSON equals a value |
| `csv_cell_equals`     | `path, row, col, expected`     | CSV cell at row/col equals a value          |
| `xlsx_cell_equals`    | `path, cell, expected, sheet?` | Excel cell equals a value                   |
| `sqlite_query_equals` | `db_path, query, expected`     | SQL query result equals a value             |

<Callout type="info">
  `xlsx_cell_equals` requires the `documents` extra: `uv add
    harbor-rewardkit[documents]`
</Callout>

<Callout type="info">
  For `csv_cell_equals`, row numbering depends on the column type. When `col` is
  an **integer**, a raw CSV reader is used and row 0 is the header row. When
  `col` is a **string** (column name), row 0 is the first data row after the
  header.
</Callout>

## HTTP criteria

| Criterion                | Parameters               | Description                                              |
| ------------------------ | ------------------------ | -------------------------------------------------------- |
| `http_status_equals`     | `url, status?, timeout?` | HTTP response has the expected status code (default 200) |
| `http_response_contains` | `url, text, timeout?`    | HTTP response body contains text                         |

## Image criteria

| Criterion           | Parameters            | Description                                    |
| ------------------- | --------------------- | ---------------------------------------------- |
| `image_similarity`  | `path1, path2`        | Pixel-level similarity ratio (returns 0.0–1.0) |
| `image_size_equals` | `path, width, height` | Image has the expected dimensions              |

<Callout type="info">
  Image criteria require the `image` extra: `uv add harbor-rewardkit[image]`
</Callout>

## Trajectory criteria

These criteria inspect the agent's ATIF trajectory file (default path: `/logs/agent/trajectory.json`).

| Criterion                  | Parameters                     | Description                                                                                      |
| -------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------ |
| `trajectory_tool_used`     | `tool_name, min_count?, path?` | Agent used a specific tool at least `min_count` times (default 1)                                |
| `trajectory_tool_not_used` | `tool_name, path?`             | Agent did not use a specific tool                                                                |
| `trajectory_turn_count`    | `max_turns, path?`             | Penalizes exceeding a turn budget — returns 1.0 at `max_turns`, linearly decays to 0.0 at double |

## Optional extras

| Extra       | Criteria                                | Install                              |
| ----------- | --------------------------------------- | ------------------------------------ |
| `documents` | `xlsx_cell_equals`                      | `uv add harbor-rewardkit[documents]` |
| `image`     | `image_similarity`, `image_size_equals` | `uv add harbor-rewardkit[image]`     |
| `all`       | All of the above                        | `uv add harbor-rewardkit[all]`       |


# Reward Kit



import { Callout } from "fumadocs-ui/components/callout";
import { File, Folder, Files } from "fumadocs-ui/components/files";

<Callout type="info" title="Agent skill available">
  Want your coding agent to help design verifiers with Reward Kit? Install the `rewardkit` skill:

  ```bash
  npx skills add harbor-framework/harbor --skill rewardkit
  ```
</Callout>

Rewardkit let's you design verifiers against an agent's workspace and trajectory. It writes scores to a JSON file and has zero external dependencies. Verifiers are defined using directory structure: Each folder maps to a reward, making them easy to read, share, and extend. All criteria are evaluated in parallel by default. You can define criteria in two ways:

* **Programmatic:** Python functions that check files, run commands, or inspect outputs
* **Judge-based:** LLM- or Agent-as-a-Judge evaluation configured via reusable TOML files

<Callout type="info" title="Standalone package">
  Rewardkit is an independent Python package. You can use it with Harbor or on
  its own.
</Callout>

## Installation

```bash
uv tool install harbor-rewardkit
```

For criteria or judges that read images or common document files (PDF, DOCX, PPTX, XLSX), install the extras:

```bash
uv tool install harbor-rewardkit[all]
```

## Using with Harbor

Harbor copies your task's `tests/` directory into the container at `/tests/` and runs `test.sh`. Put your criteria files alongside `test.sh`:

<Files>
  <Folder name="tests" defaultOpen>
    <File name="checks.py" />

    <File name="judge.toml" />

    <File name="test.sh" />
  </Folder>
</Files>

```bash title="tests/test.sh"
#!/bin/bash
uvx --with harbor-rewardkit@0.1 rewardkit /tests
```

This discovers all criteria in `/tests/`, runs them against the workspace at `/app`, and writes the result to `/logs/verifier/reward.json`:

```json title="/logs/verifier/reward.json"
{ "reward": 0.75 }
```

All defaults match Harbor's conventions.

If your judge criteria need API keys, pass them through `[verifier.env]` in `task.toml`:

```toml title="task.toml"
[verifier]
timeout_sec = 300.0

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
```

There's a complete [working example](https://github.com/harbor-framework/harbor/tree/main/examples/tasks/reward-kit-example) you can use as a starting point.

To swap providers without editing the rubric, see [Provider routing](#provider-routing).

## Built-in criteria

Many common checks we saw being used in popular benchmarks are already built in. Just call them from a Python file in your tests directory:

```python title="tests/check.py"
import rewardkit as rk

rk.file_exists("output.txt")
rk.file_contains("output.txt", "hello")
rk.command_succeeds("python main.py")
```

There are 20+ built-in criteria covering frequently used criteria for grading files, commands, JSON, CSV, HTTP, images, and agent trajectories. See [Built-in Criteria](/docs/rewardkit/built-in-criteria) for the full list.

## Custom criteria

When you need logic specific to your task, define your own with the `@criterion` decorator. The first parameter is always `workspace: Path`, and the function returns a `bool` or `float`. Zero-parameter criteria are registered automatically:

```python title="tests/check.py"
import rewardkit as rk
from pathlib import Path
from rewardkit import criterion

@criterion
def has_valid_output(workspace: Path) -> bool:
    output = (workspace / "output.txt").read_text()
    return len(output.splitlines()) >= 10
```

Criteria can also take custom arguments. These need to be called explicitly through `rk` to register:

```python title="tests/check.py"
@criterion(description="output has at least {n} lines")
def has_n_lines(workspace: Path, n: int) -> bool:
    output = (workspace / "output.txt").read_text()
    return len(output.splitlines()) >= n

rk.has_n_lines(10, weight=2.0)
rk.has_n_lines(50, weight=1.0)
```

## Judge criteria

Some things are hard to check with code — code quality, edge case handling, readability. For these, you can have an LLM evaluate the workspace by creating a TOML file:

```toml title="tests/quality.toml"
[judge]
judge = "anthropic/claude-sonnet-5"
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
```

Both `.py` and `.toml` files can coexist in the same directory — their scores are averaged together into one reward.

See [Judge Criteria](/docs/rewardkit/judge-criteria) for the full TOML configuration reference including agent judges and trajectory evaluation.

## Scoring config for a dimension

All `.py` criteria in a directory form a single group, combined with any judges in that directory by a weighted average. By default the group has weight 1.0 and averages its criteria. To change either, add a `reward.toml` to the directory:

```toml title="tests/structure/reward.toml"
weight = 2.0  # this group's share of the dimension (default 1.0)

[scoring]
aggregation = "all-pass"  # gate the group instead of averaging it
threshold = 0.5           # only for threshold
```

`weight` decides how much the deterministic checks count relative to a judge in the same directory. It stays local to that directory; parent aggregations treat each child directory as weight 1.0 unless their own `weights` map overrides it. `[scoring]` takes the same `aggregation` values as a judge TOML: `weighted-mean` (default), `weighted-sum`, `all-pass`, `any-pass`, `threshold`, `required-pass`. With `all-pass`, one failing check collapses the whole group to `0.0`.

Unknown keys raise, so typos fail loudly rather than being ignored. A `reward.toml` that declares `weight` or `[scoring]` in a directory with no `.py` files also raises, since there would be nothing to configure.

Directories can be nested to create independently scored groups. Each directory combines its local `.py` bucket, local judge TOMLs, and immediate child directories. A nested directory exports one score to its parent:

<Files>
  <Folder name="tests" defaultOpen>
    <File name="reward.toml" />

    <Folder name="correctness" defaultOpen>
      <File name="reward.toml" />

      <Folder name="files" defaultOpen>
        <File name="checks.py" />
      </Folder>

      <Folder name="behavior" defaultOpen>
        <File name="checks.py" />
      </Folder>
    </Folder>

    <Folder name="quality" defaultOpen>
      <File name="judge.toml" />
    </Folder>
  </Folder>
</Files>

A non-root directory may declare one `[[reward]]` table. It aggregates every immediate scoring input; omit `name` because the directory name identifies the score:

```toml title="tests/correctness/reward.toml"
[[reward]]
aggregation = "weighted-mean"
weights = { files = 2.0, behavior = 1.0 }
```

Child directories have weight `1.0` by default. The optional inline `weights` map overrides weights at this aggregation boundary. Use child directory names, judge TOML filename stems, and `$checks` for the local `.py` bucket. Local bucket and judge weights remain their defaults when not overridden, but a child directory's internal weights never leak into its parent.

Without a local `[[reward]]`, a directory uses `weighted-mean`. Membership is implicit: all immediate checks, judges, and child directories participate.

## Multi-reward tasks

When you want separate scores for different dimensions (correctness, structure, quality), organize criteria into subdirectories. Each subdirectory becomes its own reward:

<Files>
  <Folder name="tests" defaultOpen>
    <File name="test.sh" />

    <Folder name="correctness" defaultOpen>
      <File name="check.py" />
    </Folder>

    <Folder name="structure" defaultOpen>
      <File name="files_exist.py" />
    </Folder>

    <Folder name="quality" defaultOpen>
      <File name="quality.toml" />
    </Folder>
  </Folder>
</Files>

This produces separate scores:

```json title="/logs/verifier/reward.json"
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6 }
```

A judge TOML can also remain directly under `tests/` while dimension directories
exist. It becomes a top-level reward named after the TOML filename stem and can
be referenced by that name from a root `reward.toml` aggregation.

To reuse custom criteria across subdirectories, define them in a root-level file with `shared=True`:

```python title="tests/criteria.py"
from pathlib import Path
from rewardkit import criterion

@criterion(shared=True)
def word_count_correct(workspace: Path) -> float:
    # test logic ...
    return score
```

```python title="tests/correctness/check.py"
import rewardkit as rk

rk.word_count_correct(weight=3.0)
```

## Weights and scoring

Every criterion accepts an optional `weight` (default 1.0):

```python
rk.file_exists("output.txt", weight=3.0)
rk.file_exists("readme.md", weight=1.0)
```

All criteria in a reward directory are aggregated using a weighted average. Judge criteria are first aggregated per judge — you can control this with the `[scoring]` section in the TOML file:

```toml title="tests/quality.toml"
[judge]
judge = "anthropic/claude-sonnet-5"
files = ["/app/main.py"]
weight = 2.0  # weight of this judge in the overall reward (default 1.0)

[[criterion]]
description = "Is the code correct?"
type = "binary"
weight = 2.0

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5

[scoring]
aggregation = "all-pass"  # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
```

`required-pass` requires every non-`optional` criterion to pass (criteria marked `optional = true` don't gate). See [Judge criteria](/docs/rewardkit/judge-criteria) for details.

`weighted-sum` computes `sum(value × weight)` without normalization. It is the
only aggregation that accepts negative weights, and its result may fall outside
`[0, 1]`.

## Isolation

Some criteria modify the workspace by running commands that make edits, create files, etc. To prevent one criterion from affecting another, you can run it in isolation. This uses overlayfs to mount the workspace as read-only with writes going to a temporary layer that gets discarded after the criterion completes.

For programmatic criteria, pass `isolated=True`:

```python
rk.command_succeeds("python main.py", isolated=True)
```

For agent judges, set `isolated = true` in the `[judge]` section:

```toml
[judge]
judge = "claude-code"
isolated = true
```

## Output

Rewardkit writes two files side by side:

* `reward.json` — per-reward scores, e.g. `{"correctness": 0.75, "quality": 0.9}`
* `reward-details.json` — per-criterion results including individual scores, judge reasoning, and any errors. `harbor view` renders this under **Verifier Logs → Rewards** as a collapsible tree.

Agent judge details also include normalized token `usage` and paths to copied native JSONL `judge_logs`. Failed Codex attempts add text logs containing their stdout and stderr.

### Aggregating dimensions

To add aggregated scores on top of the per-dimension keys, create a root-level `tests/reward.toml` with one or more `[[reward]]` tables. Each table adds one key to `reward.json`, aggregating the per-dimension scores with the same modes used within a judge (`weighted-mean`, `weighted-sum`, `all-pass`, `any-pass`, `threshold`):

```toml title="tests/reward.toml"
[[reward]]
name = "reward"
aggregation = "all-pass"

[[reward]]
name = "soft_score"
aggregation = "weighted-mean"
weights = { correctness = 2.0, structure = 1.0, quality = 1.0 }
```

```json title="reward.json"
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6, "reward": 0.0, "soft_score": 0.78 }
```

The per-dimension scores are kept; the aggregated keys are added alongside them. Top-level dimensions have equal weight by default, and each root aggregation can override those weights with its own short inline map. `reward-details.json` keeps the full recursive breakdown. Set the name of an aggregated score to `reward` to make it the main score that is parsed when scoring the task using Harbor.

## Comparing verifiers

You can pass multiple test directories to compare different verifier designs side by side:

```bash
rewardkit /tests/v1 /tests/v2
```

This runs each directory independently and prints a comparison table for overlapping reward names:

```
Comparison:
------------------------------------------
reward          v1      v2    diff
------------------------------------------
correctness  0.7500  0.9000  -0.1500
structure    1.0000  1.0000  +0.0000
------------------------------------------
```

The combined output is written to `reward.json` with namespaced keys like `"v1/correctness"` and `"v2/correctness"`.

## CLI

```bash
rewardkit <tests_dirs...> \
    --workspace /app \
    --output /logs/verifier/reward.json \
    --max-concurrent-programmatic 8 \
    --max-concurrent-llm 8 \
    --max-concurrent-agent 2
```

All flags are optional with sensible defaults. Passing multiple test directories runs each independently and prints a comparison table.

## Provider routing

Judges call LiteLLM, which reads credentials from host environment variables. A few flags help you avoid pinning a rubric to one provider:

* `--je KEY=VALUE` sets env vars for the run (repeatable). Same shape as Harbor's `--ve`.
* `--judge MODEL_OR_AGENT` overwrites the rubric's `[judge].judge` field. This can also be done by setting `REWARDKIT_JUDGE` in the environment, so Harbor users can pass `--ve REWARDKIT_JUDGE=...` to do the same thing.
* `--model MODEL` overwrites the rubric's `[judge].model` field when the judge is an agent (e.g. `claude-code`, `codex`). Equivalent to setting `REWARDKIT_MODEL`; Harbor users can pass `--ve REWARDKIT_MODEL=...`.

```bash
rewardkit /tests \
  --judge bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0 \
  --je AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  --je AWS_REGION_NAME=us-east-1

# Agent judge with an overridden model
rewardkit /tests \
  --judge claude-code \
  --model anthropic/claude-sonnet-5
```

The [LiteLLM provider docs](https://docs.litellm.ai/docs/providers) list the env var configurations for each provider.

### Subscription auth

For Anthropic LLM judge models, a Claude subscription token is used when it is the only Anthropic credential present. Create one with `claude setup-token` and set `CLAUDE_CODE_OAUTH_TOKEN`. When both `ANTHROPIC_API_KEY` and the token are set, the API key has priority; set `REWARDKIT_FORCE_SUBSCRIPTION=1` to force the subscription token instead.

For the `codex` agent judge, set `OPENAI_API_KEY` or pass the contents of a ChatGPT-authenticated Codex `auth.json` file as `CODEX_AUTH_JSON`. `OPENAI_API_KEY` has priority; set `REWARDKIT_FORCE_SUBSCRIPTION=1` to require subscription auth. RewardKit downloads and caches its pinned Codex runtime automatically on first use.

## Python API

```python
import rewardkit as rk

# Run and get scores
scores = rk.run("/tests", workspace="/app")

# Inspect discovered rewards without running them
rewards = rk.discover("/tests", workspace="/app")
```


# Judge Criteria



import { Callout } from "fumadocs-ui/components/callout";
import { TypeTable } from "fumadocs-ui/components/type-table";

Judge criteria let you use an LLM or agent CLI to evaluate work, configured via TOML files. This makes it trivial to reuse and share rubrics between tasks.

## LLM judge

```toml title="tests/quality.toml"
[judge]
judge = "anthropic/claude-sonnet-5"
files = ["/app/main.py", "/app/utils.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
weight = 2.0

[[criterion]]
description = "Rate the test coverage on a scale from 0 to 100"
type = "numeric"
min = 0
max = 100
```

The `judge` field accepts any [LiteLLM model string](https://docs.litellm.ai/docs/providers). It can be overridden at invocation time without editing the rubric. See [Provider routing](/docs/rewardkit#provider-routing).

## Agent judge

Agent judges can explore the filesystem and run commands. Codex agent judges
download their pinned runtime automatically on first use.

```toml title="tests/review.toml"
[judge]
judge = "claude-code"
model = "anthropic/claude-sonnet-5"
isolated = true

[[criterion]]
description = "Does the solution handle edge cases?"
type = "binary"
```

<Callout type="info">
  Agent judges are slower and more expensive but can interact with the workspace
  directly.
</Callout>

### MCP servers

Each `[[judge.mcp_servers]]` entry matches a Harbor task's `[[environment.mcp_servers]]`. Per-server `allowed_tools` lists the tools the judge may call; omit it to allow all of the server's tools. Codex does not support `sse` servers.

```toml title="tests/review.toml"
[judge]
judge = "claude-code"

[[judge.mcp_servers]]
name = "playwright"
transport = "stdio"
command = "npx"
args = ["@playwright/mcp@latest", "--headless", "--isolated"]
allowed_tools = ["navigate", "click"]

[[criterion]]
description = "Does the rendered page match the spec?"
type = "binary"
```

## Individual mode

Set `mode = "individual"` to grade one criterion per call instead of batching them all into one. LLM judges make one request per criterion; agent judges run one turn per criterion, sequentially. For LLM judges, each criterion can also scope its own `files` (below):

```toml title="tests/quality.toml"
[judge]
judge = "anthropic/claude-sonnet-5"
mode = "individual"

[[criterion]]
description = "Is the analysis correct?"
files = ["/app/analysis.pdf"]

[[criterion]]
description = "Is the spreadsheet well-structured?"
files = ["/app/data.xlsx"]
```

Criteria without `files` fall back to `[judge].files`.

If a judge call times out, RewardKit records the affected criteria as `0.0`
with an error and warning in `reward-details.json`.

## Configuration reference

Judge TOMLs are validated when the tests directory is scanned, before any judge runs. Unknown keys and unknown values raise, so a typo fails loudly instead of quietly changing how the task is scored.

### `[judge]` section

<TypeTable
  type={{
  judge: {
    description:
      'LiteLLM model name (e.g. "anthropic/claude-sonnet-5") or agent name ("claude-code", "codex", "fx").',
    type: "string",
    default: '"anthropic/claude-sonnet-5"',
  },
  model: {
    description: "For agent judges, the LLM the agent should use.",
    type: "string | null",
    default: "null",
  },
  files: {
    description: "Workspace file paths to include in the judge prompt.",
    type: "list[string]",
    default: "[]",
  },
  mode: {
    description:
      'Whether to evaluate all criteria in one call (`batched`) or one call per criterion (`individual`).',
    type: '"batched" | "individual"',
    default: '"batched"',
  },
  timeout: {
    description: "Seconds to wait for the judge response.",
    type: "integer",
    default: "300",
  },
  reasoning_effort: {
    description:
      "Effort level for LLM judges. Accepts the levels LiteLLM supports; a given model may not support all of them.",
    type: '"none" | "minimal" | "low" | "medium" | "high" | "xhigh"',
    default: '"medium"',
  },
  isolated: {
    description:
      "For agent judges, mount the workspace as read-only via overlayfs.",
    type: "boolean",
    default: "false",
  },
  cwd: {
    description: "For agent judges, the working directory the agent runs in.",
    type: "string | null",
    default: "null",
  },
  "mcp_servers": {
    description:
      "For agent judges, MCP servers to configure before running. Each entry matches a Harbor task's `[[environment.mcp_servers]]`, plus a per-server `allowed_tools` allowlist. Codex does not support `sse` servers.",
    type: "list[table]",
    default: "[]",
  },
  reference: {
    description: "Path to a reference solution file for comparison.",
    type: "string | null",
    default: "null",
  },
  "atif-trajectory": {
    description: "Path to an ATIF trajectory JSON to include in the prompt.",
    type: "string | null",
    default: "null",
  },
  weight: {
    description:
      "Weight of this judge's score when averaging with other criteria in the same directory.",
    type: "number",
    default: "1.0",
  },
  prompt_template: {
    description:
      "Custom prompt template (.md or .txt). Must contain a {criteria} placeholder.",
    type: "string | null",
    default: "null",
  },
}}
/>

### `[[criterion]]` entries

<TypeTable
  type={{
  description: {
    description: "What to evaluate — this text is sent to the judge.",
    type: "string",
  },
  type: {
    description: "Output format.",
    type: '"binary" | "likert" | "numeric"',
    default: '"binary"',
  },
  name: {
    description:
      "Identifier for this criterion. Auto-generated from description if omitted.",
    type: "string | null",
    default: "null",
  },
  id: {
    description:
      'Stable rubric identifier carried through to `reward-details.json` for provenance (e.g. "1.1", "2.3"). Independent of `name`; survives rewording the description.',
    type: "string | null",
    default: "null",
  },
  points: {
    description: "Scale size for likert type.",
    type: "integer",
    default: "5",
  },
  min: {
    description: "Minimum value for numeric type.",
    type: "number",
    default: "0.0",
  },
  max: {
    description: "Maximum value for numeric type.",
    type: "number",
    default: "1.0",
  },
  weight: {
    description:
      'Importance multiplier for score aggregation. Negative weights support penalty criteria and require `aggregation = "weighted-sum"`.',
    type: "number",
    default: "1.0",
  },
  files: {
    description:
      'Files scoped to this criterion. Requires `[judge].mode = "individual"`. Falls back to `[judge].files` when omitted.',
    type: "list[string]",
    default: "[]",
  },
  negate: {
    description:
      "Invert the normalized score. Use for a criterion describing behavior the answer should NOT exhibit (see below).",
    type: "boolean",
    default: "false",
  },
  optional: {
    description:
      'Exempt this criterion from gating under `aggregation = "required-pass"`.',
    type: "boolean",
    default: "false",
  },
}}
/>

### `[scoring]` section

Controls how the judge's criteria are aggregated into a single score. This only affects criteria within this TOML file — it does not change how programmatic and judge scores are combined across the directory.

```toml
[scoring]
aggregation = "all-pass"  # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
threshold = 0.7           # only used with "threshold" aggregation
```

`required-pass` returns 1.0 only when every non-`optional` criterion passes (`value > 0`); `optional` criteria never gate. With no non-optional criteria it warns and scores 0.0.

`weighted-sum` computes `sum(value × weight)` without normalization. It is the
only aggregation that accepts negative weights, and its result may fall outside
`[0, 1]`.

## Score normalization

* **Binary**: yes/true/1 → 1.0, anything else → 0.0
* **Likert**: normalized to \[0, 1] as `(raw - 1) / (points - 1)`
* **Numeric**: normalized to \[0, 1] as `(raw - min) / (max - min)`

## Negated criteria

Set `negate = true` for a criterion describing behavior the answer should **not** exhibit. The judge scores presence as usual, then the score is inverted (`value → 1 - value`): present → 0.0, absent → 1.0. The raw judge answer is kept in `reward-details.json` so the flip is auditable.

```toml
[[criterion]]
description = "States there is no task execution history tracking in the database"
type = "binary"
negate = true   # the answer should NOT make this (false) claim
```

## Trajectory evaluation

To evaluate the agent's process rather than just its output, point the judge at the trajectory file:

```toml
[judge]
judge = "anthropic/claude-sonnet-5"
atif-trajectory = "/logs/agent/trajectory.json"
files = ["/app/main.py"]

[[criterion]]
description = "Did the agent take an efficient approach?"
type = "likert"
points = 5
```

The trajectory content is truncated proportionally to fit within the model's context window while preserving all steps.

## Custom prompt templates

You can provide your own prompt instead of the built-in one:

```toml
[judge]
judge = "anthropic/claude-sonnet-5"
prompt_template = "my_prompt.md"
```

The template must contain a `{criteria}` placeholder where criterion descriptions get injected.


# Motivation & Design



## Motivation

When creating new tasks for Harbor, writing the verifier is often the most tedious part. So we looked at verifier implementations across popular benchmarks and found that they often are complicated scripts that are hard to read, extend, and reuse across tasks. This often leads to copy-and-pasting of boilerplace code between tasks causing subtle errors in verifiers during evals.

**Reward Kit** solves this. It packages common patterns into reusable components that can be shared and reused across tasks. Reward Kit gets rid of boilerplate code by enabling LLM/agent-as-a-judge evaluation and partial credit. Because independent reward components are defined through the directory structure, verifiers are easy to understand at a glance even without reading the code. Evaluating multiple criteria in parallel and in isolation is natively supported in Reward Kit.

## Design principles

* **Simplicity:** Verifiers are defined by directory structure, making them easy to read at a glance. Common checks are one-liners. Custom logic is a decorated function. Judges are reusable TOML files.
* **Reuse and sharing:** Reward criteria are plain files in a directory. Share them across tasks and version in git.
* **Zero boilerplate:** 20+ built-in criteria for files, commands, JSON, CSV, HTTP, images, and trajectories. LLM- and Agent-as-a-Judge is supported natively.
* **Isolation:** Criteria can be run in isolated filesystem snapshots so they can't interfere with each other or corrupt the agent's workspace.


# Jobs



Jobs are run results. Upload a job to [Harbor Hub](https://hub.harborframework.com/jobs) to get a shareable link, then download the full job or a single trial by ID. Run `harbor auth login` first.

## Upload an existing job

```bash
harbor upload jobs/my-job
harbor upload jobs/my-job --public
harbor upload jobs/my-job --private
harbor upload jobs/my-job --share-org my-org --share-user alice
```

New uploads are private unless you pass `--public`. Re-uploading is idempotent: without a visibility flag, Harbor keeps the server-side visibility unchanged; with `--public` or `--private`, it updates visibility.

Useful flags:

* `-c, --concurrency <n>`: max concurrent trial uploads.
* `--share-org <org>`: share with an organization. Repeatable.
* `--share-user <username>`: share with a GitHub user. Repeatable.
* `-y, --yes`: confirm shares with orgs you are not a member of.

## Upload while running

```bash
harbor run -d "my-org/my-dataset@latest" -a "<agent>" -m "<model>" --upload
harbor run -d "my-org/my-dataset@latest" -a "<agent>" -m "<model>" --upload --public
harbor run -d "my-org/my-dataset@latest" -a "<agent>" -m "<model>" --upload --share-org my-org
```

`--upload` streams trials as they finish and finalizes the job archive at the end. `--public`, `--private`, `--share-org`, and `--share-user` require `--upload`.

If a run finishes but upload does not, rerun:

```bash
harbor upload <job-dir>
```

## Resume with upload

```bash
harbor job resume -p jobs/my-job --upload
harbor job resume -p jobs/my-job --upload --private --share-user alice
```

This fills in missing trials and finalizes a partially uploaded job.

## Share an uploaded job

Find job IDs from the job page in Harbor Hub.

```bash
harbor job share <job-id> --org my-org
harbor job share <job-id> --user alice --user bob
```

Private jobs are visible to the owner and explicit shares. Public jobs are visible to everyone. Shares add access; they do not replace public/private visibility.

## Download results

Use job and trial download commands for uploaded results. Top-level `harbor download` is for tasks and datasets.
Job and trial IDs are easy to find from the matching job and trial pages in Harbor Hub.

```bash
harbor job download <job-id>
harbor trial download <trial-id>
```

Defaults:

* Jobs download to `./jobs/<job-name>`.
* Trials download to `./trials/<trial-name>`.
* Use `-o, --output-dir <dir>` to choose a parent directory.
* Use `--overwrite` to replace an existing local job or trial directory.


# Tasks and Datasets



import { HARBOR_REGISTRY_TASKS_URL, HARBOR_REGISTRY_DATASETS_URL } from "@/lib/harbor-registry";

Tasks and datasets are shared as registry packages: `org/name@tag`.

## Publish

Publish local tasks and dataset manifests before sharing them:

* [Publishing tasks](/docs/tasks/publishing)
* [Publishing a dataset](/docs/datasets/publishing)

## Visibility

Use `--public` or `--private` when publishing. Private packages are visible to the publishing org. Public packages are visible to everyone.

Update visibility later:

```bash
harbor task visibility "my-org/my-task" --public
harbor dataset visibility "my-org/my-dataset" --private
```

Organization grants are separate from visibility: they share a package with
selected organizations without making it public.

## Organization access

```bash
harbor task access "my-org/my-task"
harbor dataset access "my-org/my-dataset"
```

Share or unshare with one or more organizations. Repeat `--org` for multiple
recipients:

```bash
harbor task share "my-org/my-task" --org research --org evaluation
harbor dataset unshare "my-org/my-dataset" --org evaluation
```

`unshare` removes only the direct grant. Public or derived grants can still
provide access.

```bash
harbor dataset share "my-org/my-dataset" --org research --dry-run
```

Useful flags:

* `--org <org>`: recipient organization. Repeatable.
* `--dry-run`: preview without changing access.
* `-y, --yes`: apply without prompting. Required for non-interactive mutations, including `--json`.
* `--json`: machine-readable output.

## Use a shared package

Share a package by reference in commands that consume package entries:

```bash
harbor run -d "my-org/my-dataset@v1.0" -m "<model>" -a "<agent>"
```

Download a task or dataset locally:

```bash
harbor download "my-org/my-task@latest"
harbor download "my-org/my-dataset@latest"
```

By default, `harbor download` exports to the current directory. Use `--output-dir <path>` to choose a location, or `--cache` to store packages under `~/.cache/harbor/tasks`.

If a dataset references tasks you cannot access, Harbor reports the exact count and stops before downloading any task or dataset files.

## Browse

* <a href={HARBOR_REGISTRY_TASKS_URL}>
    Registry tasks
  </a>
* <a href={HARBOR_REGISTRY_DATASETS_URL}>
    Registry datasets
  </a>


# Harbor Hub API Key















To authenticate any command via the [CLI](/docs/hosted-harbor/cli) or
[API](/docs/hosted-harbor/api), you must first create a Harbor API key.

## CLI

If using the CLI, an API key is minted automatically when you successfully run `harbor auth login`.
The key is stored in `~/.harbor/credentials.json` and revoked upon `harbor auth logout`.
You can also authenticate the CLI via the `HARBOR_API_KEY` environment variable. The next
section demonstrates creating an API key via the web UI.

## Web UI

To mint an API key on the web UI, first go to [The Hub](https://hub.harborframework.com)

Click on your profile in the top right corner:

<div className="dark:hidden">
    <img alt="The Harbor Hub home page, listing published datasets" src={__img0} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="The Harbor Hub home page, listing published datasets" src={__img1} placeholder="blur" />
</div>

Then click on your settings tab:

<div className="dark:hidden">
    <img alt="Profile Page" src={__img2} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Profile Page" src={__img3} placeholder="blur" />
</div>

Then scroll down to the API keys section and click `Create API key`. You will give the key a name
and an expiration date. Store this key somewhere safe. It has full access to your Harbor Hub account.

<div className="dark:hidden">
    <img alt="Profile Page" src={__img4} placeholder="blur" />
</div>

<div className="hidden dark:block">
    <img alt="Profile Page" src={__img5} placeholder="blur" />
</div>


# Hub



The [Harbor Hub](https://hub.harborframework.com) can be used to store, share, and host jobs, datasets, and tasks.
When a job is uploaded, the hub provides a comprehensive overview page allowing you to examine costs, token consumption, and performance across many dimensions.

The harbor CLI ships with commands analogous to the viewers provided on the hub website.
Please ensure you have either logged in with `harbor auth login` or minted an API key
(see [Creating a Harbor Hub API Key](/docs/hub/api-key))

All of the harbor hub viewer commands share the same entrypoint `harbor hub`.

## Listing Jobs

Using the command `harbor hub job list` you can print out a list of all jobs visible to your user on the hub.

### Flags

The list command supports numerous flags that offer fine-grained observability into your jobs.

* `--scope my | shared | all` alters the visibility scope of the returned jobs, defaulting to `my` for jobs owned by your user
* `--search TEXT` returns only jobs whose names contain a matching substring
* `--agent NAME NAME_2 ...` filters jobs by agent name. Accepts multiple entries (e.g. --agent codex claude-code) will return jobs that utilize either codex or claude-code
* `--provider NAME NAME_2 ...` filters jobs by model provider. Accepts multiple entries
* `--model NAME NAME_2 ...` filters jobs by model. Accepts multiple entries

## Job Overview

Use the command `harbor hub job show JOB_ID JOB_ID_2 ...` to print an overview similar to the one provided on the hub.

This viewer displays the following information

* Number of trials
* Number of errors
* Number of retries
* Average return on metrics
* Cost (in USD)
* Token Usage

## Per-Task Job Breakdown

Use the command `harbor hub job tasks JOB_ID` to see a per-task breakdown of a job.

### Flags

* `--search TEXT` returns only tasks whose names contain a matching substring
* `--agent / --provider / --model` same as above

## Trials Overview

Use the command `harbor hub job trials JOB_ID JOB_ID_2 ...` to list trials across one or more jobs.
In interactive terminals, you can switch pages to audit all trials without running a second command.

### Flags

* `--search TEXT` filter by text matching trial names
* `--agent / --provider / --model` same as above
* `--limit N` page size (defaults to 100 here)
* `--failed-only` only trials that errored/failed
* `--include-retries` include retry history, not just the latest execution
* `--sort-by COL` started\_at | task\_name | name| error\_type
* `--sort-order asc|desc`

## Comparing 2 or More Jobs

Use command `harbor hub job compare JOB_ID JOB_ID_2 ...` to get the side-by-side grid for performance on tasks

## Job Visibility

Use command `harbor hub job shares JOB_ID` to see who a job is shared with. This returns orgs and users

## Renaming Jobs

Use `harbor hub job rename JOB_ID "NEW NAME"` to change the name displayed on
the Hub. This does not change the job ID, artifact identity, or submitted
configuration. Add `--json` to print the rename RPC response.

## Transferring Job Ownership

Use `harbor hub job transfer JOB_ID ORGANIZATION` to transfer a completed job
and all of its trials to another organization you own. Job and trial IDs do not
change. The destination must already have access to every execution dependency
used by the job.

The command prompts for confirmation; pass `--yes` / `-y` when scripting. By
default, the source organization keeps shared read access and job-specific
secrets are permanently removed. Use `--drop-source-access` to remove the
source grant or `--keep-job-credentials` to retain and reauthorize secrets you
originally supplied.

## Deleting Jobs

Use command `harbor hub job delete JOB_ID JOB_ID_2 ...` to permanently delete jobs you own from the hub, including all of their trials and shares.

The command prompts for confirmation before deleting anything; pass `--yes` / `-y` to skip the prompt (required when scripting or piping).

Only the job's owner can delete a job. Jobs linked to a leaderboard submission and hosted jobs that are still running cannot be deleted.

## Managing Leaderboards

When creating a leaderboard, optionally restrict it to specific dataset-version
UUIDs in the create config:

```yaml
dataset_version_ids:
  - 11111111-1111-4111-8111-111111111111
dataset_version_refs:
  - latest
```

IDs and refs may be combined; the API resolves refs and stores their UUIDs.
Omitting both fields associates every version that exists when the leaderboard
is created. Setting both to empty lists associates none. Versions published
later are never added automatically.

Use `harbor hub leaderboard show BOARD` to display a curated leaderboard, or
add `--json` to print the complete read API response. `BOARD` may be a UUID or
an `org/package/name` slug.

Export an update-ready YAML or JSON definition, edit it, and apply it:

```bash
harbor hub leaderboard export BOARD --output board.yaml
harbor hub leaderboard update BOARD --config board.yaml
```

Simple definition fields can be updated directly; these flags override values
from `--config` when both are provided:

```bash
harbor hub leaderboard update BOARD --title "New title" --description "New description" --visibility private
```

Rows have the same round-trip workflow. Use `--all` to export every row:

```bash
harbor hub leaderboard row list BOARD
harbor hub leaderboard row create BOARD --config new-rows.yaml
harbor hub leaderboard row export ROW_ID --output row.yaml
harbor hub leaderboard row update ROW_ID --config row.yaml
harbor hub leaderboard row update ROW_ID --status hide

harbor hub leaderboard row export BOARD --all --output rows.yaml
harbor hub leaderboard update BOARD --config board.yaml --rows rows.yaml
```

`row list` shows canonical ranks, configured leaderboard columns, status, trial
count, timestamps, and row IDs. It supports `--limit`, `--page`, `--json`, and
`--quiet` using the same paging behavior as `hub job list`.

The create config contains a `rows` list. Each row accepts `metadata`, `metrics`,
`status`, and optional `trial_ids`. You can also create a leaderboard and its
initial rows atomically with `harbor hub leaderboard create --rows new-rows.yaml`.

Definition and batch row changes commit atomically. A schema change is rejected
when any resulting row is invalid. Delete incompatible rows explicitly with
`harbor hub leaderboard row delete`, or update them in the same command with
`--rows`. Use `--dry-run` with the combined `--config` and `--rows` migration
form to validate it without committing.

The API mirrors these resource boundaries: `leaderboard-update` changes only
the definition; row create, read, update, delete, and trial changes use their
dedicated endpoints. Only `leaderboard-migrate` accepts definition and row
changes together.

Manage row provenance separately:

```bash
harbor hub leaderboard row trial list ROW_ID
harbor hub leaderboard row trial set ROW_ID --trial-id TRIAL_ID
harbor hub leaderboard row trial add ROW_ID --trial-id TRIAL_ID
harbor hub leaderboard row trial remove ROW_ID --trial-id TRIAL_ID
```

Leaderboard reads and mutation responses return `n_trials` rather than every
association.
`row trial list` pages through the associations; use `--json` for one page or
`--quiet` to stream every trial ID.

`set` can also read the complete replacement list from YAML or JSON:

```yaml
trial_ids:
  - 11111111-1111-1111-1111-111111111111
  - 22222222-2222-2222-2222-222222222222
```

```bash
harbor hub leaderboard row trial set ROW_ID --trial-ids-file trials.yaml
```

`set` replaces every association; use `set --clear` to remove them all. Trial
changes do not recompute row metadata or metrics.

## Combined Flags

In addition to those listed. the following options are available

* `-q, --quiet` prints only IDs for piping into xargs
* `--no-trunc` show full cell content (wrap cells instead of ellipsis)
* `--no-headers` omit the header row
* `--page N` pull a particular page (disables interactive pages)
* `--json` get result as raw JSON
* `--debug` show full traceback on failure


# RL



Performing RL on containerized agentic tasks is a common use case but difficult to implement and scale. Harbor provides a simple interface for training on containerized agentic environments.

We plan to integrate with most popular RL frameworks. For now, we have worked with the SkyRL team to implement their interface for RL on Harbor tasks.

In general, performing RL on Harbor tasks requires generating rollouts and recording tokens and rewards. This usually involves implementing an RL framework's rollout interface.

We recommend using the `TrialConfig` or `JobConfig` classes to configure the rollouts. We also recommend using [cloud sandboxes](/docs/run-jobs/cloud-sandboxes) for scaling rollouts and reducing startup/teardown overhead.

Rollout interfaces typically receive batches of data to be used for the rollouts. We recommend using the `TaskConfig` class as the batch object. These configs can point to remote or local tasks.

```python
task_configs = [
    TaskConfig(
        path="cancel-async-tasks",
        git_url="https://<version-control-url>/<org>/<repo>.git",
        git_commit_id="<hash>"
    ), # Remote task
    TaskConfig(
        path="tasks/fix-dockerfile-syntax",
    ), # Local task
]
```

## Example of a rollout interface

```python title="rollout_interface.py"
from pathlib import Path

from pydantic import BaseModel

from harbor.job import Job
from harbor.models.environment_type import EnvironmentType
from harbor.models.job.config import JobConfig, OrchestratorConfig
from harbor.models.trial.config import EnvironmentConfig, TaskConfig

from your_rl_framework import SomeRLFrameworkRolloutInterface, Rollout


class HarborRolloutInterface(SomeRLFrameworkRolloutInterface):
    async def run(self, task_ids: list[TaskId]) -> list[Rollout]:
        job = Job(
            config=JobConfig(
                jobs_dir=Path("jobs"),
                environment=EnvironmentConfig(
                    type=EnvironmentType.DAYTONA,
                ),
                agents=[
                    AgentConfig(
                        name=AgentName.TERMINUS_2,
                        model_name="hosted_vllm/<model-name>",
                        kwargs={
                            "base_url": "https://<vllm-server-url>",
                        }
                    )
                ]
                orchestrator=OrchestratorConfig(
                    n_concurrent_trials=32,
                ),
                tasks=task_configs,
            ),
        )

        result = await job.run()

        rollouts = []
        for trial_result in result.trial_results:
            reward = (
                trial_result.verifier_result.rewards.get("reward", 0)
                if trial_result.verifier_result and trial_result.verifier_result.rewards
                else 0
            )

            if (
                trial_result.agent_result
                and trial_result.agent_result.metadata
                and "token_ids" in trial_result.agent_result.metadata
                and "mask_ids" in trial_result.agent_result.metadata
            ):
                token_ids = trial_result.agent_result.metadata["token_ids"]
                mask_ids = trial_result.agent_result.metadata["mask_ids"]
            else:
                raise ValueError(
                    f"Missing token_ids or mask_ids for trial {trial_result.trial_name}"
                )

            rollout = Rollout(
                reward=reward,
                token_ids=token_ids,
                mask_ids=mask_ids,
            )

            rollouts.append(rollout)

        return rollouts

```

## Collecting tokens from agents

There are two strategies for collecting tokens:

1. Intercepting tokens from a vLLM server
2. Returning tokens as part of the agent result metadata

(1) assumes you are using a vLLM server to generate tokens and have a framework that handles the interception. (2) assumes your agent is configured to return tokens as part of the agent result metadata. We are working to add flags to Terminus 2 to include this information. If you plan to train using a custom harness, be sure to include this information in the agent result metadata.


# SFT



Harbor includes utilities for turning trials (agent task completion attempts) into conversational traces that can be fed into supervised fine-tuning pipelines for agentic LLMs. Export helpers live under `harbor.utils.traces_utils` and power several CLI entry points.

<Callout title="Requires ATIF trajectory format" type="info">
  The SFT exporter works with any agent that produces trajectories in the [ATIF format](/docs/agents/trajectory-format). This includes Terminus-2, OpenHands, Claude Code, Gemini CLI, and other agents with ATIF support. Agents without ATIF trajectory support cannot be exported.
</Callout>

For best results with SFT data export, configure Terminus-2 with appropriate trajectory settings:

```python
from harbor.models.agent.trajectory_config import TrajectoryConfig

trajectory_config = TrajectoryConfig(
    raw_content=True,      # Preserve exact LLM responses for SFT
    linear_history=True    # Split on summarization for clean sequences
)
```

Learn more about trajectory configuration options in the [Terminus-2 documentation](/docs/agents/terminus-2#trajectory-configuration).

* Each exported row represents one `agent/episode-*` directory and captures the input `debug.json` messages plus the final agent reply from `response.json` or `response.txt`.
* Rows include metadata such as `agent`, `model`, `model_provider`, `task`, `trial_name`, `episode`, and `run_id`, letting you merge runs from multiple jobs.
* `--sharegpt` adds a ShareGPT-style column to support instruction-tuning datasets expecting the `{"from": "...", "value": "..."}` schema.
* Success filtering (`--filter success|failure`) inspects `result.json` and lets you keep only passing or failing attempts for curriculum-style datasets.

Run `harbor traces export` on a trial directory (or a parent directory) to build a `datasets.Dataset`. The command prints the number of rows produced and, when `--push` is set, uploads directly to the Hugging Face Hub.

```bash
harbor traces export \
  --path trials \
  --recursive \
  --episodes last \
  --filter success \
  --sharegpt \
  --push \
  --repo my-org/harbor-terminus2-sft
```

### Key options

import { TypeTable } from 'fumadocs-ui/components/type-table';

<TypeTable
  type={{
  "--episodes": {
    description: "Choose whether every episode (all) or only the final attempt per trial (last) is exported.",
    type: "all | last",
    default: "all"
  },
  "--sharegpt / --no-sharegpt": {
    description: "Adds (or disables) a ShareGPT-compatible column alongside the OpenAI-format conversations.",
    type: "flag",
    default: "Disabled"
  },
  "--filter": {
    description: "Keep only runs with positive (or zero) rewards (when result.json is present).",
    type: "success | failure | all",
    default: "all"
  },
  "--push": {
    description: "Upload the assembled dataset to the repo given by --repo. Provide HF_TOKEN or HUGGINGFACE_TOKEN in your environment for auth.",
    type: "flag",
    default: "Disabled"
  },
  "--verbose": {
    description: "Print per-trial discovery details, useful for debugging missing episodes.",
    type: "flag",
    default: "Disabled"
  }
}}
/>

If you want to persist the dataset locally (e.g., to Parquet), call the Python helper directly:

```python
from harbor.utils.traces_utils import export_traces

dataset = export_traces("trials", episodes="last", success_filter="success")
dataset.to_parquet("harbor-terminus2-success.parquet")
```

The `datasets` library is an optional dependency; install it if you plan to export traces.

`harbor run` can export traces automatically once a job completes. Pass trace flags alongside your job invocation:

```bash
harbor run \
  --config examples/configs/features/job.yaml \
  --agent claude-code \
  --model anthropic/claude-3-sonnet-20240229 \
  --export-traces \
  --export-sharegpt \
  --export-episodes last \
  --export-push \
  --export-repo my-org/harbor-job-run
```

When `--export-traces` is set, Harbor exports from the produced job directory using the same machinery as `harbor traces export`. The `--export-*` options mirror the standalone CLI flags and default to in-memory exports unless `--export-push` is provided. Errors during export are surfaced at the end of the job run without interrupting evaluation.

`harbor sweeps run` can emit split datasets that separate successful and failed trajectories. Supply `--push` together with one of the repo arguments:

```bash
# Push a DatasetDict with "success" and "failure" splits
harbor sweeps run \
  --config examples/configs/features/job.yaml \
  --max-sweeps 3 \
  --trials-per-task 2 \
  --push \
  --export-repo my-org/harbor-sweeps
```

You can also push successes and failures to independent repos by combining `--push` with `--export-separate` (alias `--no-export-splits`) plus `--export-repo-success` and `--export-repo-failure`. These exports reuse the same trace discovery logic and default to the last episode from each trial.