Harness Setup

harness_judge checks are graded by an AI coding agent — the harness. eval-banana never calls a model API directly; it shells out to the agent's CLI, which reads the files and returns the verdict. This page shows how to select and configure that agent.

A harness is required only for judge checks. If any discovered harness_judge check has no harness configured, eb run and eb validate fail fast before running anything. Deterministic checks never need a harness.

Built-in agent templates

eval-banana ships templates for these agents — set [harness] agent to one of them:

codex · gemini · claude · openhands · opencode · pi

Each template knows how to build the agent's command line (the executable, the flags, how to pass a prompt and a model). You only supply the choice of agent and, optionally, a model and reasoning effort.

Minimal configuration

# .eval-banana/config.toml
[harness]
agent = "codex"
# codex GPT-5.6 tiers: gpt-5.6-sol (flagship), gpt-5.6-terra, gpt-5.6-luna
model = "gpt-5.6-sol"
# reasoning_effort = "high"   # codex default; override to low/medium/xhigh/max

You can also set these per run without touching the file:

eb run --harness-agent codex --harness-model gpt-5.6-sol --harness-reasoning-effort high

Pass the agent explicitly during definition-only validation when there is no project config (or when callers must not inherit an ancestor config):

eb validate --no-project-config --harness-agent codex

Validation checks that the required agent setting is present but does not execute it. Model and reasoning-effort flags apply only to run.

Each executed harness result records the resolved agent, model, and nullable reasoning effort in its details. The agent process must exit 0 and emit a valid JSON verdict. Any non-zero judge exit is an errored check with score 0, even if its output contains a parseable passing verdict.

Those fields describe values that actually have an injection surface. A model requires the selected template's model_flag or model_env_vars; reasoning effort requires a reasoning_effort_flag token containing {effort}. An unsupported selection errors before launch and leaves both effective details null rather than claiming configuration the judge never received.

--no-project-config makes --cwd the project root and disables upward .eval-banana/config.toml discovery. Use it for automation that must rely only on explicit flags, environment variables, and this installed eval-banana version's built-in agent templates.

Credentials

The harness subprocess inherits the parent shell environment, so provide API keys exactly as you would to run the agent by hand:

Running in CI

Because the harness reads keys from the environment, wiring it into CI is just exporting the right secret:

- name: Run evals
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: eb run

eb run exits 0 on a passing run and 1 otherwise, so it works directly as a gate.

Extra environment variables

Inject additional env vars into the harness subprocess with [harness.env]:

[harness.env]
CI = "1"
PYTHONUNBUFFERED = "1"

Custom and overridden agents

Add [agents.<name>] sections to override a built-in template or define a new agent. Omitted fields inherit from the built-in template of the same name; a brand-new agent must provide command:

# Tune the built-in codex template
[agents.codex]
# GPT-5.6 tiers: gpt-5.6-sol (flagship, default), gpt-5.6-terra, gpt-5.6-luna
default_model = "gpt-5.6-sol"
reasoning_effort = "high"
 
# Define a brand-new agent
[agents.myagent]
command = ["my-cli", "run"]
shared_flags = ["--headless"]
prompt_flag = "--prompt"
model_flag = "--model"

Provider-wide env with placeholders

[agents.<name>.provider_env] sets env vars for that agent's subprocess. Values may contain {env:VARNAME} placeholders resolved from the parent shell — handy for routing an agent through an OpenAI-compatible gateway:

[agents.claude.provider_env]
ANTHROPIC_BASE_URL = "https://openrouter.ai/api"
ANTHROPIC_AUTH_TOKEN = "{env:OPENROUTER_API_KEY}"
ANTHROPIC_API_KEY = ""

Choosing a model tier (codex)

For codex, the GPT-5.6 family has three tiers, selectable via harness.model, [agents.codex] default_model, --harness-model, EVAL_BANANA_HARNESS_MODEL, or a per-check model: override:

See Harness Judge Checks for overriding the model on a single check.