Reports

Every eb run writes a timestamped directory of results and prints a summary to the console. This page covers what you get and how the verdict is computed.

Output directory layout

Each run creates a directory under the configured output_dir (default .eval-banana/results), keyed by a run id:

.eval-banana/results/<run_id>/
├── report.json                # machine-readable, the full report
├── report.md                  # human-readable Markdown summary
└── checks/
    ├── <check_id>.json        # per-check result
    ├── <check_id>.stdout.txt  # captured stdout (only if non-empty)
    └── <check_id>.stderr.txt  # captured stderr (only if non-empty)

• report.json is the structured report — parse this in tooling. Its per-check stdout is a bounded tail, not the whole stream (see Large output below).
• report.md is a condensed, human-readable summary with each check's verdict and reason.
• checks/<check_id>.json holds one check's status, score, and reason.
• .stdout.txt / .stderr.txt are written only when the check produced output — for deterministic checks, this is where the diagnostics you printed to stderr land. .stdout.txt always holds the complete captured stdout stream, even though report.json keeps only a bounded tail of it.

Every per-check JSON object includes check_definition_sha256 in sha256:<64 lowercase hex> form. Its versioned, length-framed input covers the exact YAML bytes and, for a deterministic script_path check, the referenced script bytes frozen before execution. The runner executes that same script snapshot, so the digest identifies the effective definition that produced the verdict instead of only the YAML pointer to it. The private launcher preserves the original resolved script path as __file__ and sys.argv[0], plus its parent as the first import path. Inline scripts are already part of the YAML component.

The v1 digest input starts with eval-banana/check-definition-sha256/v1\0. Each component is encoded as u64be(name_length) || name || u64be(content_length) || content. The first is always definition.yaml; a readable external script adds referenced-script. A missing or unreadable external script adds an empty referenced-script-unavailable component and yields an error result.

For harness_judge, details also records the resolved agent_type, model, and nullable reasoning_effort. These are the effective values after template, project, environment, and CLI selection; model also reflects a per-check override. Automation can therefore bind the verdict to the judge configuration that produced it.

If a caller already allocates an attempt-unique output directory, it can write this same layout without the generated run-id child:

eb run --flat-output --output-dir /absolute/path/to/attempt/eval

The exact directory must be absent or empty and cannot be a symlink. Existing contents are refused, never overwritten. Runs without --flat-output keep the default timestamped layout shown above.

Large output

A check's stdout field in report.json (and in checks/<stem>.json) holds only a bounded tail — the last 2000 characters — prefixed with a truncation banner naming the full-output file whenever it was shortened. The complete, untruncated stream is always written to checks/<stem>.stdout.txt. This keeps report.json small (tens of KB) even when a harness_judge agent emits multi-megabyte stream-json output, so tooling can read the report without loading megabytes it does not need.

For harness_judge, details.raw_response_path records the run-relative path to that full stdout file (null when the check produced no output), replacing the pre-0.4.0 details.raw_response that duplicated the entire stream inline. Open that file only when the bounded tail in report.json is not enough.

Console summary

The console output prints these labels:

• Run ID: — the run id;
• Score: — points_earned/total_points;
• Percentage: — the pass ratio as a percentage;
• Passed: — yes or no;
• a per-check list, showing each check's reason (or, for errored checks, the error detail).

The three statuses

Every check resolves to exactly one status, which sets its score:

A harness judge must satisfy both halves of its protocol: process exit 0 and a valid JSON verdict. A non-zero judge exit is always error with score 0, even if stdout happens to contain {"score": 1}.

All checks carry equal weight, so the run score is simply points_earned / total_points.

How a run passes

A run passes only when both conditions hold:

run_passed = (points_earned / total_points) >= pass_threshold
             AND errored_checks == 0

Two things follow:

• pass_threshold defaults to 1.0, so by default every check must pass. Lower it in configuration to tolerate some failures.
• A single error fails the entire run, no matter the threshold or how many other checks passed. An error means a check could not be trusted to give a verdict, so the run is not clean.

Exit code

eb run exits 0 when run_passed is true and 1 otherwise. That makes it a drop-in CI gate:

- name: Run evals
  run: eb run     # non-zero exit fails the job

If a check keeps landing in error, fix the check itself — a script the runner can't locate or launch, or a judge whose output contains no valid {"score": 0|1} verdict — before trusting the run. (A script that runs and exits non-zero is a failed, not an error.) See Troubleshooting.