Reports

Every eb run writes a timestamped directory of results and prints a summary to the console. This page covers what you get and how the verdict is computed.

Output directory layout

Each run creates a directory under the configured output_dir (default .eval-banana/results), keyed by a run id:

.eval-banana/results/<run_id>/
├── report.json                # machine-readable, the full report
├── report.md                  # human-readable Markdown summary
└── checks/
    ├── <check_id>.json        # per-check result
    ├── <check_id>.stdout.txt  # captured stdout (only if non-empty)
    └── <check_id>.stderr.txt  # captured stderr (only if non-empty)

report.json is the complete, structured report — parse this in tooling.
report.md is a condensed, human-readable summary; report.json remains the full record.
checks/<check_id>.json holds one check's status, score, and reason.
.stdout.txt / .stderr.txt are written only when the check produced output — for deterministic checks, this is where the diagnostics you printed to stderr land.

Console summary

The console output prints these labels:

Run ID: — the run id;
Score: — points_earned/total_points;
Percentage: — the pass ratio as a percentage;
Passed: — yes or no;
a per-check list, showing each check's reason (or, for errored checks, the error detail).

The three statuses

Every check resolves to exactly one status, which sets its score:

Status	Meaning	Score
`passed`	The condition held (exit `0`, or judge `score == 1`)	1
`failed`	The condition did not hold (non-zero exit, or judge `score == 0`)	0
`error`	The check could not be run or its verdict could not be read (script file missing or OS launch failure; malformed/absent judge verdict; harness timeout)	0

All checks carry equal weight, so the run score is simply points_earned / total_points.

How a run passes

A run passes only when both conditions hold:

run_passed = (points_earned / total_points) >= pass_threshold
             AND errored_checks == 0

Two things follow:

pass_threshold defaults to 1.0, so by default every check must pass. Lower it in configuration to tolerate some failures.
A single error fails the entire run, no matter the threshold or how many other checks passed. An error means a check could not be trusted to give a verdict, so the run is not clean.

Exit code

eb run exits 0 when run_passed is true and 1 otherwise. That makes it a drop-in CI gate:

- name: Run evals
  run: eb run     # non-zero exit fails the job

If a check keeps landing in error, fix the check itself — a script the runner can't locate or launch, or a judge whose output contains no valid {"score": 0|1} verdict — before trusting the run. (A script that runs and exits non-zero is a failed, not an error.) See Troubleshooting.