Skip to main content
A run scores each case with the suites you picked. Each suite gives the case a pass or a fail with a one-line reason. A suite that doesn’t apply to a case is skipped, not failed.

The judge

Judged suites use one model call per case, which covers every judged suite at once. The judge is told to be strict and fair and to return its verdict as JSON. It sees the question, the answer, the passages, the expected answer and your policy, and nothing else.

Tool call arguments

A case passes when the app made a call with the expected name, and every expected argument matches:
  • Text is compared ignoring case and surrounding spaces.
  • Objects match when every expected key matches. Extra keys from the app are allowed.
  • Lists must have the same items in the same order.
The reason names the first argument that differed:

Policy and tone

Write the policy in plain sentences. For example:
Policy

Pass rate

A suite’s pass rate is the cases that passed divided by the cases it applied to. The run’s overall pass rate counts every verdict across suites.