Skip to main content
The Evals MCP server is part of the Evals Pro plan.
Evals’ MCP server is a remote server at:
It speaks Streamable HTTP and answers each request with JSON. A coding agent can run your test set after a change and read which cases regressed, without leaving the editor.

Authentication

There are two ways in. Both send Authorization: Bearer … with every request.

Sign in with Jabrod

Add the server without a key. The first time your client connects, it opens Jabrod’s sign-in page, auth.jabrod.com, in your browser: sign in, and allow the app to use your account. The token it gets works only for this server. In Claude Code:
Then run /mcp in Claude Code, choose jabrod-evals, and sign in. This needs a client that supports MCP authorization. Sign in with Jabrod explains the flow.

API key

For clients that can’t sign in, make a personal API key at jabrod.com/app/keys. It starts with jbd_sk_ and is shown once. Send it with every request:
On a plan without MCP access, every tool answers: “The Evals MCP server is part of Evals Pro.”

Client configuration with an API key

Tools

create_dataset cases

Each case takes the same fields as a test set row:
A case can also carry output, the answer your app already gave, to score recorded outputs instead of calling the app. Run those with traces: true.

start_run

  • What to test: target_id for a target, or workspace_id for a Context workspace.
  • Suites: any of grounded, correctness, refusals, tool_args, policy. The default is grounded, correctness and refusals. policy needs the policy text. See Suites and scoring.
  • One run at a time. Starting a second run while one is in progress returns an error.
  • A run is asynchronous. Call get_run until status is done or failed.

A regression check from an agent

  1. list_runs to find the last good run of the test set.
  2. start_run with the same test set and target.
  3. get_run until it is done.
  4. compare_runs with the old run as base_run_id and the new one as head_run_id. Read regressed first.

Errors

A problem you can fix comes back as a tool result with isError: true and a sentence saying what to do. Requests without a valid key get HTTP 401.