The Evals MCP server is part of the Evals Pro plan.
Authentication
There are two ways in. Both sendAuthorization: Bearer … with every request.
Sign in with Jabrod
Add the server without a key. The first time your client connects, it opens Jabrod’s sign-in page, auth.jabrod.com, in your browser: sign in, and allow the app to use your account. The token it gets works only for this server. In Claude Code:/mcp in Claude Code, choose jabrod-evals, and sign in.
This needs a client that supports MCP authorization. Sign in with Jabrod explains the flow.
API key
For clients that can’t sign in, make a personal API key at jabrod.com/app/keys. It starts withjbd_sk_ and is shown once. Send it with every request:
Client configuration with an API key
- Claude Code
- Cursor
- Claude Desktop
- Other clients
Tools
create_dataset cases
Each case takes the same fields as a test set row:
output, the answer your app already gave, to score recorded outputs instead of calling the app. Run those with traces: true.
start_run
- What to test:
target_idfor a target, orworkspace_idfor a Context workspace. - Suites: any of
grounded,correctness,refusals,tool_args,policy. The default isgrounded,correctnessandrefusals.policyneeds thepolicytext. See Suites and scoring. - One run at a time. Starting a second run while one is in progress returns an error.
- A run is asynchronous. Call
get_rununtilstatusisdoneorfailed.
A regression check from an agent
list_runsto find the last good run of the test set.start_runwith the same test set and target.get_rununtil it is done.compare_runswith the old run asbase_run_idand the new one ashead_run_id. Readregressedfirst.
Errors
A problem you can fix comes back as a tool result withisError: true and a sentence saying what to do. Requests without a valid key get HTTP 401.
