Skip to main content
Everything you can set for a run lives in two places: eval/conftest.yml for the models and prompts, and the evaluate_agent call for how many times each scenario runs.

Configure the models

eval/conftest.yml at the project root names two models. simulation.llm plays the user. evaluation.llm is the judge that scores criteria and quality metrics.
eval/conftest.yml
Configure them independently. The simulator writes short user messages, so a smaller, cheaper model is enough. The judge has to read a whole transcript and decide whether each criterion holds, so give it a more capable model. provider and model are required; any other key the provider accepts can be added next to them.

The three prompts

Three built-in prompt templates drive the simulated user and the judge. Each can be replaced with your own Jinja2 file.

Override a prompt

Add the path under the matching section of eval/conftest.yml. Paths are relative to the project root and cannot leave it. Leave a key out to use the built-in template.
eval/conftest.yml
Start from the built-in text below, copy it into your project, and change what does not fit your domain.

Simulated user

Tells the model to stay in character as the user from simulation_context, keep messages short, move toward the goal one step at a time, and end the conversation once the goal is met or the agent has made clear it cannot help. It receives simulation_context and returns the next user message and a done flag.

Criteria judge

Scores each of your goals.criteria against the transcript and a ledger of the events the agent recorded. It receives criteria_text, transcript, and event_ledger. The template adapts its event vocabulary to the engine of the running server; this is the text a Mantle run uses.

Metrics judge

Scores the built-in quality metrics from the transcript. It receives transcript.

How a run is judged

Criteria

Your goals.criteria sentences. The judge receives the transcript, plus a ledger of the events the agent recorded: which skill activated and completed, which memory keys were written, which tools ran and whether they returned an error. It uses the ledger to catch an agent that says “your order is placed” without having called the tool that places it. Each criterion is returned with PASS, FAIL or NOT REACHED and a rationale. NOT REACHED means the conversation never got to the point the criterion is about. It counts as not passed, so it fails the run like FAIL does. Criteria decide whether the run passes.

Built-in quality metrics

The judge also scores every run on the same fixed set of metrics, whether or not you wrote criteria: The four scores are averaged into bot_quality. The judge also writes a short summary of the conversation. Metrics are recorded, not enforced. A run can pass with task_completion: FAIL if all of its assertions and criteria passed. If task completion must be required for a scenario, write it as a criterion.

Run the evaluation

Two MCP tools do the work. Your coding agent calls them for you; you can also ask for either one by name. validate_scenario reads one scenario file and reports what is wrong with it: YAML that does not parse, a missing name or simulation_context, or an assertion type or field that is not in the scenario schema. Run it after you edit a file by hand. It does not check assertion keys against the engine; evaluate_agent does that when it reads the running server. evaluate_agent runs the scenarios. For each run it:
  1. Loads eval/conftest.yml and the scenario, and rejects any assertion key the running engine does not support.
  2. Opens a fresh sim-<uuid> conversation, sends /session_start, and applies initial_slots.
  3. Runs the simulated conversation against your agent.
  4. Fetches the tracker and evaluates the assertions.
  5. Sends the transcript and event ledger to the judge for criteria and metrics.
  6. Writes the result files and updates the batch summary.
Tell your coding agent what to run and how often:
run_count is how many times each scenario is simulated. It defaults to 1 and goes up to 10. The simulator is not deterministic, so a scenario that passes 3 of 3 says more than one that passes 1 of 1. Results are written as each run finishes; you do not have to wait for the batch to see the first failure. run_name gives the batch a short name, such as Refund flow regression, so you can tell batches apart later. It is optional and is stored in the batch’s start.json.

Read the results

Each batch writes to eval/results/<timestamp>/, and earlier batches are kept. Running one scenario three times gives:
start.json is written before the first run starts, so a batch that is still running already says what it is. It holds run_name (null when you did not pass one), started_at in UTC, run_count, and one entry per scenario with its scenario_slug, scenario_name, and scenario_type (null when the scenario sets no type). When a batch is split across several evaluate_agent calls that share one experiment_timestamp, the scenarios of every call end up in the same file, and started_at and run_count stay from the first call. Every run has two files. run_N.txt is the report you read. run_N.json is the same run as data, with scenario_name, scenario_type, eval_passed, criteria, task_completion, latency, scale_metrics, and the assertion rows when the scenario had any. Each row in criteria has the judge’s reasoning and a verdict (passed, failed or not_reached). passed is false for not_reached. judge_summary is the quality-metrics judge’s overall summary. Each assertion row has a detail. scale_metrics_reasoning and task_completion_reasoning hold the judge’s reasoning for those metrics, and transcript lists each turn’s user message and bot replies. Use it when you script over results. While a run is in progress, a small run_N.started.json file holds its UTC started_at time. It is removed once run_N.json is written, so a finished run has only run_N.txt and run_N.json (no .started.json). A run that could not be judged still writes its run_N.json. These fields say what happened:
  • scenario_name and scenario_type identify the scenario. scenario_type is the scenario’s type value, or null when the scenario sets none.
  • error is why the run failed, such as a timeout or a connection error. It is null when the run completed.
  • error_is_harness is true when error came from the evaluation itself, such as a timeout or a provider failure. It is false when the agent caused it, such as saying nothing, and when error is null. It describes error only. A run where only a judge failed has error_is_harness false, but summary.txt still counts it as a harness failure.
  • metrics_error is the quality-metrics judge’s error. criteria_error is the criteria judge’s error and appears only when that judge failed. A judge failure does not set error.
  • judge_summary is null when the quality-metrics judge failed, returned no summary, or did not run. task_completion_reasoning is null whenever task_completion is, and when the reasoning is empty. An assertion’s detail is null when there is none.

A run report

This is what one run_N.txt contains and what each part means.
  • overall_result is the pass rule applied to this run: every assertion and every criterion passed.
  • Each criterion has the judge’s rationale. A NOT REACHED criterion counts as failed: the conversation never got to it, so check whether an earlier step went wrong. Read the rationale before changing anything; it tells you which turn the judge objected to.
  • Quality metrics are recorded and do not affect overall_result.
  • Each assertion is listed with the id it checked. A failed assertion carries a short detail saying what was found instead.
  • The transcript numbers every message. When the agent greets on session start, the transcript begins with agent: lines and no leading user: line.
  • Latency is reported for visibility and does not affect pass or fail. Time to first token (ttft) is recorded when the server streamed its reply.

The batch summary

summary.txt lists each scenario with its pass count, how long it took, and where its run files are, then a total across the batch and the time it took. When any run failed, a final Scenarios requiring attention list names them with their pass rate. summary.json holds the same numbers as data.

Use a failure as feedback

A failed criterion means the judge decided the handling was wrong; the rationale says where. A failed assertion names the fact that did not hold. If the agent did the right thing in words you did not expect, loosen the criterion. If the agent did the wrong thing, change the skill, its tool constraints, or its instructions, train again, and run the same scenario again.

Inspect a conversation

Every report ends with an Inspector URL. Open it to step through that conversation turn by turn and see the memory values and tracker events behind each reply, the same view you get from rasa inspect. Simulated conversations use a sim- prefix on the sender id, which sets them apart from conversations you held yourself.