Evals

Probes help you discover weaknesses. Evals turn the behavior you care about into repeatable checks that can run before release, in CI, and on a schedule.

Cases

Cases live in evals/cases.py. Each case sends one input to an agent (agent=) and can run up to two checks:

judge: AgentAsJudgeEval scores the response against criteria (binary pass/fail) using an LLM.
reliability: ReliabilityEval checks which tools fired against expected_tool_calls.

A case looks like this:

evals/cases.py

from os import getenv

# The WebSearch agent calls parallel_search when PARALLEL_API_KEY is set, web_search otherwise.
_WEB_SEARCH_TOOL = "parallel_search" if getenv("PARALLEL_API_KEY") else "web_search"

CASES: tuple[Case, ...] = (
    Case(
        name="web_search_recent_anthropic_research",
        agent=web_search,
        input="What did Anthropic publish about agent research recently?",
        tags=("live",),
        timeout_seconds=120,
        criteria=(
            "Answers the question by citing at least one real Anthropic URL "
            "(anthropic.com domain). The response is grounded in fetched content "
            "rather than refusing to answer."
        ),
        expected_tool_calls=(_WEB_SEARCH_TOOL,),
    ),
    # add more cases here
)

A case can use either check or both. If both are set, the agent runs once and feeds the same response into both. Add tags to group your cases into suites. The template uses three tags: smoke, release, and live. This case uses the live tag because its answer depends on the open web.

Run the suite

The suite imports agents on the host and writes results to Postgres through eval_db. Start the database, activate the local virtual environment, and set OPENAI_API_KEY in .env.

Start Postgres

docker compose up -d agentos-db

Create a virtual environment

The eval suite runs on the host and needs a local virtual environment:

./scripts/venv_setup.sh

Activate it:

source .venv/bin/activate

Run the eval suite

python -m evals --tag smoke      # fast checks
python -m evals --tag release    # pre-release suite
python -m evals --tag live       # checks that depend on live sources
python -m evals                  # all cases

Other options:

python -m evals --name <case>                       # one case while iterating
python -m evals --tag release --json-output out.json
python -m evals -v                                  # stream full run panels

Each case prints its response and the verdicts for the checks it defines. The run ends with an Eval Summary table. Results are written to Postgres via eval_db. The eval history appears on os.agno.com alongside your sessions and traces, so you can compare inputs, responses, and verdicts across runs.

Diagnose failures with your coding agent

Open your coding agent and ask it to run:

Run the eval-and-improve skill in .agents/skills.

The coding agent runs the suite and classifies failures as bad criteria, real regressions, or unreliable LLM judgments. It proposes scoped fixes, updates the agent or case, and re-runs the affected checks.

Choose when to run evals

These behavioral checks call models, so choose a cadence that fits the cost and confidence you need.

Situation	Suggested run
While changing one behavior	Run the affected case with `--name <case>`.
Before a release	Run the `release` tag when you want broader confidence.
In CI	Use stable `smoke` or `release` cases. Leave `live` cases out.
After changing a model	Run the cases for affected agents.
On a schedule	Run `smoke` cases when recurring model calls are useful.

The template ships a run_evals workflow for scheduled checks. Scheduled evals are off by default because they make model calls. Set ENABLE_SCHEDULED_EVALS=True to run the smoke-tagged cases daily. See scheduling for the cron API.

What good cases look like

Specific. “Returns a JSON object with ticker and price” beats “Returns the right answer”.
Stable. Avoid prompts whose correct answer changes daily. Use phrasing like “describes a real, recent…” so the case remains useful as the answer changes.
Scoped to one behavior. One case per behavior makes failures easy to read.
Anchored to tools. expected_tool_calls catches responses that skip a required tool call.

Run your platform on Railway →

Welcome

Get Started

Features

Use Cases

Cases

Run the suite

Diagnose failures with your coding agent

Choose when to run evals

What good cases look like

Next

​Cases

​Run the suite

​Diagnose failures with your coding agent

​Choose when to run evals

​What good cases look like

​Next

Cases

Run the suite

Diagnose failures with your coding agent

Choose when to run evals

What good cases look like

Next