How to Evaluate and Secure AI Agents: Evals, Guardrails and Human-in-the-Loop

Building an AI agent that works on your laptop takes an afternoon. Building one you can trust with real data, real money or real users takes much longer, and most of the extra time goes into two things: knowing how well it works (evaluation) and limiting the damage when it does not (security and guardrails). This article is a practical checklist for both.

Why agents are hard to evaluate

• They are non-deterministic. The same task can succeed on one run and fail on the next.
• The path matters, not only the answer. An agent can reach the right result by a wasteful or dangerous route.
• Errors compound. A small per-step failure rate becomes a large failure rate over a long task (see What is Agentic AI?).
• The environment changes. Tools, websites and APIs behave differently from one day to the next.

Evaluate at three levels

Build a small, real eval set first

1. Collect real tasks. Twenty to fifty examples from actual use beat a thousand synthetic ones. Include hard cases and tasks the agent should refuse.
2. Define success so a machine can check it, wherever possible: a test suite, a validator, an exact-match field, a numerical tolerance.
3. Run each task several times. Measure the consistency of success, not only whether it can succeed once. Benchmarks such as τ-bench report how often an agent succeeds on every one of k repeated trials, which is much closer to what users experience than a single lucky run.
4. Track cost and latency with accuracy. An agent that is 3% better but five times as expensive may be a step backwards.
5. Re-run the eval on every change to the model, prompt or tools. Treat it like a test suite.

Public benchmarks, such as SWE-bench for fixing real software issues, are useful for comparing models, but they cannot tell you how an agent performs on your tasks. Your own eval set can.

Using an LLM as a judge

When success is subjective, a second model can grade outputs against a rubric. It scales well, but it has biases: it tends to favour longer answers and answers that resemble its own style. Use clear rubrics, compare against human judgments on a sample, and never let a judge be the only safeguard for something that matters.

The main security risks

• Prompt injection. Instructions hidden in text the agent reads, such as a web page, email, document or tool result, can redirect its behaviour. Indirect injection, where the attacker never talks to the agent directly, is the harder problem.
• Excessive agency. The agent has more permissions, tools or autonomy than its task needs, so a single mistake or manipulation causes real damage.
• Data exfiltration. An agent that can read private data and also reach the outside world, for example through links, email or web requests, can be tricked into leaking it. The combination of private data, untrusted input and an outbound channel is the one to design around.
• Unsafe output handling. Model output passed straight to a shell, database or browser can become an injection vector.
• Supply chain. Third-party tools, plugins and servers (see tools and MCP) run with whatever access you give them.

The OWASP Top 10 for LLM Applications is a good, regularly updated catalogue of these risks.

Guardrails that work

1. Least privilege. Scoped credentials, read-only access by default, and a sandbox for code execution with no network unless needed.
2. Treat model output as untrusted input. Validate arguments against schemas, allow-list commands and domains, and escape anything that reaches an interpreter.
3. Separate instructions from data. Label retrieved content as untrusted in the prompt, and do not let it change the task or the permissions.
4. Human approval for irreversible actions: payments, deletions, outgoing messages, production changes.
5. Hard limits. Step, time, cost and rate budgets, with a kill switch.
6. Audit logs. Record every model call, tool call and result so you can reconstruct what happened.
7. Defence in depth. No single guardrail is enough. Layer them, and assume each one will sometimes fail.

A pre-launch checklist

• Do we have an eval set from real tasks, with repeated runs and a tracked success rate?
• Can we explain, for each tool, the worst thing it could do, and have we limited it?
• Does the agent treat everything it reads from outside as untrusted?
• Are irreversible actions behind human approval?
• Are there budgets, a kill switch and complete logs?
• Have we tried to break it, with injected instructions, malformed inputs and hostile documents?
• Do we have a plan for when it fails, including a way for users to reach a human?

Keep reading

If you are new to the topic, start with What is Agentic AI?, then agents vs. workflows. For a domain where evaluation and provenance are central, see agentic AI for science.