Building an AI agent from scratch: LangGraph, MCP, auth and evals

I kept reading about AI agents without really knowing what happens inside one. So I set myself four questions and built a small lab to answer them:

1. How does an agent loop actually work? I used LangGraph to build one step by step.
2. How do agents get their tools? That meant learning MCP, the Model Context Protocol.
3. How does auth work with MCP? When an agent acts for a user, how does the server know who that user is?
4. How do you test an AI system? And how do you know the tests are telling the truth?

The code is on GitHub: agentic-eval-lab.

The example: a support agent for a pretend bank

The lab’s example is a support agent for a pretend bank. A customer can ask for their balance, ask for a refund, or ask to talk to a person. Here’s what it looks like in the terminal:

$ uv run support-agent --customer u_bob
Tools: check_balance, process_refund, escalate_ticket. Policy loaded from policy://refunds.
Chatting as u_bob with claude-opus-5-5. Ctrl+D to quit.

you> What's my balance?
  planner   intent: balance_inquiry
  agent     wants check_balance(user_id='u_bob')
  tools     check_balance(user_id='u_bob') → ok: active, balance 1500.00
  agent     replied
agent> Your account is active, and your balance is $1,500.00.
    (intent=balance_inquiry)

you> I was charged twice for an order. Can I get $20 back?
  planner   intent: refund_request
  agent     wants process_refund(user_id='u_bob', amount=20)
  guardrail checked check_balance(user_id='u_bob') → ok: active, balance 1500.00
  guardrail refund allowed
  tools     process_refund(user_id='u_bob', amount=20) → ok: new balance 1480.00
  agent     replied
agent> Done. I've refunded $20.00 for the duplicate charge. Your new balance is $1,480.00.
    (intent=refund_request)

you> I'd also like a refund of $900 for last month's order.
  planner   intent: refund_request
  agent     wants escalate_ticket(user_id='u_bob', reason='Refund of $900 is over the $500 limit')
  tools     escalate_ticket(user_id='u_bob', reason='Refund of $900 is over the $500 limit') → ok: ticket #1 opened
  agent     replied
agent> I can't process a $900 refund automatically, since refunds over $500 require a human agent. I've opened ticket #1 with the details, and a team member will follow up with you.
    (intent=refund_request, escalated)

The dimmer lines are the agent’s steps as they run. The bank’s rules are simple:

• Refunds up to $500 go through automatically.
• Anything bigger must be passed to a person.
• Suspended accounts can’t get refunds.
• A refund can’t be more than the balance.

Because the rules are this clear, every conversation has one right outcome that can be checked in code: refunded, escalated or refused. That’s what makes the agent testable, which matters later in testing an AI system.

Part 1: The agent loop, with LangGraph

What an agent really is

Stripped down, an agent is a loop:

The model never runs anything itself. It only asks, and your code decides whether and how to do it. The “agent” is mostly ordinary code around a model call.

How a graph creates an agent loop

LangGraph describes the agent as a graph. Each node is a named step that reads and updates a shared state object, and each edge decides which step runs next. A loop appears when an edge points back to an earlier node: after the tools run, the graph goes back to the model, and it keeps going round until the model stops asking for tools.

In the lab’s graph, only two of the six steps call the model: one to sort the request and one to pick tools. The rest, like the guardrail that checks refunds, are plain code.

Rules, policies and guardrails belong in code

A model is not deterministic. Given the same conversation, it usually does the right thing, but not every time. So any rule that must never be broken, like “never refund more than $500 without a person”, needs a safeguard that doesn’t depend on the model.

The lab checks in two places. A guardrail node in the graph stops the clear cases early, like a refund over $500. Then the backend, the bank’s own system, enforces all four of the bank’s rules on every request, so it catches anything the guardrail lets through.

The prompt still includes the policy, so the model can explain a refusal to the customer. But the prompt is not what stops a bad refund: the code is. Even if the model gets it wrong, no refund can break the rules.

When a tool call goes wrong

The guardrail and the backend stop requests that break the rules. But even a valid request can fail when it runs, so the loop also needs a plan for when a tool call goes wrong:

Code: agent/graph.py · backend/store.py

Part 2: Tools, with MCP

What MCP is

MCP (Model Context Protocol) is a standard way for an AI app to find and use tools. A server exposes capabilities, and a client (here, the agent) discovers and calls them. The messages are JSON-RPC. You can think of it as a typed API whose documentation the model reads. This is the message the agent sends to make a refund, and the reply it gets back:

// Request: call the process_refund tool
{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "process_refund",
    "arguments": {
      "user_id": "u_bob",
      "amount": 20
    }
  }
}

// Response: the tool's result (trimmed)
{
  "jsonrpc": "2.0",
  "id": 7,
  "result": { "new_balance": "1480.00" }
}

How the agent talks to the server

The agent doesn’t know the tools in advance. It asks the server for them when it starts, and then calls them as the model needs them:

An MCP server can expose three kinds of things. The main difference is who decides to use each one:

Here’s a tool definition. The type hints become the JSON schema the model sees, and the docstring becomes its description:

@server.tool()
def process_refund(user_id: str, amount: float) -> RefundResult:
    """Refund `amount` to the customer's balance.

    Refunds over $500 need a person, and suspended accounts
    or a low balance are refused.
    """

So a tool’s description matters as much as its code: the model reads every word of it to decide what to call.

Where the server runs

So far we’ve seen what the messages look like, but not how they get from the agent to the server. MCP calls this the transport, and it depends on where the server runs. The lab’s server supports both standard options:

stdio is the standard input and output of a process, the same channel you use when you pipe text into a command line tool. Nothing listens on the network, so only the program that started the server can talk to it.

Code: mcp_server/server.py

Part 3: Auth for MCP

If a bank exposes an MCP server, how does it know which customer the agent is acting for? And how does it stop the agent touching someone else’s account?

How it works

MCP uses standard OAuth 2.1, the same “Sign in with…” flow used across the web. The lab includes a small demo bank login server, so every step is visible. The agent finds out it needs a token, the customer signs in at the bank, and the agent gets a token to send with every request:

PKCE makes sure only the app that started the sign-in can swap the one-time code for a token, so an intercepted code is useless.

Notice that the MCP server never sees a password and never issues tokens. It only checks them. Signing in is the bank’s job.

To learn more about OAuth 2.1 and PKCE:

• What’s new with OAuth 2(.1)? from Frontegg
• The only 3 OAuth 2.1 flows you need from Duende Software
• OAuth 2.1 complete guide: what developers need to know from IAMDevBox

What the token is used for

Every tool checks three things before doing anything: that there is a token, that it has the right scope (a permission, like refunds:write), and that it belongs to the customer whose account the tool would touch:

if token is None:
    return "forbidden: not signed in"
if scope not in token.scopes:
    return f"forbidden: requires scope {scope!r}"
if token.subject != user_id:
    return f"forbidden: signed in as {token.subject}, cannot act on {user_id}"

So a token for Alice gets forbidden on Bob’s account, even if a prompt injection convinces the agent to try. A token without the refunds permission can still check balances but can’t refund.

What the lab simplifies

The lab cuts some corners to keep sign-in easy to follow. A real bank would not:

Code: mcp_server/auth.py · auth_server/

Part 4: Testing an AI system

Once the agent works, how do you know it keeps working?

Tests check your code. Evals measure the model.

“Testing an AI app” splits into two quite different things:

Some examples from the lab show the difference:

The fake is a scripted model: it returns pre-written responses, tool calls included, in order. With it, the whole agent (graph, guardrail, retries, escalation) runs against the real MCP server and a real database without calling an API, so the same input gives the same result every time. A few tests are plain unit tests instead, like the backend’s refund rules.

These tests catch real bugs in the code around the model before any eval runs, in seconds and without a single API call.

Check what happened, not what the agent said

An agent can say “I’ve refunded $20” without refunding anything. So evals should mostly look at what it did.

Each eval case is a scenario: a customer, what they say, and what must be true afterwards. Here’s the one for a refund over the limit, written in YAML:

- id: refund-over-limit
  customer: u_bob
  turns: ["I'd like a refund of $900, please."]
  expect:
    refunds: []  # no refund rows
    tickets: 1  # one ticket for a person
    escalated: true
    tools:
      required: [escalate_ticket]
      forbidden: [process_refund]  # must not even try
      max_calls: 3

To run it, the lab gives the agent a fresh database, sends it the customer’s message, and lets it finish. Then it compares what happened with expect, in four layers, each matching the lines of the same colour above:

Why check the tools as well as the database? Because the guardrail can hide a mistake. If the agent tries the $900 refund anyway, the guardrail blocks it, and the database ends up correct. Checking only the end state would pass that agent, even though it tried to break the rule. The trajectory layer fails it, because process_refund is forbidden in this scenario.

This is one of 26 scenarios. Together they cover:

Test your tests

To know the evals work, run them against agents that are deliberately broken. Using the scripted model, the lab has one that always refunds, one that refunds twice, one that refunds the wrong customer, and one that skips the balance check. Each must fail the right layers. If a broken agent passes, the bug is in the eval.

Right once isn’t reliable

The usual measure is pass¹: run each scenario once, or average all runs, and report the share that passed. That’s the agent’s accuracy. But it hides how consistent the agent is, and for an agent that moves money, consistency is critical.

pass^k is the chance the agent gets a scenario right k times in a row. The lab runs every scenario 5 times, then estimates pass^k as the chance that k runs, picked at random from those 5, all passed. This method comes from τ-bench, a benchmark for customer service agents that introduced pass^k.

Here’s what it says about one scenario that passed 4 of its 5 runs:

Take pass². There are 10 different ways to pick 2 runs out of 5. With 4 passing runs, 6 of those pairs are both passes, so pass² is 6 out of 10, or 60%. pass⁵ has only one way to pick all 5 runs, and it includes the failed one, so pass⁵ is 0%. In general, with c passing runs out of n, pass^k is the number of ways to pick k of the c passes, divided by the number of ways to pick k of all n runs.

80% sounds fine. But the chance of handling the same request correctly for five customers in a row is zero, based on these runs.

It matters more when comparing agents. Take two agents, each tested on 10 scenarios with 5 runs per scenario. Both pass 40 of their 50 runs, so both score 80% on pass¹. For a whole suite, the lab works out pass^k for each scenario and takes the average. With 5 runs, a scenario’s pass⁵ is 100% if all 5 runs passed and 0% otherwise, so the suite’s pass⁵ is simply the share of scenarios that passed every run:

Agent A’s failures are spread out, so no scenario gets 5 out of 5, and its pass⁵ is 0%. Agent B’s failures are all in two scenarios, so the other eight are perfect, and its pass⁵ is 80%. A is unreliable everywhere. B is reliable except in two places you can find and fix.

Small error rates also add up: an agent that’s right 95% of the time is right 10 times in a row only 60% of the time (0.95¹⁰ ≈ 0.60).

Five runs is also a small sample. They catch a scenario that fails often, but can miss one that fails only now and then. Say the agent gets a scenario wrong 1 time in 10. Each run passes with a 90% chance, so all 5 pass with a 0.9⁵ ≈ 59% chance. More than half the time, the report shows a perfect 5 out of 5 and the problem goes unnoticed. Rare failures only show up with many more runs.

Running every scenario hundreds of times is slow and expensive, so teams usually combine three things:

Grading replies with an LLM judge

The layers so far check what the agent did. Some things only show up in what it said. When the agent escalates a $900 refund, the database can confirm a ticket was opened, but not whether the customer was told why.

Code can’t judge that, so the lab asks another model. An LLM judge is a separate model call that reads the conversation and answers one yes-or-no question, called a criterion:

Each criterion gets its own call, so a failure points to exactly what was missing.

The judge needs testing too

A judge is another model with its own mistakes, so it needs testing before it’s trusted. The lab tests its judge on a set of conversations where the right verdicts are already known.

The first version passed a reply that only said “A colleague will follow up.”, because the criterion said “for example because it’s over the limit”, which made the reason optional. Making the reason required fixed it: the judge then got every verdict right.

So most of the work in calibrating a judge is fixing the wording of the criteria, not changing the model.

A perfect score is a warning sign

The full suite, 26 scenarios × 5 runs on Claude Haiku 5.5, scored 100% on every layer. The whole run cost about 12 cents.

That sounds like good news, but it says very little. If everything passes, the suite can’t tell a better prompt from a worse one: it has hit its ceiling. Showing the agent handles these 26 situations is useful. Finding where it breaks is more useful, and that needs harder scenarios.

Code: agent/scripted.py · evals/scenarios/ · evals/

Part 5: Seeing inside a run, with Langfuse

When an eval run fails, the report says which check failed, and links to that run’s trace. Here’s an example:

$ uv run support-eval run --only refund-over-limit
1 scenarios × 5 runs = 5 runs on claude-haiku-5-5 (effort medium), judged by claude-haiku-5-5
Tracing to Langfuse, session eval-20261026-212459.

  refund-over-limit #1 done (8.5s)
  refund-over-limit #2 done (9.1s)
  refund-over-limit #3 done (12.1s)
  refund-over-limit #4 done (8.8s)
  refund-over-limit #5 done (9.4s)

Judging replies…

scenario           passed  every run  failing checks
refund-over-limit   4/5    no         forbidden tools (1/5)

pass^k (right k times in a row): ^1 80%   ^2 60%   ^3 40%   ^4 20%   ^5 0%
escalation: 5 correct, 0 missed, 0 unnecessary

failed runs:
  refund-over-limit #3: trajectory: forbidden tools
    trace: http://localhost:3000/project/support-agent/traces/b81f0c2e9a4d4e7c9f3a2d61c05e7b18

The report says that run 3 asked for a forbidden refund. To see why, the lab adds tracing with Langfuse, running locally in Docker. Each eval run becomes one trace: every graph step, each model call with its full input and output, each tool call, the guardrail’s decision and the judge’s reasoning. Each run’s results are attached as scores, so it’s easy to filter to the failures.

This is one run of the over-the-limit scenario. The tree on the left mirrors the graph: planner, then agent, guardrail and tools, then agent again, which is the agent loop from earlier. The selected step shows exactly what the model returned on its first pass: a request to check the customer’s balance before deciding anything.

The whole run took about 12 seconds and cost a tenth of a cent. Sorting the request at the start took almost 4 of those seconds, which is hard to guess without a trace.

Code: tracing.py

What this doesn’t cover

• The customer’s lines are scripted in advance. If the agent asks something the script didn’t expect, the customer can’t answer it the way a real person would.
• The judge grades its own model family (Haiku judges Haiku).
• The expected answers are hand-written, so a mistake there lets a wrong agent pass.
• The policy has a gap: nothing stops two $450 refunds in a row.
• One model, 26 scenarios, 5 runs each. It’s a learning lab, not a benchmark.

Key takeaways

1. An agent is a loop around a model call. Most of the work is the code around it.
2. Put the rules in code. The model isn’t deterministic, so use the prompt to explain the rules and the code to enforce them.
3. Treat tools like an API: typed inputs, clear descriptions, and error codes the agent can act on.
4. Let the server decide who the user is. With OAuth, the token, not the prompt, says whose account the agent can touch.
5. Build a fake model on day one. It makes everything testable.
6. Score what the agent did, test your evals and your judge, and measure pass^k.
7. When everything passes, make the tests harder.

Before this lab, I had no idea how to build an agent. Now I understand the loop at its core, how an agent gets its tools and knows who it’s acting for, and, most of all, how to test one and what an eval really is. I hope this write-up is useful if you’re starting out too.

Co-written with Claude: I ran the lab and shaped the post, and Claude helped with the drafting and diagrams.