Evaluating Multi-Turn AI Agents

A Systematic Approach to Building Reliable Agent Systems

Skylar Payne

AI is Fundamentally Empirical

We can’t predict what will work.

• LLMs are highly data-dependent
• Performance varies wildly across use cases
• We have priors (educated guesses)…
• …but we must experiment to find out

This is a critical mindset shift. Unlike traditional software where you can reason about correctness, AI systems require empirical validation.

Key insight: Speed of iteration is your competitive advantage.

Fast experiments → More learning → Better systems → More value

This frames the entire talk: evals are not about “testing” in the traditional sense, they’re about rapid experimentation.

The Challenge: Understanding

Long-running agents are hard to evaluate:

• Cost: Running full eval suites gets expensive fast ($10-50 per test run)
• Latency: Multi-turn conversations take 30s-5min to execute
• Context Overload: Which of the 47 steps actually broke?

Make this visceral - these numbers are real.

The pain: you know something’s wrong, but finding it feels like searching for a needle in a haystack.

Traditional eval approaches (run it end-to-end, check final output) don’t work at this scale.

Time: 3 minutes

Why This Matters

If we can’t evaluate agents systematically:

• We can’t understand when/why they fail
• We can’t improve them reliably
• We can’t maintain them as requirements change
• We can’t ship them with confidence

The solution: Make evaluation straightforward through systematic decomposition.

Connect to engineering values: reliability, maintainability, debuggability.

This isn’t just an AI problem - it’s a software engineering problem. We need the same rigor we apply to traditional systems.

The word “straightforward” is key - we’re making a complex problem tractable through decomposition.

Time: 2 minutes

What We’ll Cover Today

1. The Framework: Evaluation pyramid and decomposition strategies
2. Implementation: Code design, tracing, and system evaluation
3. In Practice: Real results and research validation

Quick roadmap so people know what’s coming. Three main sections.

This also makes the talk easier to follow - they know we’re building toward something.

Time: 1 minute

Part 1: The Framework

Evaluation Pyramid & Decomposition

Section divider - visual break to signal transition.

We’re moving from “why” to “how”.

Time: 10 seconds (just a visual marker)

The Evaluation Pyramid

Not all evals are created equal:

Principle: Test narrow/fast at the bottom, broad/slow at the top

This is the core framework - visualize it clearly.

Testing pyramid from traditional software: unit tests (many/fast) → integration tests → E2E tests (few/slow).

Same principle applies here, but adapted for agent evaluation.

Reference: AgentBoard (2024) emphasizes step-based and trajectory-based evaluation for finer granularity.

Time: 3 minutes

Step-Based Evals: The Foundation

Test individual components in isolation:

• Tool calls: “Did it call the right function?”
• Parameter extraction: “Did it parse user input correctly?”
• Domain logic: “Did it apply business rules?”
• Validation: “Did it catch invalid inputs?”

Why: Fast feedback (<1s), easy to debug, cheap (<$0.01)

Step-based evals are your bread and butter - run them constantly.

70.90% of academic studies rely on static checkpoints at predefined milestones (from multi-turn eval survey). We’re saying: don’t wait for the milestone - test each step.

These are like unit tests but for AI components.

Time: 2 minutes

Step-Based Eval: Code Example

def test_extract_contact_info():
    """Test that we correctly extract email from user message"""
    message = "Please email me at sky@example.com"

    result = extract_contact_info(message)

    assert result.email == "sky@example.com"
    assert result.has_contact_method == True

Fast, deterministic, easy to debug

This is a real unit test. No LLM needed.

If your extraction logic is pure (functional core - we’ll get to that), you can test it with standard assertions.

Run thousands of these in seconds.

Time: 2 minutes

Turn-Based Evals: The Middle Layer

Test single-turn interactions:

• User message → Agent response
• Check partial ordering of tool calls
  • “Read must happen before write”
  • “Verify identity before granting access”
• Validate tool parameters and response quality

Why: Catches integration issues, still relatively fast (~5-10s per test)

Turn-based evals test the agent’s ability to handle one exchange correctly.

Key insight: Check partial ordering constraints - not every possible order, just critical dependencies.

Examples: - File operations: read before write, check existence before delete - Auth flows: verify identity before granting access - Destructive actions: confirm before executing

This level catches problems like: “the tool works, the parsing works, but together they produce nonsense.”

Time: 2 minutes

Turn-Based Eval: Code Example

def test_handle_refund_request():
    """Test agent correctly handles refund request"""
    conversation = Conversation()

    response = agent.handle_turn(
        conversation,
        user_msg="I need a refund for order #12345"
    )

    # Check it called the right tool
    assert response.tool_calls[0].name == "lookup_order"
    assert response.tool_calls[0].args["order_id"] == "12345"

    # Check response quality
    assert "refund" in response.message.lower()
    assert response.needs_user_confirmation == True

Now we’re testing the full turn: user input → agent reasoning → tool calls → response.

This catches integration bugs that unit tests miss.

Still relatively fast because it’s only one turn, not a full conversation.

Time: 3 minutes

Multi-Turn Evals: The Challenge

Test complete conversations:

• Full user journey from start to finish
• Simulate the user - respond to agent naturally
• Multiple turns of interaction
• Context accumulation across turns
• Did the agent achieve the user’s goal?

Problem: Expensive ($1-5 per test), slow (30s-5min), hard to debug

Solution: Break into milestones

Multi-turn evals are expensive - we need to be strategic.

Critical insight: You need to simulate the user responding to the agent. This can be: - Scripted responses (deterministic) - LLM-powered user simulator (more realistic) - Hybrid: scripted with some variation

The key insight: we can’t just run it and check the end result. We need intermediate checkpoints (milestones).

Time: 2 minutes

Milestones: Decomposing Multi-Turn

Instead of: Did the conversation succeed? ✅/❌

Ask: What are the critical checkpoints for this user flow?

• Intent captured?
• Necessary info gathered?
• Tool executed successfully?
• Policy compliance checked?
• Response delivered clearly?

Key insight: Milestones are user-flow specific - different flows need different checkpoints

Milestones are domain-specific checkpoints for a particular user journey.

Critical point: Milestones are NOT universal - they’re specific to each user flow. Password reset has different milestones than booking travel.

Each milestone is something you can eval independently.

This chunking makes evaluation manageable and highlights “where do runs die first.”

Time: 2 minutes

Milestone Example: Password Reset

Flow: User can’t log in → Agent resets password

1. Intent: Understood password issue
2. Identity: Confirmed user identity
3. Diagnosis: Forgot vs locked account
4. Remedy: Choose reset link vs unlock
5. Execute: Send reset successfully
6. Confirm: User can proceed

Each milestone is a gradable checkpoint

Password reset journey with milestones

This is concrete. For any login assistance conversation, you can check: did we pass each milestone?

This is way more actionable than “did the conversation succeed?”

From research: AgentBoard (2024) uses fine-grained progress rate metrics exactly like this.

Now you can eval: “We’re failing at milestone 2 (identity verification) 30% of the time”

Time: 3 minutes

Milestone-Based Evaluation: Code

def evaluate_password_reset(trace):
    """Analyze trace to check milestones"""
    milestones = {
        "intent": check_intent_type(trace, expected="password_reset"),
        "identity": check_verified(trace, "verify_user_identity"),
        "diagnosis": check_tool_result(trace, "check_account_status",
                                       has_field="account_locked"),
        "remedy": check_tool_called(trace, "send_reset_link",
                                    with_param="user_id"),
        "execute": check_attribute(trace, "reset.sent", equals=True),
        "confirm": check_final_state(trace, "user_can_login")
    }

    return MilestoneReport(milestones=milestones,
                          first_failure=find_first_failure(milestones))

Key insight: Check WHAT HAPPENED in the trace, not just that spans exist.

Examples: - Did intent classification return the right type? - Did identity verification succeed? - Did tools get called with correct parameters? - Did attributes get set to expected values?

This is non-intrusive - no changes to your agent code needed. Just query the trace.

Time: 3 minutes

Failure Funnels: Visualizing Drop-Off

Track milestone completion rates across many runs:

Biggest drop = highest leverage fix

This is the failure funnel - it shows you exactly where to focus.

In this example, 28% drop at “Remedy Selected” means that’s your problem.

From your blog post: 42% → 58% success by fixing the biggest leak (malformed amount fields + schema validation).

Don’t try to fix everything - find the biggest drop and fix that first.

Time: 3 minutes

Failure Funnel: Real Example

Password Reset Flow - Initial deployment:

The funnel showed 28% drop at remedy selection

• Problem: Agent chose “unlock account” when should send reset link
• Root cause: Ambiguous phrasing in account status tool response
• Fix: Clarified tool output format + added examples
• Result: 42% → 70% overall success rate

Don’t fix everything - fix the biggest leak first

This is the power of failure funnels - they tell you exactly where to focus.

28% drop at remedy selection meant that was THE problem to solve.

Fixed that one issue → massive improvement across the whole flow.

This is systematic engineering: measure, identify bottleneck, fix, measure again.

Time: 3 minutes

Part 2: Implementation

Code Design, Tracing, & System Evaluation

Section divider - transition to implementation details.

Now we get into the “how” of actually building this.

Time: 10 seconds

Code Design for Testability

Functional Core, Imperative Shell

Functional Core

• Pure functions
• No I/O or LLM calls
• Deterministic
• Easy to test

Imperative Shell

• API calls
• Database writes
• LLM invocations
• Thin layer

Result: 80% of your code is fast to test with step-based evals

This is about architecture - make your system testable by design.

Functional core = domain logic, decision-making, parsing, validation Imperative shell = API calls, database writes, LLM invocations

When you structure it this way, most of your code doesn’t need the LLM to test.

This is a well-known pattern from functional programming, applied to AI systems.

Time: 2 minutes

Before: Tangled Code

async def handle_refund(user_message: str):
    """Everything tangled together - hard to test"""
    # LLM call embedded in business logic
    response = await llm.complete(
        f"Extract order ID from: {user_message}"
    )
    order_id = response.text.strip()

    # Database call embedded
    order = await db.get_order(order_id)

    # Business logic mixed with I/O
    if order.amount > 100:
        await db.create_refund(order_id, order.amount)
        return "Refund processed"
    else:
        return "No refund needed"

Problem: Can’t test business logic without LLM/DB

This is what NOT to do.

Everything is tangled: LLM calls, database calls, business logic all mixed together.

To test the refund logic, you need to mock the LLM and the database.

Tests are slow, brittle, and expensive.

Time: 2 minutes

After: Functional Core + Imperative Shell

# CORE: Pure function, easy to test
def should_process_refund(order: Order) -> RefundDecision:
    """Business logic - no I/O"""
    if order.amount > 100:
        return RefundDecision(
            approve=True,
            amount=order.amount,
            reason="Amount exceeds threshold"
        )
    return RefundDecision(approve=False, reason="Below threshold")

# SHELL: Thin layer for I/O
async def handle_refund(user_message: str):
    """Orchestration - delegates to core"""
    order_id = await extract_order_id(user_message)  # LLM
    order = await db.get_order(order_id)  # DB

    decision = should_process_refund(order)  # CORE - pure!

    if decision.approve:
        await db.create_refund(order_id, decision.amount)  # DB
    return format_response(decision)  # Pure formatting

Functional core and imperative shell architecture

Now the business logic is extracted into a pure function.

You can test should_process_refund with simple assertions - no mocks needed.

The shell is thin and mostly just orchestration. You test it with integration tests.

But 80% of your tests are step-based evals of the pure functions.

Time: 4 minutes

Testing the Functional Core

def test_refund_decision_above_threshold():
    """Fast, deterministic, no mocks"""
    order = Order(id="123", amount=150)

    decision = should_process_refund(order)

    assert decision.approve == True
    assert decision.amount == 150
    assert "threshold" in decision.reason

def test_refund_decision_below_threshold():
    order = Order(id="456", amount=50)

    decision = should_process_refund(order)

    assert decision.approve == False

Runs in milliseconds. No LLM. No database. No mocks.

This is the payoff - tests are simple, fast, and reliable.

You can run thousands of these in seconds.

This is how you get fast iteration: test your logic without the expensive parts.

Then use turn-based and multi-turn evals to test integration.

Time: 2 minutes

Tracing, Not Just Logging

Don’t log tokens. Trace domain events.

• Traces capture structure of execution
• Domain events are stable across model changes
• Traces enable milestone evaluation
• Traces support distributed debugging

This is a key shift: from logging to tracing.

Logging: “here’s what happened” (unstructured) Tracing: “here’s the execution flow with context” (structured)

Domain events: high-level business logic (“user intent classified”, “order retrieved”, “refund approved”)

These are stable - they don’t change when you change the model or prompt.

This is how you build evals that last.

Time: 2 minutes

Bad: Token Logging

async def classify_intent(message: str):
    response = await llm.complete(
        f"Classify intent: {message}"
    )

    # Logging tokens - unstable!
    logger.info(f"LLM returned: {response.text}")

    return response.text

Problem: - Changes every time you update the model - Hard to search/query - No structure for evaluation

This is what NOT to do.

Logging the raw LLM response is not useful for evals.

It changes constantly (model updates, prompt tweaks).

You can’t build stable evals on unstable data.

Time: 1.5 minutes

Good: Domain Event Tracing

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def classify_intent(request: UserRequest) -> Intent:
    with tracer.start_as_current_span("classify_intent") as span:
        # Trace input (domain model)
        span.set_attribute("request.user_id", request.user_id)
        span.set_attribute("request.session_id", request.session_id)

        # Make LLM call
        response = await llm.complete(f"Classify: {request.message}")

        # Parse to domain model
        intent = parse_intent(response.text)

        # Trace output (domain model)
        span.set_attribute("intent.type", intent.type)
        span.set_attribute("intent.confidence", intent.confidence)
        span.set_attribute("intent.entities", intent.entities)

        return intent

Trace structure with domain events vs raw tokens

Now you’re tracing domain events.

The trace captures: what intent was classified, with what confidence, what entities were extracted.

This is stable - it doesn’t change when you update the model.

You can build evals on top of this: “did we classify the right intent?”

OpenTelemetry is one option, but the principle applies to any tracing system.

Time: 3 minutes

Trace-Based Milestone Evaluation

def evaluate_milestone_from_trace(trace):
    """Check if intent recognition milestone was achieved"""
    intent_span = trace.get_span_by_name("classify_intent")

    if not intent_span:
        return False  # Milestone not reached

    # Check domain attributes
    intent_type = intent_span.get_attribute("intent.type")
    confidence = intent_span.get_attribute("intent.confidence")

    # Milestone achieved if intent was classified with high confidence
    return intent_type is not None and confidence > 0.7

Traces make milestones queryable

This is the payoff - you can evaluate milestones directly from traces.

The trace structure tells you what happened.

The domain attributes tell you the outcomes.

You can build your failure funnel by querying traces across many runs.

This is why tracing at the domain level is so powerful.

Time: 2 minutes

Evaluate the System, Not the API

Prompt playgrounds miss the mark

They test:

• “Did the LLM generate good text?”

You need to test:

• “Did the SYSTEM solve the user’s problem?”

The system includes: prompts + tools + business logic + error handling + retry logic + validation

This is a critical mindset shift.

Prompt playgrounds are useful for rapid iteration, but they’re not evals.

Real evals test the entire system behavior, including: - What happens when tools fail - What happens when users provide unexpected input - What happens when rate limits are hit - What happens when data is malformed

Your comms labeling success wasn’t just better prompts - it was better system design (schema validation, fallbacks).

Time: 2 minutes

Part 3: In Practice

Real Results & Research Validation

Section divider - bringing it together with evidence.

We’ve covered framework and implementation, now let’s see proof it works.

Time: 10 seconds

Research Backing This Approach

Recent work supports layered evaluation:

• AgentBoard (2024): Fine-grained progress metrics, step-level analysis
• MT-Eval (2024): Multi-turn interaction patterns (recollection, expansion, refinement, follow-up)
• Survey on LLM Agent Evaluation (2025): 250 sources, emphasizes complementary granular + holistic metrics
• LangSmith Multi-turn Evals (2024): Industry validation of milestone-based approach

The field is converging on multi-level evaluation strategies

Add academic credibility - this isn’t just practical wisdom, it’s backed by research.

70.90% of studies use static checkpoints - we’re enhancing that with the pyramid approach.

The evaluation pyramid, milestones, and failure funnels are emerging as best practices.

Time: 2 minutes

Summary: The Complete System

1. Embrace experimentation - AI is empirical, iterate fast
2. Test at multiple levels - Step → Turn → Multi-turn pyramid
3. Break into milestones - Make multi-turn conversations gradable
4. Use failure funnels - Find and fix the biggest leaks
5. Design for testability - Functional core, imperative shell
6. Trace domain events - Stable foundation for evals
7. Evaluate the system - Not just the LLM

Start small: Pick one capability, build one pyramid

Bring it all together - these seven principles form a complete approach.

Key message: This is systematic, not ad-hoc. You can ship agents with confidence.

Emphasize “start small” - don’t try to eval everything at once.

Pick your most important user journey and build the pyramid for that.

Time: 2 minutes

Resources

Blog post:

• https://skylarbpayne.com/posts/multi-turn-eval/

Research Papers:

• Survey on Evaluation of LLM-based Agents
• Evaluating LLM-based Agents for Multi-Turn Conversations
• AgentBoard Framework
• MT-Eval Benchmark
• TheAgentCompany Benchmark
• LangSmith Multi-turn Evals

Offer resources for deeper learning.

These papers provide the academic foundation.

Blog post has practical examples and code.

Invite questions!

Common questions to prepare for: - “How do you define milestones?” → Use domain events - “What about non-deterministic behavior?” → That’s why you need statistical evals across many runs - “What tools?” → OpenTelemetry for tracing, any test framework for assertions - “How do you handle evolving requirements?” → Domain events are more stable than implementation details

Time: Remaining time for Q&A

Questions?

Contact me: me@skylarbpayne.com