How to Actually Evaluate Your LLM (And Stop Guessing)

Your AI is broken. You just don't know how broken yet.

You ship a customer service bot. Metrics look good. Users complain anyway.

Where did it break? Turn 2? Turn 3? All of them?

Each response seems... reasonable? But the conversation is a trainwreck.

You have 200 similar threads. Reading them all will take three days, and you'll miss half the issues anyway.

There's a better way.

The Framework That Actually Works

The process has four steps:

(1) Analyze - understand what's failing by working with domain experts to identify patterns.

(2) Measure - build automated evaluators that catch specific failure modes.

(3) Improve - make targeted fixes and re-run evals to prove they worked.

(4) Repeat - integrate into CI/CD so quality never regresses.

The key insight: You can't measure what you don't understand.

Most teams jump straight to building automated evaluators and miss application-specific failures. Metrics look good. Users stay frustrated.

Let me show you the process that actually works.

Day 0: Define Your Dimensions

Before touching any traces, organize the problem space with dimensions - axes of variation where your AI is most likely to break.

The goal: actively try to break your system by targeting its weakest points.

For our hypothetical SupportBot (a customer service AI), I'd use these three dimensions:

1. Issue Type: Order status inquiry, return/refund request, product question, account issue, complaint/escalation.

2. Customer Context: New customer (first purchase), repeat customer, high-value customer (over $1000 lifetime), angry customer (complaint language).

3. Conversation Complexity: Single-turn (one question, one answer), multi-turn clarification (2-3 exchanges), complex negotiation (4+ turns), context-dependent (references earlier turns).

These aren't arbitrary. Each dimension targets a specific way the system breaks:

• Return requests + Multi-turn means the bot loses track of original intent
• Angry customers + Context-dependent means the bot misses emotional cues
• Order status + Repeat customers means the bot doesn't use history correctly

You're building adversarial test cases on purpose. If your dimensions don't make you nervous about your system, they're not aggressive enough.

Create 20 Test Scenarios (By Hand)

Generate specific test cases from dimension combinations. Focus on scenarios where you expect the system to fail.

The goal is to stress-test your AI, not to prove it works.

Scenario 7: (Return request, Repeat customer, Context-dependent)

Hypothesis: Bot will forget the original issue after 3+ turns of clarification

Test conversation:

Customer: "I need to return my order"
Bot: "I'd be happy to help! Can you provide your order number?"
Customer: "I don't have it handy, I ordered it last Tuesday"
Bot: "I see you have two orders from last Tuesday.
      One for a blue widget ($49.99) and one for red socks ($12.99).
      Which would you like to return?"
Customer: "The widget"
Bot: "Great! I can help you place an order for a widget..."

Expected failure: Bot loses "return" intent after product identification.

Instead, give team members character sheets to roleplay:

ANGRY_CUSTOMER persona:
- Uses frustrated language ("This is ridiculous", "Are you even listening?")
- Expects bot to remember previous statements
- Will escalate if bot repeats itself
- Response pattern: Gets progressively shorter/terser

Real humans create authentic friction that reveals real bugs.

Day 1: Open Coding (4 Hours with Domain Expert)

This is where the magic happens. Sit with your Head of Support and review traces together.

Morning (2 hours): Prep

1. Extract 50 recent conversations from production
2. Quick LLM pass to categorize them for coverage check
3. Select 25-30 diverse conversations

Afternoon (2 hours): Open Coding

Review each conversation. The domain expert writes freeform observations. No categories. No checkboxes. Just "what went wrong?"

Example - Conversation 8 of 30:

Customer: "Where's my order? It's been 2 weeks."

Bot: "I'd be happy to check on that! Can you provide your order number?"

Customer: "I don't have it. Can you look it up by my email?"

Bot: "Sure! I found 3 orders under your email. Which one?"

Customer: "The one from 2 weeks ago, obviously."

Bot: "Your order from 2 weeks ago (Order #7723) shows delivered 5 days ago. The tracking shows it was left on your front porch."

Customer: "I never got it. Someone must have stolen it."

Bot: "I'm sorry to hear that! You can file a claim with the shipping carrier..."

Notes: Bot gave correct info but missed that customer needs a REPLACEMENT, not instructions for filing claims. Should proactively offer replacement when delivery confirmed but customer says they didn't receive.

By conversation 23, your domain expert says: "This is the same pattern as conversations 9, 14, and 19."

That's your signal to stop. When you stop seeing new patterns (usually around 25 conversations), you've hit saturation.

Day 2: Axial Coding (Find the Pattern)

Take all freeform observations and group them.

I print them out. Spread on table. Make piles:

For SupportBot, the complete taxonomy:

1. Intent drift across turns (28%)
2. Information without action (22%)
3. Short-term memory failure (18%)
4. Escalation trigger miss (15%)
5. Product knowledge gaps (10%)
6. Tone mismatch (7%)

Now you know exactly what's broken.

Day 3: Full Labeling Session

Your domain expert labels all conversations (200+) using the taxonomy.

Build a simple UI:

☐ Intent drift
☑ Information without action
☐ Memory failure
☐ Escalation miss
☐ Product knowledge
☐ Tone mismatch

Point of first failure: ○ Turn 1  ○ Turn 2  ● Turn 3  ○ N/A
Conversation outcome: ○ Resolved  ● Unresolved

8 hours later: 203 labeled conversations.

Now you have data: 58% resolution rate overall. The top 3 failures account for 68% of issues. Intent drift happens in conversations over 3 turns. Information without action correlates with returns and refunds.

This quantification tells you what to fix and in what order.

Days 4-5: Build LLM-as-Judge Evaluators

Automate evaluation for top 2 failure modes.

Evaluator 1: Intent Drift Detector

You are evaluating whether a support bot maintained the customer's
original intent throughout the conversation.

Rules:
- FAIL if the customer states an intent (return, refund, track order)
  and the bot later addresses a different intent without explicit
  customer redirect
- PASS if bot stays on task or only switches when customer explicitly
  changes topic

Examples:
[FAIL: Customer wants return, bot switches to new order]
[PASS: Customer wants return, bot completes return flow]
[PASS: Customer changes mind from return to exchange, bot adapts]

Answer only: PASS or FAIL

The key: Never evaluate on data you used for prompt engineering. That's like grading yourself on the practice exam.

When to Use Deterministic vs. LLM Evaluators

Not every evaluation needs an LLM. Choose the right tool for the job:

Real example from SupportBot:

• ✅ Deterministic: Does response include order number when customer asks for order status?
• ✅ Deterministic: Is response under 500 characters?
• ✅ LLM-judge: Did bot maintain the customer's original intent across the conversation?
• ✅ LLM-judge: Is the tone appropriate for an angry customer?

Start with deterministic when possible. Add LLM judges only for criteria humans can judge but code cannot.

Days 6-7: The Fix

You know what's broken. Now fix it.

1. Intent Tracking

# Add explicit intent tracking
conversation_state = {
    "primary_intent": None,  # Set on turn 1
    "confirmed_actions": []   # Log confirmations
}

# Before each response:
if current_topic != conversation_state["primary_intent"]:
    # Flag drift, ask confirmation

2. Action Prompting

Updated system prompt:
"After providing information, ALWAYS suggest next action:
- If order lost: Offer replacement or refund
- If return window open: Start return process
- If product question: Add to cart or compare"

3. Memory Enhancement

# Track entities mentioned
mentioned_entities = {
    "order_number": None,
    "product": None,
    "shipping_address": None
}
# Never re-ask for info already provided

Days 8-9: Re-Evaluation

Replay the original 50 failure conversations through the fixed system.

Before:

• 58% resolution rate
• Intent drift: 28%
• Info without action: 22%
• Memory failure: 18%

After:

• 79% resolution rate (+21 points!)
• Intent drift: 9% (-19)
• Info without action: 8% (-14)
• Memory failure: 6% (-12)

Domain expert spot-checks 30 conversations: "Way better. Bot actually completes tasks now instead of just chatting."

Day 10: CI Integration

Add to your CI pipeline:

# .github/workflows/eval.yml
- name: Run SupportBot Evaluators
  run: |
    python eval/run_evaluators.py \
      --min-resolution-rate 0.75 \
      --min-tpr 0.85 \
      --min-tnr 0.82

Now every prompt change gets evaluated. If metrics regress, build fails.

The Real Lesson: Process Over Tools

What makes this work? The systematic process:

This works for any LLM application:

• Code generation assistants
• Document Q&A systems
• Content moderation
• Data extraction pipelines
• Whatever you're building

The 10-day structure is portable. The specific dimensions change. The process doesn't.

From "It Feels Broken" to "We Fixed These 3 Things"

The second statement leads to action, budget, and trust.

The first leads to "let's try a different model" (which won't fix the real issues).

Start Here

Don't try to implement all 10 days at once.

If those 2 hours reveal patterns you didn't know about, keep going.

If they don't, your AI probably isn't complex enough to need this yet.

Disclaimer: This is a pedagogical exercise. All scenarios, companies, and examples are fictional. I'm under NDA for my actual consulting work. If this sounds like your company, it's pure coincidence.