40% Better, 75% Faster

Frigade wanted its AI assistant to show users how to complete tasks, rather than only point them to documentation. Users were reporting unhelpful responses and slow performance, but the team needed a way to diagnose those problems and compare changes.

Wicked Data worked with Frigade to build an evaluation process around representative queries, recorded traces, and human annotations. That process helped the team identify retrieval problems and measure the effect of changes to response delivery.

Building the evaluation process

The process included:

1. Representative Query Sets: Curated queries and inputs reflecting genuine user scenarios.
2. Automated System Testing: A custom CLI script to automate query testing against live system versions.
3. Trace Recording: Braintrust integration for detailed AI interaction trace recording, vital for AI observability.
4. Annotation & Acceptability Scoring: A Braintrust-configured process for annotating traces to assess AI response acceptability.
5. Performance Metrics & Reporting: Custom CLI scripts to download annotated logs and compute key metrics, creating a version-controlled “performance snapshot” in Git for trend analysis.

For more on trace recording, see AI Observability: You can’t fix what you can’t see.

The reports tracked:

• Input/Output Tokens
• Latency (notably Time to First Record)
• Percent Acceptable Responses
• Document Relevance
• Detailed breakdowns by response type, customer, and other dimensions.

Changes to the system

Reviewing the evaluation results led to changes in:

• Document Chunking and Indexing: Optimizing information processing for superior retrieval.
• Document Embedding: Enhancing metadata to improve query-document matching.
• User-Perceived Latency: Transitioning to streaming responses and prioritizing “time to first record,” significantly boosting user experience. Strategies for tackling such challenges are explored in “Understanding and Addressing AI Latency.”
• Structured Output Format: Ensuring more dependable integration and predictable AI outputs.
• Guidelines and Instructions: Refining system prompts for enhanced consistency and accuracy.

Results

The engagement produced the following improvements:

• 40% increase in response acceptability.
• 35% increase in queries retrieving relevant documents.
• 75% reduction in user-perceived latency.
• 70% increase in the accuracy of rejecting queries outside the assistant’s scope.

Handoff to the team

The work included mentoring Frigade’s engineers on the evaluation process so they could review new failures and continue testing changes themselves.

The query sets, annotations, and versioned performance snapshots gave the team a way to compare changes over time. The implementation details matter: the CLI made repeated testing easier, while the annotated traces helped explain why a response was unacceptable.

Work with me

If your team needs help building an evaluation process around its own data and codebase, see the engagement options.

Schedule a free consultation to discuss your team’s specific challenges →

Get new essays by email →