Frigade, an innovator in AI-powered in-product assistance, aimed to solve the pervasive problem of user frustration within complex enterprise software. Partnering with Wicked Data, Frigade implemented a rigorous evaluation-driven development framework. This collaboration led to substantial enhancements in AI performance, making it 40% Better in terms of response acceptability and 75% Faster through reduced user-perceived latency. The engagement not only elevated Frigade’s product but also equipped their team with the methodologies to independently drive future AI advancements.
The Challenge: Navigating Complexity in Enterprise Software
Enterprise software offers immense power but often at the cost of complexity. Users, focused on their core job responsibilities, frequently struggle with:
- Locating necessary features or information.
- Deciphering outdated or insufficient documentation.
- Generic responses from unhelpful chatbots.
- Protracted wait times for human support.
This common scenario breeds inefficiency, user anxiety, and overall dissatisfaction. Frigade envisioned a better way: an intelligent assistant embedded directly within the product, capable of understanding its nuances and providing instant, contextual guidance. The hurdle was engineering an AI that could genuinely learn and assist – a persistent challenge in the industry.
The Solution: Frigade AI, Supercharged by Wicked Data’s Expertise
Frigade’s ambition was an AI assistant that actively shows users how to achieve tasks, rather than merely pointing to documentation. To realize this vision and accelerate development, Frigade collaborated with Wicked Data. Wicked Data provided deep expertise in building AI-enabled applications and, critically, a proven methodology for rapidly delivering high-quality AI quickly.
The Key Unlock: Evaluation-Driven Development & AI Observability
The engagement’s cornerstone was implementing an evaluation-driven development (EDD) methodology. Given the multifaceted nature of complex AI systems, where numerous elements can falter, a “vibes-based” development approach is inefficient. EDD provides a systematic path to measure progress and concentrate efforts effectively. Essential to this is comprehensive AI observability; as detailed in “AI Observability: You can’t fix what you can’t see,” deeply understanding system behavior is non-negotiable.
Wicked Data instituted a robust evaluation process including:
- Representative Query Sets: Curated queries and inputs reflecting genuine user scenarios.
- Automated System Testing: A custom CLI script to automate query testing against live system versions.
- Trace Recording: Braintrust integration for detailed AI interaction trace recording—vital for AI observability.
- Annotation & Acceptability Scoring: A Braintrust-configured process for annotating traces to assess AI response acceptability.
- Performance Metrics & Reporting: Custom CLI scripts to download annotated logs and compute key metrics, creating a version-controlled “performance snapshot” in Git for trend analysis.
This framework facilitated granular analysis of metrics including:
- Input/Output Tokens
- Latency (notably Time to First Record)
- Percent Acceptable Responses
- Document Relevance
- Detailed breakdowns by response type, customer, and other dimensions.
5. From Insights to Impact: Iterative Improvements
This robust evaluation system enabled Wicked Data and Frigade to meticulously “follow the breadcrumbs” in the data, pinpointing critical areas for enhancement. This data-centric approach spurred improvements in:
- Document Chunking and Indexing: Optimizing information processing for superior retrieval.
- Document Embedding: Enhancing metadata to improve query-document matching.
- User-Perceived Latency: Transitioning to streaming responses and prioritizing “time to first record,” significantly boosting user experience. Strategies for tackling such challenges are explored in “Understanding and Addressing AI Latency.”
- Structured Output Format: Ensuring more dependable integration and predictable AI outputs.
- Guidelines and Instructions: Refining system prompts for enhanced consistency and accuracy.
Measurable Results: A Leap in AI Performance & User Experience
The systematic application of EDD and targeted refinements yielded significant, quantifiable outcomes, making Frigade’s AI 40% Better and 75% Faster:
- 40% increase in response acceptability, ensuring users receive consistently helpful and accurate guidance.
- 35% increase in queries retrieving relevant documents, making the assistant demonstrably more knowledgeable.
- 75% reduction in user-perceived latency, delivering faster assistance.
- 70% increase in the accuracy of rejecting queries outside the assistant’s scope, maintaining focus and reliability.
These advancements directly translated into a vastly improved user experience, minimizing frustration and empowering Frigade’s users to achieve their objectives with greater ease and speed.
Empowering Frigade for Ongoing Success: Mentorship and Knowledge Transfer
Beyond the immediate technical deliverables, Wicked Data emphasized mentoring and knowledge transfer to the Frigade team. This engagement armed Frigade with the requisite skills and processes to independently manage their evaluation cycles. Consequently, Frigade’s team can now harness their intrinsic product understanding to generate new insights and continuously iterate on their AI system.
Conclusion: Build Better AI, Faster, with Wicked Data Frigade’s success story vividly illustrates the synergy between an innovative AI vision and a rigorous, evaluation-driven development methodology. Wicked Data furnished the expertise and framework not only to construct a superior AI assistant but also to embed best practices crucial for sustained, long-term success.
If your engineering team is aiming to deliver high-caliber AI systems and you wish to accelerate their proficiency and achieve significant results swiftly, let’s connect. Wicked Data can empower you with the strategies and processes to reach your goals.
Start today with a free consult: Book