Building AI that actually works. Evaluation-driven development for engineering teams.https://skylarbpayne.com/https://skylarbpayne.com/posts/ai-for-business/https://skylarbpayne.com/posts/ai-for-business/A Strategic Guide to SuccessMon, 25 Aug 2025 00:00:00 GMThttps://skylarbpayne.com/posts/ai-maturity-model/https://skylarbpayne.com/posts/ai-maturity-model/An AI Maturity ModelThu, 13 Feb 2025 00:00:00 GMThttps://skylarbpayne.com/posts/ai-product/https://skylarbpayne.com/posts/ai-product/Wed, 08 Jan 2025 00:00:00 GMThttps://skylarbpayne.com/posts/bringing-data-science-back-to-ai-engineering/https://skylarbpayne.com/posts/bringing-data-science-back-to-ai-engineering/Thu, 27 Nov 2025 00:00:00 GMThttps://skylarbpayne.com/posts/ai-reliability-patterns/https://skylarbpayne.com/posts/ai-reliability-patterns/Tue, 26 Aug 2025 00:00:00 GMThttps://skylarbpayne.com/posts/data-driven-case-study-1/https://skylarbpayne.com/posts/data-driven-case-study-1/How LinkedIn''s Candidate Recommendation Project Changed My View of Data-Driven DevelopmentMon, 18 Nov 2024 00:00:00 GMThttps://skylarbpayne.com/posts/answer-always-evals/https://skylarbpayne.com/posts/answer-always-evals/Thu, 31 Jul 2025 00:00:00 GMThttps://skylarbpayne.com/posts/data-is-wicked/https://skylarbpayne.com/posts/data-is-wicked/Fri, 30 Aug 2024 00:00:00 GMThttps://skylarbpayne.com/posts/healthrhythms-mood-score/https://skylarbpayne.com/posts/healthrhythms-mood-score/How HealthRhythms built their mood score systemThu, 10 Oct 2024 00:00:00 GMThttps://skylarbpayne.com/posts/leadership-bandwidth/https://skylarbpayne.com/posts/leadership-bandwidth/Why Working Harder Isn''t WorkingSat, 19 Oct 2024 00:00:00 GMThttps://skylarbpayne.com/posts/frigade-ai-better-faster/https://skylarbpayne.com/posts/frigade-ai-better-faster/How Frigade Slashed Latency & Boosted User HelpfulnessTue, 20 May 2025 00:00:00 GMThttps://skylarbpayne.com/posts/job-search-secrets/https://skylarbpayne.com/posts/job-search-secrets/It''s Not What You ThinkTue, 07 Jan 2025 00:00:00 GMThttps://skylarbpayne.com/posts/experimentation-mindset/https://skylarbpayne.com/posts/experimentation-mindset/A Leader''s GuideFri, 01 Nov 2024 00:00:00 GMThttps://skylarbpayne.com/posts/qa-ai/https://skylarbpayne.com/posts/qa-ai/Wed, 26 Mar 2025 00:00:00 GMThttps://skylarbpayne.com/posts/rag-not-dead/https://skylarbpayne.com/posts/rag-not-dead/The Enduring Value of Retrieval in the Era of Expanding Context WindowsSun, 06 Apr 2025 00:00:00 GMThttps://skylarbpayne.com/posts/ai-observability/https://skylarbpayne.com/posts/ai-observability/Mon, 17 Feb 2025 00:00:00 GMThttps://skylarbpayne.com/posts/if-it-has-to-happen-dont-put-it-in-the-prompt/https://skylarbpayne.com/posts/if-it-has-to-happen-dont-put-it-in-the-prompt/When agent instructions turn into all caps rules, the fix is often to move the requirement out of the prompt and into a workflow that can check it.Mon, 15 Jun 2026 19:00:00 GMThttps://skylarbpayne.com/posts/til-annotate-ai-traces/https://skylarbpayne.com/posts/til-annotate-ai-traces/Logs tell you what happened. Annotations tell you what it meant, why it failed, and whether the fix helped.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/evals-that-get-used/https://skylarbpayne.com/posts/evals-that-get-used/A streamlined system for AI evaluation that closes the gap between seeing problems and fixing them.Tue, 17 Mar 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-ai-call-bulkheads/https://skylarbpayne.com/posts/til-ai-call-bulkheads/AI features get scary when prompts, logs, evals, schemas, fallbacks, and product code all live in the same pile. Give the weird part one stable interface so changes have a place to go.Sun, 10 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/synthetic-datagen/https://skylarbpayne.com/posts/synthetic-datagen/Sun, 23 Mar 2025 00:00:00 GMThttps://skylarbpayne.com/posts/til-break-ai-workflows-into-evaluable-steps/https://skylarbpayne.com/posts/til-break-ai-workflows-into-evaluable-steps/One giant prompt can hide five separate jobs. Split the work so each part has a smaller contract and a failure you can actually name.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-evaluate-rag-retrieval-separately/https://skylarbpayne.com/posts/til-evaluate-rag-retrieval-separately/A bad RAG answer does not tell you whether retrieval failed, generation failed, or the product asked an impossible question. Split the blame before fixing anything.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-instrument-ai-calls-before-you-debug/https://skylarbpayne.com/posts/til-instrument-ai-calls-before-you-debug/If an AI answer goes sideways and you cannot see the prompt, model, latency, tokens, retrieved context, and failure path, you are debugging from vibes.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-make-ai-pipelines-safe-to-replay/https://skylarbpayne.com/posts/til-make-ai-pipelines-safe-to-replay/If every eval run emails a customer, updates production state, or fires a webhook, you do not have an eval harness. You have a hostage situation.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/openclaw-gpt-5-4-vs-opus/https://skylarbpayne.com/posts/openclaw-gpt-5-4-vs-opus/Anthropic changed OpenClaw billing. We ran evals, tuned the bootstrap files, and GPT-5.4 got a lot better.Sat, 04 Apr 2026 00:00:00 GMThttps://skylarbpayne.com/posts/pydantic-ai/https://skylarbpayne.com/posts/pydantic-ai/A Story of SimplificationFri, 03 Jan 2025 00:00:00 GMThttps://skylarbpayne.com/posts/til-model-agent-workflows-as-state-graphs/https://skylarbpayne.com/posts/til-model-agent-workflows-as-state-graphs/Agents get less spooky when they have named states, constrained transitions, and a record of how each decision moved the process forward.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-quality-check-rag-chunks/https://skylarbpayne.com/posts/til-quality-check-rag-chunks/Before blaming the model, inspect the chunks. Duplicate, empty, bloated, or low-signal chunks can wreck retrieval quietly.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-record-and-replay-ai-workflows/https://skylarbpayne.com/posts/til-record-and-replay-ai-workflows/When a multi-step AI run fails once and then refuses to fail again, replay beats superstition. Capture the calls, context, and intermediate state.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-put-human-approval-before-risky-agent-tools/https://skylarbpayne.com/posts/til-put-human-approval-before-risky-agent-tools/Agents should not get to delete files, send messages, spend money, publish content, or mutate production just because the next step looks obvious.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/til-validate-rag-citations/https://skylarbpayne.com/posts/til-validate-rag-citations/A citation is not proof just because the model printed a source name. Verify that the source exists and actually supports the claim.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/why-your-team-is-slow/https://skylarbpayne.com/posts/why-your-team-is-slow/The Hidden Cost of Missing Business ContextTue, 29 Oct 2024 00:00:00 GMThttps://skylarbpayne.com/posts/multi-turn-eval/https://skylarbpayne.com/posts/multi-turn-eval/Mon, 01 Sep 2025 00:00:00 GMThttps://skylarbpayne.com/posts/most-important-job-leaders/https://skylarbpayne.com/posts/most-important-job-leaders/Thu, 14 Nov 2024 00:00:00 GMThttps://skylarbpayne.com/posts/mindstrong-data-infra/https://skylarbpayne.com/posts/mindstrong-data-infra/Tue, 11 Oct 2022 00:00:00 GMThttps://skylarbpayne.com/posts/real-life-evals/https://skylarbpayne.com/posts/real-life-evals/A Practical Guide to Evaluation-Driven ImprovementSat, 08 Feb 2025 00:00:00 GMThttps://skylarbpayne.com/posts/ai-latency/https://skylarbpayne.com/posts/ai-latency/Tue, 25 Feb 2025 00:00:00 GMThttps://skylarbpayne.com/posts/til-structure-llm-outputs-at-the-boundary/https://skylarbpayne.com/posts/til-structure-llm-outputs-at-the-boundary/If the rest of your app needs data, make the model return data. Do not make downstream code scrape nice-sounding paragraphs forever.Thu, 28 May 2026 00:00:00 GMThttps://skylarbpayne.com/posts/dspy-engineering-patterns/https://skylarbpayne.com/posts/dspy-engineering-patterns/Any sufficiently complicated AI system contains an ad hoc, informally-specified, bug-ridden implementation of half of DSPy.Sat, 21 Mar 2026 00:00:00 GMThttps://skylarbpayne.com/posts/good-interfaces/https://skylarbpayne.com/posts/good-interfaces/Tue, 01 Apr 2025 00:00:00 GMT