Case Study · Feb 2025
Shipping a resilient retrieval pipeline in 10 days
By Alex Monroe · 8 minute read
When a global retailer asked for a multilingual assistant in under two weeks, we knew the retrieval layer would make or break the launch. Here’s how we stabilized a noisy product feed and kept hallucinations at bay.
Start with the failure budget
We sat down with merchandising and support leads to map the worst-case scenarios. That produced a simple rule: every response needed at least two corroborating documents, and stale inventory was unacceptable. That clarity shaped everything downstream.
“Design the guardrails early and you can move faster later. The lack of surprises is the win.”
Taming the feed
We wrapped the upstream feed in a reconciliation job that compared product deltas against a normalized warehouse. Anything suspicious landed in a review queue with a one-click correction workflow.
- Automatic detection of duplicate SKUs and conflicting metadata.
- Time-to-live tags on every document with a background revalidation job.
- Snapshot testing against a golden set of user journeys.
The result was a retriever that aged gracefully, even when upstream systems wobbled.
Eval before you ship
Before launch we ran hourly evals with production prompts. The dashboard above surfaced any regression within minutes. When legal updated policies mid-launch, we swapped in a new policy pack and watched the scores settle in real time.
What it unlocked
Day one support tickets dropped by 35%. More importantly, the merchandising team trusted the assistant to stay on-script. With the groundwork in place we layered on automated translation and spec generation in the second week.
Key takeaways
- Codify failure budgets with the business before you architect anything.
- Stabilize upstream data with reconciliation jobs and fast correction loops.
- Invest in evals and observability so changes stay safe.
Want to go deeper? Drop us a note at hello@lakeshore.design.