B2B SaaS Applied AI Digital product Backend Anonymous case

Internal AI copilot for a B2B support team

We built an internal copilot with semantic search over the knowledge base, integrated into the support team's workflow, with continuous evaluation and impact metrics.

B2B SaaS ·

38% Resolution time reduction From 25 to 15 minutes average per complex ticket.
84% Team adoption Measured by weekly active usage in the first month.
200 Cases in continuous evaluation Regression set that blocks deploys with quality drops.

Problem

The support team spent 25 minutes on average resolving each complex ticket because information was scattered across wiki, historical tickets and product docs. Onboarding new agents took 6 weeks.

Intervention

We designed a copilot that indexes internal documentation, resolved tickets and runbooks through a RAG pipeline with embeddings. We integrated it as an extension into the support tool they already used (Zendesk). We built an evaluation set with 200 real questions to measure quality and block regressions on every prompt or model deploy.

Outcome

After 12 weeks in production, average resolution time dropped 38% and onboarding time for new agents went from 6 to 3 weeks. Copilot adoption was 84% of the team in the first month.

Stack

Python FastAPI OpenAI pgvector PostgreSQL Zendesk API LangChain

The real problem, not the trendy one

When the client first reached out, they were saying “we need AI”. What they had was a concrete problem: support tickets taking too long and driving turnover in the team.

We started by validating that AI was actually the right solution and not a simple workflow. We interviewed agents and analyzed ticket types. We confirmed a contextual search assistant over proprietary knowledge could solve most of the problem.

Build with guardrails

From day one we built the evaluation set with 200 representative questions. Every change to the prompt, model or pipeline runs against that set and blocks the deploy if quality drops. That gave us confidence to iterate fast without breaking what already worked.

Measure what matters

We defined three metrics before starting: resolution time, adoption and internal satisfaction. We reported all three weekly. That allowed the client to justify the investment and the Caps team to prioritize improvements with measurable impact.

We didn't expect a copilot to actually save us time, but the rigor with which they built and measured it made the difference. Today it's part of onboarding.
Support Lead Head of Customer Support · Anonymous client

Does your challenge look like this one?

Tell us your context. If we've solved something similar, we can skip ahead to the useful conversation.

Book a discovery call View all