The challenge

The platform already had a session-summary prompt running on Azure OpenAI, but outputs were inconsistent: wrong structure, wrong voice (coach feedback drifting into third person), missing sections, and no way to test changes because there were very few real transcripts. A client demo was days away. Cost per session mattered as much as quality, because the model runs on every coaching conversation as the business scales.

What we built

• Layer 1 — Session Summary Engine — Rewrote the coach, user and admin summary prompts into a shared context block plus role-specific instructions, fixing voice, structure and the missing coach-feedback section. Human-readable narrative only — no JSON leaking into the UI.
• Layer 2 — Coaching Insights — Moved goal tags, pain tags, tone and contradiction detection out of free-text prompts into deterministic library-based extraction plus a strict JSON schema, so quadrant scores and trend data are valid every run and cheap to compute.
• Layer 3 — Coaching Plan Engine — A plan generator that references Layer 2 tags and scores explicitly, producing structured, compliant plans coaches can edit rather than rewrite.
• Layer 4 — QA & Compliance — An automated pass that scores every output for framework alignment, safe-zone language and coaching quality before it reaches a coach or user.
• Test-transcript programme — Synthetic transcripts across personas (stress, retirement anxiety, property, cost-of-living pressure) with built-in triggers the system must flag, and a pass/fail log for every prompt iteration.
• Integration pack for the in-house developer — Final prompt library, Layer 2 schema, how to load context blocks in Azure, where to enforce JSON, where to log contradictions, and the mapping to coach/admin UI fields.
• Millie — product design — A phased design for the assistant layer: rule-based coach assistant first, adaptive engagement driver second, agentic orchestrator third — each phase gated on data maturity and QA.

How we worked

Started as a short prompt-engineering audit under a hard deadline and grew into an ongoing engineering partnership. Every prompt change shipped with before/after samples; every layer got a documented input/output contract so the platform team could integrate without guessing.

Outcome

Consistent, correctly structured summaries in time for the client demonstration, a repeatable pipeline where each layer feeds the next through validated JSON, and a testing method that lets the team change prompts with confidence instead of hope. The engagement expanded to full-time product engineering on the platform.