LLMs and voice

Clinic voice agent

An AI phone receptionist: Twilio Media Streams, Pipecat, Deepgram speech and Claude tool use, with guardrails screening every caller turn before the LLM.

Charts of voice-to-voice latency by call type

The problem

Most calls to a clinic are administrative: opening hours, bookings, cancellations. A few are emergencies or requests for medical advice, and those must never depend on what a language model decides. On a phone line the reply also has to come fast enough to feel like a conversation.

What I built

  1. Connected a Twilio number through Media Streams to a FastAPI WebSocket, where Pipecat runs speech-to-text, the agent and text-to-speech with turn detection and barge-in.
  2. Put rule-based guardrails in front of the LLM. Emergencies get a fixed script and a staff alert, and requests for diagnoses or doses are refused.
  3. Gave Claude typed tools for FAQ search, free slots, booking, rescheduling, cancelling and messages. A validating executor only books after the caller confirms the read-back.
  4. Supported Deepgram or offline Whisper and Kokoro, and wrote a fake-Twilio simulator that calls the real server end to end.

Results

Guardrails and retrieval on small hand-written text sets, and simulated calls through the real server with synthetic voices.
CheckSetResult
Emergency recall, rules25 dev lines1.00
Emergency recall, rules20 held-out lines0.45
False alarms on admin requests20 held-out0
FAQ retrieval, own BM2536 caller questionsHit@1 0.83, Hit@3 0.94
Simulated calls, Deepgram Flux + Aura20 calls20 / 20 right script, 0.79 s median
Simulated calls, Whisper + Kokoro on the CPU20 calls19 / 20 right script, 2.24 s median

What the results show

  • The rules look perfect on the lines they were tuned on and miss more than half of new emergencies, such as "I can't catch my breath". Keyword rules alone are not enough.
  • Speech is not the safety bottleneck. No Deepgram transcript changed a guardrail decision (108 / 108), and errors cluster in names, birth dates and phone numbers, which is why the agent reads them back.
  • Latency is mostly turn detection. Emergency statements wait about twice as long for the end of turn as questions do (median 1.01 s vs 0.51 s).
  • A stricter end-of-turn threshold of 0.8 cuts early cut-offs from 5 to 3 of 108 lines but adds about 0.25 s to the median reply, so it stays at 0.7.

Limitations

  • The Claude conversation agent and its 13 scripted calls have not been run, because they require an Anthropic API key. None of the numbers above measure the agent.
  • Medical-advice recall on held-out lines is only 0.47, and all test sets are small and written by me.
  • Voices are synthetic and clean. Real callers bring accents, noise and panic.

Full results, commands and figures are in the repository README.