Skip to main content
These guides connect a phone call directly to a speech-to-speech model: a single model that listens and speaks natively, with no intermediate text step. One model handles the full conversation, including speech recognition, understanding, response generation, and voice synthesis, as well as the conversational mechanics a voice agent needs: voice activity detection, turn handling, and barge-in when the caller interrupts. Configuration is minimal. You pick a model, choose a voice, and write instructions.
PipelineSpeech-to-Speech
  1. Phone Call
  2. Plivo
    WebSocket
  3. Your ServerRelay
    WebSocket
  4. Speech-to-Speech ModelRealtime API
Your server sits between two WebSocket connections: Plivo streams the caller’s audio over one, and the model’s realtime endpoint is on the other. The server relays audio between the two and translates each side’s message format. In most cases the audio itself passes through untouched.

​
Why Speech-to-Speech

  • Lowest latency: one model turn instead of three chained stages
  • Natural conversations: tone, emotion, and context survive because audio is never flattened to text
  • Simple architecture: turn handling and interruption live inside the model, so your server stays small

​
Trade-Offs

  • Observability: there is no text between stages to inspect. You depend on the transcripts the model chooses to emit, which makes debugging, evaluation, and monitoring harder than with a pipeline that produces text at every hop.
  • Steerability: instruction adherence in audio-native models is generally less mature than in text LLMs. A pipeline lets you use a proven text model whose prompting behavior you already understand.
  • Tool calling: function calling from a speech-to-speech model can be less consistent than from a text LLM, which matters if your agent depends on reliable structured actions.
  • Component choice: you adopt the model as a whole. You cannot swap in a different LLM or a specific voice vendor, and pricing is bundled rather than per stage.
If these trade-offs matter more than latency for your use case, you have three alternatives, in increasing order of control: Voice Agent Pipelines are the fastest to adopt but tie you to one provider’s stack; Pipecat lets you compose the STT, LLM, and TTS services you choose within an open-source framework; and a fully custom pipeline, where you build the orchestration yourself on the raw audio stream, gives you complete ownership of every stage.

​
Other Ways to Build

  • Voice Agent Pipelines: provider-hosted STT, LLM, and TTS stages behind one realtime endpoint, when you want control over each stage
  • Pipecat: an open-source framework that manages the Plivo transport and composes services, including speech-to-speech models, in code