Deepgram Voice Agent Integration

The Deepgram Voice Agent API runs speech-to-text (STT), an LLM, and text-to-speech (TTS) over one WebSocket connection. You choose the model for each stage. Plivo provides the telephony, and a small bridge server that you run connects the two. This guide uses a ready-made bridge from the python-agents-examples repository. It handles inbound and outbound calls, interruptions, function calls, and hangup.

Prerequisites

Requirement	Description
Plivo account	Sign up and copy the Auth ID and Auth Token from the Plivo console.
Phone number	A voice-enabled Plivo number.
Deepgram API key	A key with Voice Agent API access from the Deepgram console.
Python	Python 3.10 or later, and uv.
Tunnel	`cloudflared` or `ngrok` to expose your local server.

Quick start

Clone the example

Download only the example’s folder:

git clone --depth 1 --filter=blob:none --sparse https://github.com/plivo/python-agents-examples.git
cd python-agents-examples
git sparse-checkout set deepgram-voiceagent
cd deepgram-voiceagent
cp .env.example .env

This needs Git 2.25 or later. To download the whole repository instead, run git clone https://github.com/plivo/python-agents-examples.git.In .env, set DEEPGRAM_API_KEY, PLIVO_AUTH_ID, PLIVO_AUTH_TOKEN, and PLIVO_PHONE_NUMBER.

Start the server with a tunnel

Plivo needs a public HTTPS URL to reach the server. Use a Cloudflare quick tunnel, or ngrok if you already use it.

Cloudflare tunnel
ngrok

uv run python -m inbound.server --tunnel

The --tunnel flag starts a Cloudflare quick tunnel. You don’t need a Cloudflare account. A quick tunnel gets a new URL each time you start the server.Plivo accepts a new tunnel URL only after its hostname resolves, which can take about a minute. The server retries until Plivo accepts the URL.On startup, the server:

Creates or updates a Plivo application named Deepgram_VoiceAgent, with the tunnel URL as its Answer URL.
Links your Plivo number to that application.

Start ngrok in a separate terminal:

ngrok http 8000

In .env, set PUBLIC_URL to the HTTPS URL that ngrok prints. Then start the server:

uv run python -m inbound.server

On startup, the server:

Creates or updates a Plivo application named Deepgram_VoiceAgent, with PUBLIC_URL as its Answer URL.
Links your Plivo number to that application.

Call the number

Before you call, wait for the server to log a line like this, with your Plivo number:

Ready! Call +14150000001 to talk to the agent (Ctrl+C to stop)

Until then, your number isn’t linked to the server, and calls don’t reach the agent.

Call your Plivo number. The agent greets you. Ask it a question, and try talking while it speaks to check that it stops.

Place an outbound call

The example has a separate server for outbound calls. It runs on port 8001, so you can start it in a second terminal while the inbound server keeps running on port 8000.

Cloudflare tunnel
ngrok

uv run python -m outbound.server --tunnel

The outbound server starts its own tunnel, because a tunnel forwards to one local port. It doesn’t change your number’s settings.

Start a second ngrok tunnel for port 8001:

ngrok http 8001

Then start the outbound server with that tunnel’s HTTPS URL:

PUBLIC_URL=https://<second-ngrok-host> uv run python -m outbound.server

The outbound server doesn’t change your number’s settings.

When the server is ready, it logs a Ready! line with a cURL command for the Plivo Create a Call API. The command contains your Plivo number and the server’s Answer URL. Set to to the number to call, and replace <auth_id> and <auth_token> with your Plivo credentials:

curl -X POST "https://api.plivo.com/v1/Account/<auth_id>/Call/" \
  -u '<auth_id>:<auth_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "from": "+14150000001",
    "to": "+14155550123",
    "answer_url": "https://<your-tunnel-host>/outbound/answer?opening_reason=you%20requested%20a%20demo",
    "hangup_url": "https://<your-tunnel-host>/outbound/hangup",
    "answer_method": "POST",
    "hangup_method": "POST"
  }'

With a Cloudflare tunnel, if Plivo returns Must be a valid url, the new tunnel hostname doesn’t resolve yet. Wait a minute and run the command again.

How it works

PipelineDeepgram Voice Agent

Phone Call
PlivoAudio Stream
WebSocket
Your ServerBridge
WebSocket
Deepgram Voice AgentSTTLLMTTS

The bridge server has two parts:

The Answer URL. When a call connects, Plivo requests this URL. The server returns <Stream> XML that tells Plivo where to stream the call audio.
The WebSocket server. Plivo streams the call audio here. For each call, the server opens a separate WebSocket connection to Deepgram and passes messages between the two.

You don’t need to change how the example handles calls. The sections below explain what happens during a call, which helps when you read logs or debug a problem. The Plivo messages are described in the Audio Streaming protocol, and the Deepgram messages in the Voice Agent API documentation.

The call starts

When a call connects, Plivo requests the server’s Answer URL: /answer for inbound calls, or /outbound/answer for outbound calls. The server returns this XML:

<Response>
  <Stream bidirectional="true" keepCallAlive="true" contentType="audio/x-mulaw;rate=8000">wss://your-host/ws</Stream>
</Response>

Plivo then streams the call audio to the server’s /ws WebSocket.The server connects to Deepgram at wss://agent.deepgram.com/v1/agent/converse. This URL is the same for every agent. Your API key selects your Deepgram project, and the first message on the connection selects the agent:

Deepgram sends Welcome.
The server sends one Settings message. It describes the agent and the audio format.
Deepgram replies with SettingsApplied.

{
  "type": "Settings",
  "audio": {
    "input": {"encoding": "mulaw", "sample_rate": 8000},
    "output": {"encoding": "mulaw", "sample_rate": 8000, "container": "none"}
  },
  "agent": {"listen": {}, "think": {}, "speak": {}, "greeting": "Hi, how can I help?"}
}

The agent field contains the full agent definition, or the ID of an agent saved in your Deepgram project. See Saved agent configurations. The server opens a new Deepgram connection for each call, so it can choose a different agent for each call.Plivo streams audio as mulaw at 8 kHz. The settings ask Deepgram for the same format in both directions, so the server passes audio through without converting it. If you change the audio settings, keep them at mulaw, 8000, and container: "none".Deepgram ignores audio that arrives before SettingsApplied. The server stores the caller’s audio until then and sends it afterward.

For EU data processing, set DEEPGRAM_AGENT_URL=wss://api.eu.deepgram.com/v1/agent/converse in .env.

The caller interrupts the agent

When the caller speaks while the agent is talking, Deepgram stops the agent’s reply and sends UserStartedSpeaking. Part of the reply may already be waiting to play at Plivo. The server sends this message to Plivo to clear it:

{"event": "clearAudio", "streamId": "<stream_id>"}

The server also discards any remaining audio from the interrupted reply. See _on_user_started_speaking() in inbound/agent.py.

The agent calls a function

The example’s functions run on your server. Deepgram sends a FunctionCallRequest, the server runs the function, and it returns the result in a FunctionCallResponse:

{"type": "FunctionCallRequest", "functions": [{"id": "fc_12345", "name": "check_order_status", "arguments": "{\"order_number\": \"TF-123456\"}", "client_side": true}]}
{"type": "FunctionCallResponse", "id": "fc_12345", "name": "check_order_status", "content": "{\"status\": \"shipped\"}"}

The LLM uses the content in its reply. See _on_function_call_request() in inbound/agent.py.

The agent ends the call

The server waits for the agent’s goodbye to finish playing before it hangs up:

The LLM calls the end_call function and then says goodbye.
When Deepgram sends AgentAudioDone, the server sends Plivo a checkpoint after the last audio chunk:

{"event": "checkpoint", "streamId": "<stream_id>", "name": "turn_3_4"}

Plivo sends playedStream with the same name after all audio before the checkpoint has played.
The server hangs up with the Plivo Hang up a call API.

If Plivo doesn’t send playedStream within 15 seconds of end_call, the server hangs up anyway.

The caller presses a key

Plivo sends each key press as a dtmf event that contains the digit. The example doesn’t handle these events yet. To let the agent respond to key presses, add code to _receive_from_plivo() in inbound/agent.py that sends the digit to Deepgram as text:

{"type": "InjectUserMessage", "content": "I pressed 5 on my keypad."}

Customize the agent

The example agent is a customer support agent for a sample company called TechFlow. Inbound and outbound calls use separate agents. This table shows where to change each one:

To change	Inbound	Outbound
Instructions	`inbound/system_prompt.md`	`outbound/system_prompt.md`
Greeting	`AGENT_GREETING` in `.env`	`opening_reason` parameter
Models	`DEEPGRAM_*` in `.env`	Same as inbound
Functions	`inbound/agent.py`	`outbound/agent.py`

Restart the server after each change. The agent says the greeting exactly as written, so don’t put instructions in it.

Call details

For each call, the server adds details about the call to the agent’s instructions. Inbound calls. The server adds the caller’s number, the call ID, and the current time, from Plivo’s request to the Answer URL. The agent can use the caller’s number for SMS or callbacks without asking for it. Outbound calls. You describe each call with optional query parameters on the Answer URL. URL-encode each value.

Parameter	Used for	If left out
`opening_reason`	The greeting and the call’s topic	Follow-up on recent interest
`objective`	What the agent tries to achieve	Qualify interest, offer a meeting
`context`	Extra details for the agent	No extra details

The server uses these parameters as follows:

It builds the greeting from opening_reason: “Hi, this is Alex from TechFlow. I’m reaching out because opening_reason. Is now a good time for a quick chat?” Without opening_reason, the agent uses a default greeting.
It fills these placeholders in outbound/system_prompt.md: {{greeting}} with the greeting the agent spoke, and {{opening_reason}}, {{objective}}, and {{context}} with the parameter values.
It adds the customer’s number, your caller ID, the call ID, and the current time, from Plivo’s request to the Answer URL.

The example reads only these three parameters. To pass other details, change CALL_DETAIL_PARAMS in outbound/server.py and the code in outbound/agent.py that builds the instructions.

Choose your models

Set the models in .env. Each value is a Deepgram model ID, and the same values apply to inbound and outbound calls. These are the defaults:

DEEPGRAM_LISTEN_MODEL=flux-general-en
DEEPGRAM_THINK_PROVIDER=open_ai
DEEPGRAM_THINK_MODEL=gpt-4.1-mini
DEEPGRAM_SPEAK_MODEL=aura-2-thalia-en

Deepgram runs all the models below, so you don’t need an account or key with the model’s provider. Speech-to-text

`DEEPGRAM_LISTEN_MODEL`	Notes
`flux-general-en`	Default. English.
`flux-general-multi`	Multilingual.
`nova-3`	Also set `DEEPGRAM_LISTEN_LANGUAGE`, for example `en`.

LLM

`DEEPGRAM_THINK_PROVIDER`	`DEEPGRAM_THINK_MODEL`	Pricing tier
`open_ai`	`gpt-4.1-mini`	Standard. Default.
`open_ai`	`gpt-5.4-mini`	Standard
`open_ai`	`gpt-5.5`	Advanced
`anthropic`	`claude-haiku-4-5`	Standard
`anthropic`	`claude-sonnet-5`	Advanced
`google`	`gemini-2.5-flash`	Standard

Text-to-speech Replace <voice> with a voice name. The default is aura-2-thalia-en.

`DEEPGRAM_SPEAK_MODEL`	Voices for customer service and IVR
`aura-2-<voice>-en`	`thalia`, `andromeda`, `helena`, `arcas`
`flux-<voice>-en`	`alexis`, `haley`, `miles`, `colin`

Aura-2 also has voices in Spanish and other languages. Flux TTS voices speak English with several accents, including Indian English voices such as meena and naveen. For every option, see Deepgram’s STT models, LLM models, Aura voices, and Flux TTS voices.

Add your own functions

The example’s functions return sample data. Each function is defined in two places in inbound/agent.py:

A JSON schema in FUNCTION_DEFINITIONS. The LLM reads it to decide when to call the function.
A branch in _handle_function_call() that runs the function.

This is the schema for the order lookup:

{
    "name": "check_order_status",
    "description": "Look up the status of a customer's order.",
    "parameters": {
        "type": "object",
        "properties": {
            "order_number": {
                "type": "string",
                "description": "Order number (usually starts with TF-)",
            },
            "email": {
                "type": "string",
                "description": "Customer's email if order number unavailable",
            },
        },
    },
},

To connect your own system, replace the code in the matching Python function, such as check_order_status(), with a call to your API. To add a function, add a schema and a branch in both inbound/agent.py and outbound/agent.py. Keep the end_call function, because the agent uses it to hang up.

Go live

Deploy the server

A server that runs on your computer stops when your computer sleeps or shuts down. For production, run the server on a host that stays online and has a fixed HTTPS URL. The example includes a Dockerfile for this:

docker build -t deepgram-voiceagent .
docker run --env-file .env -e PUBLIC_URL=https://your-host.example.com -p 8000:8000 deepgram-voiceagent

On startup, the server creates or updates the Deepgram_VoiceAgent Plivo application with PUBLIC_URL as its Answer URL, and links PLIVO_PHONE_NUMBER to it. To run the outbound server, use port 8001, its own public URL, and the outbound command:

docker run --env-file .env -e PUBLIC_URL=https://outbound.example.com -p 8001:8001 deepgram-voiceagent \
  uv run python -m outbound.server

The Docker image doesn’t include cloudflared, so --tunnel doesn’t work inside it. For local development, run the server with uv as shown in the Quick start. Choose a host that keeps a WebSocket connection open for the whole call:

Host	Supported
Fly.io, Render, Railway, a VM, or Kubernetes	Yes
Google Cloud Run	Yes. Set the request timeout longer than your longest call, up to 60 minutes.
Vercel or Netlify functions, AWS Lambda	No. These can’t keep a WebSocket open for a whole call.

Verify requests from Plivo

The example validates the signature on each Plivo webhook to verify that the request came from Plivo. It uses your PLIVO_AUTH_TOKEN for this. See Signature validation for how Plivo signs requests. For validation to succeed:

Set PUBLIC_URL to exactly the URL that Plivo calls, including https://.
If the number or the call belongs to a Plivo subaccount, use that subaccount’s Auth Token.

Connect more numbers

The server links one number, PLIVO_PHONE_NUMBER. To send calls from other numbers to the same agent, link each number to the Deepgram_VoiceAgent application.

API
Console

Refer to Update an Account Phone Number for all parameters.

curl -X POST "https://api.plivo.com/v1/Account/<auth_id>/Number/<your_plivo_number>/" \
  -u '<auth_id>:<auth_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "app_id": "<app_id>"
  }'

import os
import plivo

client = plivo.RestClient(os.environ["PLIVO_AUTH_ID"], os.environ["PLIVO_AUTH_TOKEN"])
client.numbers.update(number="<your_plivo_number>", app_id="<app_id>")

You can find the app_id of Deepgram_VoiceAgent in the console under Voice > Applications, or with the Application API.

On the Numbers page of the console, under Your Numbers, click the number.
In the Number Configuration box, select XML Application from the Application Type drop-down.
Select Deepgram_VoiceAgent, then click Update Number.

To use a different agent for some numbers, run a separate server for each agent, each with its own Plivo application.

Make outbound calls

Place outbound calls with the Plivo Create a Call API, as in the Quick start. Set answer_url to the outbound server’s /outbound/answer URL, and add the call’s details as query parameters. See Call details. Plivo includes these parameters in the request signature. If a proxy between Plivo and the server changes the query string, signature validation fails. The outbound server doesn’t store calls. To check a call’s status, use the Plivo Call API or the Plivo logs. The optional hangup_url, /outbound/hangup, logs each call’s duration and hangup cause.

Saved agent configurations

By default, the server sends the full agent definition to Deepgram for every call. You can instead save the agent in your Deepgram project and send only its ID. This is useful when several servers must run the same agent, or when you want to switch back to an earlier version by changing the ID. Create the saved agent once with Deepgram’s REST API. The example’s README has ready-to-run commands that save the example’s own inbound and outbound agents. Creating and deleting saved agents requires a Deepgram API key with the agent:write scope. Then set the ID in .env and restart the server:

DEEPGRAM_INBOUND_AGENT_ID=<uuid>
DEEPGRAM_OUTBOUND_AGENT_ID=<uuid>

The server then sends the agent’s ID in the Settings message. It sends the caller’s details with an UpdatePrompt message, and the greeting with an InjectAgentMessage message. You can set an ID for one direction and keep the full definition for the other. Saved agents have some limits:

You can’t edit a saved agent. After you change the instructions, functions, or models, create a new saved agent and use its ID.
You manage saved agents through the Deepgram API only. Everyone in your Deepgram project can read them, so don’t put secrets in the instructions.
The greeting starts about half a second later than it does without a saved agent.
If the ID doesn’t exist, every call fails. The server checks the ID on startup and stops if it doesn’t exist.

For more information, see Deepgram’s Reusable Agent Configurations.

Observability

The example logs to the console by default. You can also send logs to a JSON file, publish them to a Redis stream, and export OpenTelemetry traces. For the full reference, see Observability in the example README.

Log levels

LOG_LEVEL controls the agent’s pipeline logs in inbound/agent.py and outbound/agent.py. Structured events, the session start line, warnings, and errors appear at every level.

`LOG_LEVEL`	What you see
`quiet`	Structured events, session start, warnings, and errors
`normal` (default)	Adds transcripts, tool calls, and barge-ins
`verbose`	Adds Deepgram events and checkpoints

Structured events

Each call emits these events. Every event carries an event field with its name, and the call_id and leg_call_id fields described in Correlate a call with Plivo logs.

Event	When	Key fields
`call_answered`	The call starts	`from_number`, `to_number`
`user_text`	The caller finishes a turn	`turn`, `text`
`agent_text`	The agent responds	`turn`, `text`
`turn_complete`	A reply ends	`barge_in`, `total_latency_ms`
`session_end`	The call ends	`duration_s`, `turns`

A reply ends when it finishes playing or when the caller interrupts it. turn_complete also has per-stage latencies, which come from Deepgram LatencyReport messages in milliseconds. session_end also has the barge-in count and the average time to first audio.

Local development

Keep the default text output. Each pipeline log line starts with the call UUID, the seconds since the session started, and the pipeline stage, so you can follow one call in the console:

[<call UUID>] [<elapsed>s] [<stage>] <message>

To debug a call, set LOG_LEVEL=verbose and restart the server.

LOG_LEVEL=verbose uv run python -m inbound.server --tunnel

Look for Warning, InjectionRefused, and error lines, and check the session_end summary line for the turn count and average TTFS.

Production

Set LOG_FORMAT=json to write each log record to stderr as a JSON object, which your log platform can parse. To also write JSON logs to a file, set LOG_FILE. The file rotates at 100 MB and keeps 7 days of logs.

LOG_FORMAT=json
LOG_FILE=voice-agent.jsonl

To publish logs to a Redis stream, install the streaming extra and set REDIS_EVENTS_URL. The server adds every log record to the stream, not only the structured events, so filter on the event field in your consumer. The stream keeps about the last 10,000 entries.

uv sync --extra streaming

REDIS_EVENTS_URL=redis://localhost:6379
REDIS_STREAM_KEY=voice-agent:events

To export OpenTelemetry traces, install the observability extra and set OTEL_EXPORTER_OTLP_ENDPOINT to an OTLP gRPC endpoint. Without the extra, tracing is off and the server runs normally.

uv sync --extra streaming --extra observability

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

At startup, the server logs Redis Streams sink enabled and OTel tracing enabled when each output is active.

The Docker image installs the streaming extra but not the observability extra. The image runs uv sync --locked --no-install-project --no-dev --extra streaming. To export traces from a container, add --extra observability to that command in the Dockerfile.

Correlate a call with Plivo logs

Every log record for a call, both the structured events and the pipeline lines, carries two fields with the complete call IDs:

call_id is the Plivo ParentCallUUID when the answer webhook includes one. Otherwise it is the call UUID from the stream’s start event.
leg_call_id is the call UUID from the stream’s start event.

To see everything for one call in JSON logs or the Redis stream, filter on call_id. The server also logs the CallUUID when the answer webhook arrives and again, with Duration and HangupCause, when the hangup webhook arrives. Search for that UUID in Plivo voice logs to see the call’s details on the Plivo side.

Troubleshooting

Issue	Solution
Plivo doesn’t request the Answer URL	Check that the number is linked to the `Deepgram_VoiceAgent` application, and that the application’s Answer URL is your current public URL. Check the call in the Plivo logs.
The server logs `signature mismatch`	Signature validation failed. Check that `PUBLIC_URL` matches the URL that Plivo calls, and that `PLIVO_AUTH_TOKEN` belongs to the account that owns the number or call. See Signature validation.
`--tunnel` keeps logging `Waiting for Plivo to accept`	This is expected for up to three minutes while the new hostname resolves. If it stops trying, restart the server to get a new URL, or use ngrok.
Deepgram rejects the connection with a 401 error	Deepgram expects the header `Authorization: Token <key>`, not `Bearer`. Check that the key’s project has Voice Agent API access.
Deepgram sends `Error` right after `Settings`	A field in `Settings` is invalid. The most common cause is an `agent.language` or `speak.provider.language` field, which Deepgram rejects. Other causes are an unknown model ID or a saved agent ID that doesn’t exist. The `description` field of the `Error` message names the problem.
The call is silent, or the agent doesn’t greet the caller	Check that the audio settings are `mulaw` at 8000 Hz, with `container: "none"` for output.
The agent keeps talking when the caller interrupts	Check that the server sends `clearAudio` with the stream’s `streamId`.
The session ends when no audio is sent	Deepgram closes connections that receive nothing. The example sends a `KeepAlive` message after five seconds without traffic.
The agent repeats its greeting or reads out instructions	The agent says the greeting exactly as written. Put instructions in the system prompt instead.
The call doesn’t hang up after the goodbye	The server waits for Plivo’s `playedStream` message. Check that the server sends the `checkpoint` with the stream’s `streamId`.

Example on GitHub

The complete inbound and outbound example

Deepgram Voice Agent API

Deepgram’s Voice Agent documentation

Audio Streaming best practices

Voicemail detection, connection failures, and stream logs

Audio Streaming troubleshooting

WebSocket, audio quality, and performance issues

Concepts

Integration Guides

Workflows

Deploy

Reference

Deepgram Voice Agent Integration

Prerequisites

Quick start

Clone the example

Start the server with a tunnel

Call the number

Place an outbound call

How it works

Customize the agent

Call details

Choose your models

Add your own functions

Go live

Deploy the server

Verify requests from Plivo

Connect more numbers

Make outbound calls

Saved agent configurations

Observability

Log levels

Structured events

Local development

Production

Correlate a call with Plivo logs

Troubleshooting

Example on GitHub

Deepgram Voice Agent API

Audio Streaming best practices

Audio Streaming troubleshooting

​Prerequisites

​Quick start

Clone the example

Start the server with a tunnel

Call the number

Place an outbound call

​How it works

​Customize the agent

​Call details

​Choose your models

​Add your own functions

​Go live

​Deploy the server

​Verify requests from Plivo

​Connect more numbers

​Make outbound calls

​Saved agent configurations

​Observability

​Log levels

​Structured events

​Local development

​Production

​Correlate a call with Plivo logs

​Troubleshooting

​Related

Example on GitHub

Deepgram Voice Agent API

Audio Streaming best practices

Audio Streaming troubleshooting

Prerequisites

Quick start

How it works

Customize the agent

Call details

Choose your models

Add your own functions

Go live

Deploy the server

Verify requests from Plivo

Connect more numbers

Make outbound calls

Saved agent configurations

Observability

Log levels

Structured events

Local development

Production

Correlate a call with Plivo logs

Troubleshooting

Related