Advanced Features for Audio Streaming

What: What a voice agent on Plivo Audio Streaming can do beyond talking.
Watch out: A tool call that runs longer than a second is silence to the caller unless you cover it.

A voice agent that only talks is a recording with extra steps. The agents customers actually deploy look things up, follow a script, remember who is calling and act on what they hear. Everything below runs inside your agent, above Plivo’s WebSocket. The examples use Pipecat, which supports Plivo out of the box.

Written against pipecat-ai 1.11.0. Pipecat moves quickly and import paths change between releases — check the version you have installed before copying a snippet.

Call Your Own Systems Mid-Conversation

The agent decides, during the call, that it needs real data — an order status, an available slot, a customer record — and calls your code to get it.

from pipecat.adapters.schemas.direct_function import tool_options
from pipecat.services.llm_service import FunctionCallParams
from pipecat.processors.aggregators.llm_context import LLMContext

@tool_options(cancel_on_interruption=False, timeout_secs=30)
async def get_order_status(params: FunctionCallParams, order_id: str):
    """Look up the status of a customer's order.

    Args:
        order_id: The order number the customer gave you.
    """
    status = await my_backend.lookup(order_id)
    await params.result_callback({"status": status})

context = LLMContext(tools=[get_order_status])

The function’s parameters and description come from its signature and docstring, so there is no separate schema to maintain.

Cover the Silence While It Runs

There is no spinner on a phone call. A lookup that takes three seconds is three seconds of dead air, which is when callers say “hello? are you there?” and start hanging up.

from pipecat.frames.frames import TTSSpeakFrame

@llm.event_handler("on_function_calls_started")
async def on_function_calls_started(service, function_calls):
    await tts.queue_frame(TTSSpeakFrame("Let me check on that.", append_to_context=False))

cancel_on_interruption defaults to True. With the default, the caller saying “hello?” during your lookup cancels the lookup — so a call that was already too quiet also loses its answer. Set it to False for anything the conversation depends on.

Define the Call Flow as Data

A single system prompt holding a whole conversation together stops working the moment the call has real steps. Pipecat Flows models the call as named nodes, each with its own instructions and its own tools, and transitions between them. The flow is a YAML file the agent loads at runtime.

initial_node: initial

nodes:
  initial:
    role_message: >-
      You are a friendly insurance agent on the phone. Your responses will be
      converted to audio, so avoid special characters and say amounts in words.
    task_messages:
      - role: developer
        content: Greet the customer and ask how old they are.
    functions:
      - name: collect_age
        transition_to: marital_status

  marital_status:
    task_messages:
      - role: developer
        content: Ask whether they're single or married.
    functions:
      - name: collect_marital_status
        transition_to: quote_results

Tools that capture data or do work live in a handlers file. A function that only moves the conversation along needs no code at all — it is a transition_only entry in the YAML. Because the flow is data, one deployed agent can run whichever flow a call needs, and changing what the agent says or where a step leads is a YAML edit rather than a deploy. Write the flow in Python instead when a tool needs JSON Schema constraints an ordinary signature cannot express, or when a node’s shape depends on the conversation.

Remember a Returning Caller

The agent’s conversation history is a list you can read and write, so a caller who phones back can be greeted with what happened last time rather than starting from nothing. Save it when the call ends:

messages = context.get_messages()
with open(f"conversations/{caller_number}.json", "w") as f:
    json.dump(messages, f)

Restore it when the same number calls again:

with open(f"conversations/{caller_number}.json") as f:
    context.set_messages(json.load(f))

You need the caller’s number to key that file, and the stream’s start event does not carry it — it carries callId, streamId, accountId, tracks, mediaFormat and extra_headers, and nothing else. The caller’s number reaches you as From on the call webhook. Pass it into the stream yourself:

<Response>
  <Stream bidirectional="true" extraHeaders="caller={{From}}">
    wss://your-server.example.com/stream
  </Stream>
</Response>

It then arrives on the start event as extra_headers, and the agent can open with context instead of “how can I help you?”.

Reloading a whole prior conversation grows the context every call. For a caller who rings often, summarize the old history before restoring it rather than replaying it verbatim.

Give the Agent Your Existing Tools

If your systems already speak MCP, the agent can use those servers directly rather than having every integration rewritten as a voice-specific function.

from mcp.client.session_group import StreamableHttpParameters
from pipecat.services.mcp_service import MCPClient

mcp = MCPClient(
    server_params=StreamableHttpParameters(url=os.getenv("MCP_SERVER_URL")),
    tools_filter=["search_orders", "create_ticket"],
)
tools = await mcp.register_tools(llm)

tools_filter matters on a phone call. An MCP server exposing forty tools gives the model forty things to consider before every reply, and that shows up as latency the caller hears. Register the handful the agent actually needs.

Add Background Audio

Customers describe an agent with complete silence behind it as sounding artificial. Real people are never in silence — there is always a room around them. Faint ambience under the speech makes the call sound like a person at a desk rather than a recording. Pipecat mixes it in the output transport, so it plays for the whole call, including the gaps when the agent is not speaking.

from pipecat.audio.mixers.soundfile_mixer import SoundfileMixer
from pipecat.pipeline.worker import PipelineParams

params = FastAPIWebsocketParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    audio_out_mixer=SoundfileMixer(
        sound_files={"office": "office-ambience-8000-mono.wav"},
        default_sound="office",
        volume=0.6,
    ),
)

pipeline_params = PipelineParams(
    audio_in_sample_rate=8000,
    audio_out_sample_rate=8000,
)

The PipelineParams line is not optional here. The mixer loads the file only if its sample rate equals the pipeline’s output rate, and Pipecat defaults to 24 kHz out — so an 8 kHz ambience file is silently skipped unless you set the rate to match. Change it during the call without rebuilding the pipeline — turn the volume down while the agent reads out a reference number, or switch the ambience off entirely.

from pipecat.frames.frames import MixerEnableFrame, MixerUpdateSettingsFrame

await task.queue_frame(MixerUpdateSettingsFrame({"volume": 0.2}))
await task.queue_frame(MixerEnableFrame(False))

The sound file’s sample rate must match the pipeline’s output rate. If they differ, the file is skipped with a warning and no ambience plays for the entire call — the agent still works, so this is easy to miss. At the 8 kHz telephony default, use an 8 kHz mono file.

Keep the volume low. On a narrowband phone line, ambience that competes with the speech makes the agent harder to understand rather than more natural.

Tune Barge-In

By default the agent stops the moment it hears the caller. On a phone line that is too eager: a cough, an “mm-hmm”, or someone talking in the background all cut the agent off mid-sentence. A browser would have removed most of that with local echo cancellation. A phone line does not. Require a minimum number of real words before treating speech as an interruption.

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.turns.user_start import MinWordsUserTurnStartStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            start=[MinWordsUserTurnStartStrategy(min_words=3)],
        ),
        vad_analyzer=SileroVADAnalyzer(),
    ),
)

When an interruption does register, Pipecat sends Plivo a clearAudio event, which flushes audio already queued for playback so the caller hears the agent stop rather than talk over them.

Tuning for a Phone Line

These are settings rather than features, but they are the ones that decide whether the agent sounds right on a call.

Setting	Why it matters on a phone call
`PipelineParams(audio_in_sample_rate=8000, audio_out_sample_rate=8000)`	Plivo streams 8 kHz. Pipecat defaults to 16 kHz in and 24 kHz out, so it resamples both ways unless told otherwise.
`user_idle_timeout=5.0`	Callers go quiet. Set a timeout so the agent prompts instead of both sides waiting.
`VoiceFormatter()` on the TTS service	Reads `$42.50` as “forty-two dollars and fifty cents” instead of “dollar sign four two point five zero”.
`STTUpdateSettingsFrame` / `TTSUpdateSettingsFrame`	Switch language or voice mid-call. `Language` covers the Indian region variants — `HI_IN`, `TA_IN`, `TE_IN`, `KN_IN`, `ML_IN`, `MR_IN`, `BN_IN`, `GU_IN`.

SileroVADAnalyzer supports 8 kHz natively, one of the few audio components built for telephony rates rather than adapted to them. It accepts only 8000 or 16000.

Build with Pipecat

Connect Pipecat to Plivo Audio Streaming

Audio Streaming Reference

Every WebSocket event, with schemas and types

Transfer to Human Agent

Hand a live call to a person

Best Practices

Voicemail detection and connection handling

Concepts

Integration Guides

Workflows

Deploy

Reference

Advanced Features for Audio Streaming

Call Your Own Systems Mid-Conversation

Cover the Silence While It Runs

Define the Call Flow as Data

Remember a Returning Caller

Give the Agent Your Existing Tools

Add Background Audio

Tune Barge-In

Tuning for a Phone Line

Build with Pipecat

Audio Streaming Reference

Transfer to Human Agent

Best Practices

​Call Your Own Systems Mid-Conversation

​Cover the Silence While It Runs

​Define the Call Flow as Data

​Remember a Returning Caller

​Give the Agent Your Existing Tools

​Add Background Audio

​Tune Barge-In

​Tuning for a Phone Line

​Related

Build with Pipecat

Audio Streaming Reference

Transfer to Human Agent

Best Practices

Call Your Own Systems Mid-Conversation

Cover the Silence While It Runs

Define the Call Flow as Data

Remember a Returning Caller

Give the Agent Your Existing Tools

Add Background Audio

Tune Barge-In

Tuning for a Phone Line

Related