Open-Source LiveKit Telephony Noise and Echo Cancellation in Python

What this package does

If a LiveKit agent handles SIP calls, the caller's microphone can contain fan noise, traffic, keyboard clicks, room rumble, or the agent's own voice returning through a handset. These are two related but different problems. Noise suppression reduces unwanted environmental sound. Acoustic echo cancellation removes the far-end signal that leaked back into the caller's microphone.

livekit-plugins-denoise is an open-source Python plugin for running both stages inside your LiveKit agent process. It is designed for LiveKit Agents 1.6+, Python 3.10+, and narrowband or wideband telephony pipelines. The project is MIT licensed and maintained at GitHub. It is not affiliated with or endorsed by LiveKit.

The short version: create one denoiser per call, enable the options you need, and put an EchoReferenceTap after session startup so the processor receives the agent's outgoing audio as its far-end reference. For LiveKit's general model guidance, see the official noise and echo cancellation documentation.

Quick answer

To add open-source LiveKit telephony noise and echo cancellation, install livekit-plugins-denoise, pass a TelephonyDenoiser to RoomOptions audio input, and then wrap session.output.audio with EchoReferenceTap after session.start(). Noise suppression can use DeepFilterNet3 or WebRTC; echo cancellation needs the outgoing agent audio and a realistic SIP delay estimate. The plugin removes denoising metering from your application, but your own compute, carrier, LiveKit infrastructure, and speech-model costs still apply.

Prerequisites and installation

The package targets Python 3.10 or newer. It depends on LiveKit Agents, the LiveKit RTC bindings, NumPy, and deepfilter-stream. The current repository release target is 0.1.1; use the unpinned command below to receive the current published release, or pin the version after 0.1.1 is published.

python -m pip install livekit-plugins-denoise

For a worker requirements file, add the dependency alongside your other agent dependencies:

livekit-plugins-denoise==0.1.1

The PyPI wheel contains the Python package, not demo recordings. Source code, tests, and examples are available in the open-source repository.

Complete combined noise and echo example

The following is the important integration shape for a SIP agent. The denoiser belongs on inbound audio. The reference tap belongs on the output chain, and it must be attached after session.start() because RoomIO replaces the output during startup.

from livekit.agents import room_io
from livekit.plugins import telephony_denoise

denoiser = telephony_denoise.TelephonyDenoiser(
    telephony_denoise.DenoiseOptions(
        echo_cancellation=True,
        noise_suppression=True,
        high_pass_filter=True,
        auto_gain_control=True,
        enhancer="deepfilter",
        stream_delay_ms=120,
    )
)

await session.start(
    agent=agent,
    room=room,
    room_options=room_io.RoomOptions(
        audio_input=room_io.AudioInputOptions(
            noise_cancellation=denoiser,
            # Avoid stacking a second AGC in RoomIO.
            auto_gain_control=False,
        ),
    ),
)

# RoomIO has installed the final output chain now.
session.output.audio = telephony_denoise.EchoReferenceTap(
    denoiser,
    next_in_chain=session.output.audio,
)

Use the package's SIP example as a complete worker skeleton. The example keeps phone audio narrowband and leaves speech-to-text, text-to-speech, and telephony credentials to the application.

Why output ordering matters

The input processor cannot cancel echo from a signal it never receives. EchoReferenceTap observes the agent's outgoing frames and feeds them to the acoustic echo canceller as the far-end reference. If the tap is omitted, noise suppression may still work, but echo cancellation has no reference signal and cannot reliably distinguish the agent's voice from the caller's voice.

Attaching the tap before session.start() is also unsafe. RoomIO constructs or replaces the output chain during startup, so an earlier assignment can be discarded. Start the session first, then insert the tap with the existing output as next_in_chain. This preserves downstream output processors and keeps the reference aligned with the audio that is actually sent.

For a conceptual overview of the phone workflow, see the inbound voice channel and the inbound customer-service voice-agent use case.

Noise suppression and echo cancellation are separate controls

Noise suppression is useful when the caller is near a fan, vehicle, keyboard, street, or another steady background source. The plugin supports two enhancer choices:

• DeepFilter uses DeepFilterNet3 through deepfilter-stream. It is the default choice when speech quality is more important than the smallest possible processing footprint.
• WebRTC uses the WebRTC audio processing module and is a practical lower-latency option for deployments that want a compact, established suppressor.

Echo cancellation is different. It models the path from the agent's output back into the caller's input. It needs the reference tap, correct channel assumptions, and a delay estimate that matches the carrier or handset path. Turning on noise suppression does not automatically cancel echo, and turning on echo cancellation does not remove every background sound.

High-pass filtering can reduce low-frequency rumble. Automatic gain control can help quiet callers, but use only one AGC in the chain. The example disables RoomIO's additional AGC because the denoiser already owns that responsibility.

Prewarming and the first call

DeepFilterNet3 may download or initialize model resources the first time the enhancer sees audio. For a production worker, call telephony_denoise.prewarm() from your process setup hook so model work happens before a caller is connected. This does not remove the need for a writable cache and outbound access during the first deployment; make those requirements explicit in your container and worker environment.

If you choose enhancer="webrtc", the neural model is not needed. That can simplify a small worker or a restricted deployment, at the cost of a different suppression profile. Measure with your own phone routes instead of assuming that one enhancer is best for every carrier.

SIP delay tuning

stream_delay_ms is a starting estimate for the time between the agent output and the same audio returning through the caller's device and carrier path. The default is 120 milliseconds. It is not a universal constant.

Tune it with controlled calls. Use a speakerphone or handset that produces a repeatable echo, speak short phrases, and change only the delay between trials. If the returned voice remains audible, the reference may be missing, the delay may be outside the useful window, or another endpoint may be generating the echo. If speech becomes hollow or clipped, reduce aggressive processing and confirm that you are not applying multiple echo cancellers to the same stream.

Document the delay that works for each meaningful telephony route. Mobile, PSTN, SIP trunk, and browser test calls can have different timing. A setting that sounds good on a local softphone is not proof that every carrier path is aligned.

Docker and worker deployment

Install the package through the worker's requirements file rather than a one-off Docker command. This keeps local development, CI, and production images on the same dependency graph:

# requirements.txt
livekit-agents==1.6.*
livekit-plugins-denoise==0.1.1

Then let the Dockerfile install the requirements file in the normal way. The container also needs the native libraries required by the LiveKit RTC wheel and any audio backend already used by your worker. Do not copy the standalone plugin source into the application image; consume the published package so upgrades are visible in dependency review.

Use a persistent or pre-populated model cache when DeepFilter is enabled. Log which enhancer and option set each worker uses, but do not log caller audio or transcripts unless your retention and consent policy allows it. For sensitive workloads, define who can inspect recordings, reference signals, and diagnostic traces before enabling them.

Common failures and fixes

The package imports but echo remains

Check that EchoReferenceTap is present, attached after session.start(), and wrapping the current session.output.audio. Confirm that the agent's output really travels through the chain being tapped. Then test stream_delay_ms instead of immediately increasing suppression strength.

The first caller hears a delay or the worker stalls

Prewarm the neural enhancer during process startup, verify model-cache permissions, and make model initialization visible in worker logs. A cold model path can make the first audio frame pay initialization cost.

Audio sounds metallic or speech is removed

Try enhancer="webrtc", reduce aggressive options, and confirm that a second noise or echo canceller is not running in the carrier, browser, SIP trunk, and agent simultaneously. Keep one clear owner for each processing stage.

Only some callers have echo

That usually points to endpoint or route differences rather than a single global setting. Compare handset, speakerphone, mobile, and softphone calls. Record route, sample rate, delay estimate, and whether the far-end reference was present for each test.

A configuration change does nothing

Check the worker image and requirements lock first. A running container may still have the previous package version. Log the plugin version at startup and redeploy the worker after changing options.

How to reason about the processing pipeline

It helps to separate the call into three observable signals. The near-end signal is the audio arriving from the caller. The far-end signal is the audio your agent is sending. The processed signal is what the agent pipeline forwards to speech recognition or downstream application logic after filtering. Echo cancellation compares the near-end signal with the far-end reference. Noise suppression estimates unwanted content that has no matching far-end signal. Keeping these names consistent makes logs and test notes much easier to interpret.

The processor also has to deal with audio frames that are not perfectly aligned to a phone conversation. SIP media can arrive in chunks that differ from the enhancer's preferred frame size, and a carrier can introduce jitter or resampling. The package buffers and resamples through its LiveKit audio-processing path rather than asking the application to hand-build a second audio clock. That is why the integration should pass frames through the normal RoomIO chain instead of converting them in a separate callback.

Treat each option as an experiment with a measurable purpose. Enable noise_suppression when the caller's speech is being masked by environmental sound. Enable echo_cancellation when the agent's own speech is returning. Enable high_pass_filter when low-frequency rumble is a recurring problem. Enable auto_gain_control when quiet callers need level assistance, and disable any duplicate AGC elsewhere. If a change cannot be tied to one symptom, it is difficult to know whether it helped or merely changed the voice character.

Keep a small route notebook for production. Record the endpoint type, carrier or trunk, codec if known, sample rate, enhancer, delay estimate, and whether the call used handset, headset, or speakerphone. Add operator ratings for residual noise, residual echo, double-talk, clipping, and perceived speech quality. These notes are more useful than a single global score because a telephony deployment is a collection of routes with different acoustic and network behavior.

When a call needs human help, preserve the original context. The denoiser should make the conversation easier to understand, not hide a failed transfer or remove the operator's ability to inspect what happened. Pair the audio pipeline with an explicit human handoff, route-level error logging, and a rollback switch. This keeps signal processing in its proper role as one dependable stage inside a larger voice-agent system.

Production checklist

Before enabling this on a live voice-agent workflow, verify:

1. The worker uses Python 3.10+ and a supported LiveKit Agents version.
2. The denoiser is created once per call, not shared across callers.
3. Inbound audio is passed to the processor and outbound audio is passed through EchoReferenceTap.
4. Only one AGC, noise suppressor, and echo canceller owns each stage.
5. DeepFilter model initialization and cache behavior are tested before traffic arrives.
6. Delay settings are recorded per route and validated with controlled calls.
7. Human escalation remains available for callers who cannot be understood; see the human handoff design guide.
8. Audio privacy, retention, and observability are documented.

This package is a signal-processing component, not a complete telephony platform. It does not provide phone numbers, carrier service, speech models, call recording policy, or guaranteed transcription quality. It gives a LiveKit agent an open-source place to own noise and echo processing, while the application remains responsible for the rest of the call path.

For route-specific reference debugging, read the SIP echo cancellation troubleshooting guide. To compare operating models, see managed versus self-hosted telephony denoising. For broader voice-agent architecture, compare this implementation with the AI voice-agent guide and the conversational IVR migration guide.