Faster-Whisper STT for Asterisk: 170ms Real-Time Transcription

In the world of telephony and interactive voice response (IVR), latency is the enemy. A delay of even half a second can lead to a disjointed, frustrating user experience. For years, integrating high-accuracy Automatic Speech Recognition (ASR) into Asterisk meant a trade-off: either accept high latency from powerful models or settle for lower accuracy with faster, on-premise solutions. That era is over. By leveraging Faster-Whisper for Asterisk STT, we can achieve real-time, sub-200ms transcription with state-of-the-art accuracy, paving the way for truly conversational and responsive AI voice agents.

This comprehensive guide will walk you through the entire process, from understanding the technology to deploying a production-ready Whisper STT server optimized for Asterisk. We'll cover the hardware setup, model selection, server code, and the critical optimizations that make 170ms transcription a reality.

Why Faster-Whisper Trumps Standard Whisper for Telephony

OpenAI's Whisper model is renowned for its accuracy and robustness. However, its original Python implementation is not optimized for speed, making it unsuitable for real-time applications like a speech to text voice agent. This is where Faster-Whisper comes in.

Faster-Whisper, developed by Guillaume Klein (creator of OpenNMT), is a complete reimplementation of the Whisper model using CTranslate2, a fast inference engine for Transformer models. The results are staggering.

Here’s a direct comparison for a real-time telephony use case:

For any serious ASR Asterisk AI project, the choice is clear. The performance gains from Faster-Whisper are not just incremental; they are transformative, making it the de facto standard for production deployments.

Choosing the Right Model: distil-large-v3 vs. large-v3

Within the Faster-Whisper ecosystem, you still have choices. The model you select directly impacts the balance between speed and accuracy. For voice agent applications, two models stand out:

1. distil-large-v3: This is our top recommendation for most voice agent scenarios. It's a distilled model created by SYSTRAN, aptly named Systran distill whisper. Distillation is a process where a smaller model is trained to mimic the behavior of a larger one. The result is a model that is 2.1x faster, 49% smaller, but retains 99% of the accuracy of the full `large-v3` model on short-form audio, which is typical for conversational AI.
2. large-v3: The full-fat, most accurate model from OpenAI. While `distil-large-v3` is nearly identical in performance for conversational snippets, `large-v3` might eke out slightly better accuracy on long, complex sentences, or in very noisy environments. Use this only if you have benchmarked `distil-large-v3` and found its accuracy insufficient for your specific domain, and you have the GPU headroom to spare.

For the remainder of this guide, we will focus on the `distil-large-v3` model as it provides the optimal balance for a responsive Faster-Whisper Asterisk STT system.

# Example for Ubuntu 22.04 with CUDA 12.1
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-1

sudo apt-get -y install libcudnn8 libcudnn8-dev

# Add this to your ~/.bashrc
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

python3 -m venv venv-faster-whisper
source venv-faster-whisper/bin/activate

# Install PyTorch for CUDA 12.1 first
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install Faster-Whisper and other dependencies
pip install faster-whisper flask webrtcvad numpy

import os
import numpy as np
import webrtcvad
from flask import Flask, request, jsonify
from faster_whisper import WhisperModel

# --- Configuration ---
# Model: "distil-large-v3", "large-v3", etc.
MODEL_NAME = "distil-large-v3" 
# Device: "cuda" or "cpu"
DEVICE = "cuda" 
# Compute Type: "int8" for speed, "float16" for higher precision
COMPUTE_TYPE = "int8" 
# Flask Server Port
PORT = 6000

# --- VAD Configuration ---
VAD_AGGRESSIVENESS = 3 # 0-3, 3 is most aggressive
VAD_FRAME_MS = 30 # 10, 20, or 30
VAD_FRAME_SAMPLES = int(16000 * VAD_FRAME_MS / 1000)

# --- Asterisk Audio Format ---
ASTERISK_RATE = 8000
WHISPER_RATE = 16000

# --- Initialization ---
print("Loading Faster-Whisper model...")
model = WhisperModel(MODEL_NAME, device=DEVICE, compute_type=COMPUTE_TYPE)
print("Model loaded.")

vad = webrtcvad.Vad(VAD_AGGRESSIVENESS)

app = Flask(__name__)

def resample_audio(audio_bytes):
    """
    Resample 8kHz raw audio from Asterisk to 16kHz for Whisper.
    """
    # Interpret 8kHz 16-bit signed little-endian mono audio
    audio_s16 = np.frombuffer(audio_bytes, dtype=np.int16)
    
    # Simple upsampling by repeating samples
    # For higher quality, use scipy.signal.resample, but numpy is faster for this
    audio_16k_s16 = np.repeat(audio_s16, WHISPER_RATE // ASTERISK_RATE)
    
    # Convert to float32 expected by Faster-Whisper
    audio_float32 = audio_16k_s16.astype(np.float32) / 32768.0
    return audio_float32

def get_speech_segments(audio_float32):
    """
    Use VAD to find and return only the speech segments.
    This is a simplified VAD logic. For production, consider a more robust state machine.
    """
    # Convert float32 back to 16-bit PCM for VAD
    audio_s16 = (audio_float32 * 32768.0).astype(np.int16)
    
    speech_segments = []
    for i in range(0, len(audio_s16), VAD_FRAME_SAMPLES):
        frame = audio_s16[i:i+VAD_FRAME_SAMPLES]
        if len(frame) < VAD_FRAME_SAMPLES:
            break # Ignore incomplete frames
        if vad.is_speech(frame.tobytes(), sample_rate=WHISPER_RATE):
            speech_segments.append(frame)

    if not speech_segments:
        return None

    return np.concatenate(speech_segments).astype(np.float32) / 32768.0

@app.route('/transcribe', methods=['POST'])
def transcribe_audio():
    lang = request.args.get('lang', 'en') # Get language from query param, default to 'en'
    
    # Get raw audio data from the POST request body
    raw_audio = request.get_data()
    if not raw_audio:
        return jsonify({"error": "No audio data received"}), 400

    try:
        # 1. Resample audio from 8kHz (Asterisk) to 16kHz (Whisper)
        audio_16k = resample_audio(raw_audio)

        # 2. Use VAD to filter out silence
        speech_audio = get_speech_segments(audio_16k)
        if speech_audio is None:
            return jsonify({"transcription": "", "error": "No speech detected"})

        # 3. Transcribe with Faster-Whisper
        segments, info = model.transcribe(
            speech_audio,
            language=lang,
            beam_size=5,
            # For real-time, we want low latency, so we process small chunks.
            # Batching is more for offline throughput.
        )
        
        transcription = "".join(segment.text for segment in segments).strip()

        print(f"Detected language '{info.language}' with probability {info.language_probability:.2f}")
        print(f"Transcription: {transcription}")

        return jsonify({"transcription": transcription, "language": info.language})

    except Exception as e:
        print(f"Error during transcription: {e}")
        return jsonify({"error": str(e)}), 500

if __name__ == '__main__':
    # For production, use a proper WSGI server like Gunicorn or uWSGI
    # gunicorn --workers 1 --threads 4 --bind 0.0.0.0:6000 stt_server:app
    app.run(host='0.0.0.0', port=PORT, debug=False)

# Run with Gunicorn
gunicorn --workers 1 --threads 4 --bind 0.0.0.0:6000 stt_server:app

Real-Time Optimization: Beyond the Basics

The server code above is a great start, but for a truly responsive speech to text voice agent, we need to squeeze out every millisecond of latency. Here are the most impactful optimizations:

• Forced Language: The `model.transcribe` function can auto-detect the language, but this process adds significant latency (often 100-200ms). For a dedicated voice agent (e.g., a French-speaking bot), always force the language. In our Flask app, we pass the language as a query parameter (`/transcribe?lang=fr`). This is the single biggest latency optimization you can make.
• VAD is Non-Negotiable: Sending silence to the Whisper model is a waste of GPU cycles. Our VAD implementation filters out non-speech parts, ensuring the model only processes relevant audio. This drastically reduces the amount of data for transcription and improves accuracy by removing silent gaps.
• `compute_type='int8'`: As configured in our script, using 8-bit integers for computation is the key to Faster-Whisper's speed on consumer GPUs (like RTX series). While `float16` is an option, `int8` provides a massive speedup with negligible impact on accuracy for this use case.
• Batch Size: For real-time STT, we are processing one utterance at a time. The `batch_size` parameter is more relevant for offline, high-throughput tasks. By keeping our audio segments short (thanks to VAD) and processing them individually, we optimize for "time-to-first-token" latency.

Benchmark Results: Achieving 170ms Transcription

Talk is cheap; benchmarks are everything. We tested this exact setup to validate its performance. The goal was to measure the "glass-to-glass" transcription time for a typical conversational utterance.

The results demonstrate the system's suitability for real-time conversational AI.

Let's break down the "Total Perceived Latency," which is what the user actually experiences:

Asterisk EAGI Integration Example

The final piece of the puzzle is connecting your Asterisk dialplan to the STT server. This is typically done with an AGI (Asterisk Gateway Interface) or EAGI script. Here is a conceptual EAGI script in Perl that demonstrates the workflow.

#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;

# --- AGI Setup ---
$| = 1; # Autoflush
my %AGI;
while (<STDIN>) {
    chomp;
    last if $_ eq "";
    $AGI{$1} = $2 if /^(agi_(\w+)):\s+(.*)$/;
}

# --- Configuration ---
my $stt_server_url = "http://127.0.0.1:6000/transcribe?lang=en";
my $timeout = 5; # seconds

# --- Main Logic ---
# Asterisk EAGI provides audio on file descriptor 3
open(my $audio_fh, "<&3");
binmode $audio_fh;

my $audio_data;
# Read the audio stream from Asterisk
{
    local $/; # Slurp mode
    $audio_data = <$audio_fh>;
}
close $audio_fh;

# --- Call the STT Server ---
my $ua = LWP::UserAgent->new;
$ua->timeout($timeout);

my $response = $ua->post($stt_server_url, Content => $audio_data, 'Content-Type' => 'application/octet-stream');

my $transcription = "";
if ($response->is_success) {
    # In a real script, use JSON::MaybeXS to parse this
    my $json_text = $response->decoded_content;
    if ($json_text =~ /"transcription":\s*"(.*?)"/) {
        $transcription = $1;
    }
} else {
    print STDERR "Failed to call STT server: " . $response->status_line . "\n";
}

# --- Set Asterisk Channel Variable ---
# This makes the transcription available in the dialplan
print "SET VARIABLE STT_RESULT \"$transcription\"\n";
# The dialplan can now check ${STT_RESULT} and act on it.

exit 0;

This script, when called from the dialplan, reads the audio, sends it to our Whisper STT server, and sets the result back into a channel variable, effectively creating a powerful ASR Asterisk AI component.

Error Handling and Production Readiness

A production system must be resilient. Consider these factors:

• Timeouts: As shown in the Perl script, always use timeouts when calling the STT server. If the server is down or slow, the call flow must continue.
• Fallback Logic: In your Asterisk dialplan, after the AGI call, check if the `${STT_RESULT}` variable is empty. If it is, an error occurred. You should have a fallback path, such as playing an error message and routing the call to a human operator or a simpler DTMF-based IVR menu.
• Load Balancing: If you anticipate high call volume, a single STT server may not be enough. You can run multiple instances of the Flask/Gunicorn server (ideally on different GPUs/machines) and use a load balancer like Nginx or HAProxy to distribute requests.
• Health Checks: Expose a `/health` endpoint on your Flask server that returns a 200 OK status. Your load balancer can use this to know if a server instance is healthy and able to accept traffic.

Frequently Asked Questions (FAQ)

Can I run this Faster-Whisper Asterisk STT setup without a GPU?

Technically, yes. You can set `DEVICE = "cpu"` in the Python script. However, the performance will be drastically slower, with transcription times likely exceeding 5-10 seconds per utterance. This is not viable for a real-time voice agent. A GPU is a mandatory requirement for this low-latency application.

What is the difference between `distil-large-v3` and `distil-whisper-large-v2`?

`distil-large-v3` is the latest distilled model, based on OpenAI's Whisper `large-v3` model. It offers improved accuracy, better language support, and more robust performance on noisy audio compared to the older `v2` version. For any new project,