Ollama Voice Bot: Run Local LLMs for AI Phone Agents (2026)

Interacting with Your Local LLM: The Ollama API and Python

import requests
import json

# Ollama API endpoint
OLLAMA_ENDPOINT = "http://localhost:11434/api/generate"

# System prompt for our phone agent
SYSTEM_PROMPT = """
You are a helpful, friendly, and concise AI phone agent for 'AquaSparkle Pools'.
Your goal is to answer customer questions about pool maintenance.
- Keep your answers under 40 words.
- You are bilingual in English and French. Respond in the language of the user's question.
- Do not ask follow-up questions. Provide a direct answer.
"""

def get_llm_response(user_query: str) -> str:
    """
    Gets a response from the local Ollama voice bot.
    """
    full_prompt = f"{SYSTEM_PROMPT}\n\nUser: {user_query}\nAI:"

    payload = {
        "model": "qwen2:7b-instruct-q4_K_M",
        "prompt": full_prompt,
        "stream": False,
        "keep_alive": -1, # Keep the model loaded in VRAM
        "options": {
            "num_predict": 80, # Max tokens
            "temperature": 0.7
        }
    }

    try:
        response = requests.post(OLLAMA_ENDPOINT, data=json.dumps(payload), timeout=15)
        response.raise_for_status()
        
        response_data = response.json()
        return response_data.get("response", "").strip()

    except requests.exceptions.RequestException as e:
        print(f"Error communicating with Ollama: {e}")
        return "I'm sorry, I'm having trouble connecting to my brain right now."

# Example usage
if __name__ == "__main__":
    question = "How often should I check my pool's pH level?"
    answer = get_llm_response(question)
    print(f"User Question: {question}")
    print(f"AI Agent Answer: {answer}")

    question_fr = "Bonjour, comment puis-je éviter les algues dans ma piscine?"
    answer_fr = get_llm_response(question_fr)
    print(f"\nUser Question: {question_fr}")
    print(f"AI Agent Answer: {answer_fr}")

Prompt Engineering for Natural Phone Conversations

Performance Tuning: Quantization and Monitoring

The Power of Quantization (q4_K_M)

• `q4` means it's a 4-bit quantization.
• `_K` is a "K-quants" strategy, an improved method over older techniques.
• `_M` means it's the "medium" size variant, offering a great balance of performance and quality.

• Unquantized (FP16): ~14 GB VRAM
• Quantized (q4_K_M): ~5 GB VRAM

Monitoring VRAM Usage with `nvidia-smi`

# Watch GPU usage in real-time
watch -n 1 nvidia-smi

Bringing it to Life: Integration with Asterisk EAGI

1. Incoming Call: A call arrives at your Asterisk server.
2. Asterisk Dialplan: The dialplan executes an `AGI` script.
3. Speech-to-Text (STT): The AGI script uses an STT engine (e.g., Vosk, Whisper) to capture the caller's audio and transcribe it to text.
4. Call Ollama API: The transcribed text is passed to your Python script, which calls the local Ollama API (as shown in our example).
5. Get LLM Response: The Ollama voice bot processes the text and returns a concise answer.
6. Text-to-Speech (TTS): The Python script sends the LLM's text response to a TTS engine (e.g., Piper, Coqui AI) to generate audio.
7. Stream Audio: The generated audio is streamed back to the caller via Asterisk.

Frequently Asked Questions (FAQ)

What is an Ollama voice bot?

An Ollama voice bot is a conversational AI system, typically used for phone calls or voice assistants, that uses the Ollama platform to run a Large Language Model (LLM) on local hardware. This approach prioritizes privacy, cost control, and low latency by avoiding reliance on cloud-based AI services.

Can I run this on a Raspberry Pi or a computer without a GPU?

While you can run Ollama and smaller models on a CPU or a Raspberry Pi, it is not recommended for real-time voice applications. The inference speed would be too slow, leading to long, unacceptable delays in conversation. A dedicated NVIDIA GPU with at least 8GB of VRAM is strongly recommended for a responsive local LLM voice assistant.

How does this compare to cloud services like Google Dialogflow or Amazon Lex?

Cloud services offer a managed, easier-to-set-up platform but come with per-use costs, potential data privacy issues, and higher network latency. An Ollama-based solution requires a higher initial technical investment but provides superior privacy, predictable costs (fixed hardware/server expenses), and the lowest possible latency for more natural conversations.

What are the best open-source STT and TTS engines to pair with this?

For a fully offline AI voice system, you'll need local STT/TTS. Great open-source options include:

• Speech-to-Text (STT): Vosk, Whisper (via whisper.cpp for CPU/GPU efficiency).
• Text-to-Speech (TTS): Piper (very fast, low resource), Coqui TTS (high quality, more complex).

How do I handle multiple concurrent calls?

Handling concurrent calls requires careful resource management. A single powerful GPU (like an L40S or A100) can handle multiple models or concurrent requests to the same model. Ollama's architecture is evolving to better support this. For high-volume scenarios, you might run multiple Ollama instances on different GPUs and use a load balancer to distribute requests from your Asterisk servers.

Why is Qwen 2.5 7B recommended over Llama 3 8B for voice?

While both are excellent, Qwen 2.5 7B has demonstrated a lower Time to First Token (TTFT) and higher tokens-per-second generation speed in many benchmarks. For voice, starting the response quickly and finishing it fast is more important than the slightly higher reasoning complexity Llama 3 might offer. Qwen's tendency towards more concise answers is also a natural fit for phone conversations.

Is it difficult to set up the Asterisk integration?

It requires familiarity with both Asterisk dialplans and scripting (like Python or Perl). The concept of using AGI is straightforward, but the implementation details—managing audio streams, calling external processes for STT/TTS/LLM, and handling errors—can be complex. However, the modularity of the approach allows you to build and test each component (STT, LLM, TTS) independently.

What are the real-world hardware costs?

As of 2026, a capable server with a GPU like an NVIDIA L4 or an RTX 4060 Ti can be purchased or rented for a reasonable price. A dedicated server with an L4 GPU might cost around $200-$400/month. A self-built machine with a consumer GPU could have an upfront cost of $1500-$2500. This is a fixed cost, which can be significantly cheaper than paying per-minute fees for thousands of call-minutes per month on a cloud platform.

How can I make my Qwen voice agent support more languages?

The Qwen model family has strong multilingual capabilities out of the box. The key is in your prompt. By instructing the model to "respond in the language of the user's question," you leverage its inherent ability to detect the input language and generate a response in the same one. Ensure your STT engine also supports the languages you wish to serve.

What does `keep_alive: -1` actually do?

The `keep_alive` parameter tells the Ollama server how long to keep a model loaded in the GPU's VRAM after it has finished processing a request. By default, it's 5 minutes. Setting it to `-1` tells Ollama to *never* unload the model. This dedicates the VRAM to that model but ensures that every single API call is a "hot" call, with zero time spent loading the model into memory, which is essential for a responsive self-hosted LLM telephony system.