Voice Channel

Human supports real-time voice interaction through the built-in voice channel and the web dashboard, enabling hands-free conversations with your AI assistant.

Overview

The voice channel provides:

• Speech-to-text (STT) — converts spoken audio to text for processing
• Text-to-speech (TTS) — reads responses aloud via Cartesia streaming
• Gemini Live — native end-to-end voice via Google’s Multimodal Live API (no STT/TTS pipeline)
• WebSocket streaming — low-latency bidirectional audio
• Manual VAD — client-managed voice activity detection for precise turn-taking
• Affective dialog — emotion-aware responses that match user tone (Gemini Live)

Modes

| Mode | How it works | Latency | | ----------------------- | ------------------------------------- | ------- | | Standard (Cartesia) | Mic → STT → Agent → TTS → Speaker | Medium | | Gemini Live | Mic → Gemini 3.1 Flash Live → Speaker | Low | | Realtime | OpenAI Realtime API (full-duplex) | Low |

Configuration

Voice settings live under the top-level voice key in config.json:

{
  "voice": {
    "mode": "gemini_live",
    "realtime_model": "gemini-3.1-flash-live-preview",
    "realtime_voice": "Puck",
    "tts_provider": "cartesia",
    "tts_voice": "your-voice-id",
    "tts_model": "sonic-2",
    "stt_provider": "whisper",
    "stt_model": "whisper-1"
  }
}

Gemini Live with Vertex AI

For production use with Vertex AI (recommended):

{
  "voice": {
    "mode": "gemini_live",
    "realtime_model": "gemini-3.1-flash-live-preview",
    "realtime_voice": "Kore",
    "vertex_region": "us-central1",
    "vertex_project": "your-project-id",
    "vertex_access_token": ""
  }
}

When vertex_region and vertex_project are set, Human uses the Vertex AI endpoint with Application Default Credentials (ADC). Leave vertex_access_token empty to use ADC automatically, or provide a pre-fetched OAuth2 token.

Voice Configuration Fields

| Field | Description | Default | | --------------------- | --------------------------------------------------------- | ------------------------------- | | mode | Voice mode: gemini_live, realtime, sonata, webrtc | — | | realtime_model | Model for Gemini Live or OpenAI Realtime | gemini-3.1-flash-live-preview | | realtime_voice | Voice name for the realtime model | Puck | | tts_provider | TTS provider for standard mode | — | | tts_voice | Voice ID for Cartesia TTS | — | | tts_model | TTS model name | — | | stt_provider | STT provider for standard mode | — | | stt_model | STT model name | — | | vertex_region | Vertex AI region (enables Vertex endpoint) | — | | vertex_project | Vertex AI project ID | — | | vertex_access_token | Pre-fetched OAuth2 token (empty = use ADC) | — |

Build Flag

Voice support is included by default. To explicitly enable:

cmake .. -DHU_ENABLE_VOICE=ON

Supported Providers

| Provider | STT | TTS | Notes | | -------------- | ------ | ------ | ------------------------------------- | | Gemini Live | Native | Native | End-to-end multimodal, lowest latency | | OpenAI Whisper | ✓ | — | Best transcription accuracy | | Cartesia | — | ✓ | Streaming TTS with emotion controls | | OpenAI TTS | — | ✓ | Multiple voices | | Deepgram | ✓ | — | Low latency |

Security

Audio data is streamed directly to the configured provider and is not stored locally. All connections use TLS. Gemini Live uses Vertex AI with ADC for production authentication — API keys are supported for development only.