> ## Documentation Index
> Fetch the complete documentation index at: https://mantle.rasa.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Speech providers

> Supported speech recognition and speech synthesis providers, with their API keys, defaults, and settings.

Every voice channel turns the caller's audio into text with a speech recognition
(ASR) provider, and turns the agent's replies into audio with a speech synthesis
(TTS) provider. You declare each provider as a model group in `integrations.yml`,
then name that group on the channel's `asr` or `tts`, or as `voice.asr` and
`voice.tts` in [`agent.yml`](https://mantle.rasa.com/reference/agent-yml#asr-and-tts). Recognition and
synthesis are chosen separately, so you can mix providers. To use a provider
that is not built in, see [Custom speech providers](https://mantle.rasa.com/reference/custom-speech-providers.md).

## Choose a provider

Set `provider` on the model group, and choose models and voices per language in
[`language_map`](https://mantle.rasa.com/reference/integrations-yml#language_map). Point the channel, or
`agent.yml`, at the group's id:

```yaml integrations.yml theme={null}
model_groups:
  - id: phone_asr
    models:
      - provider: deepgram_flux
        api_key: ${DEEPGRAM_API_KEY}
        language_map:
          en:
            language: en
            model: flux-general-en
  - id: phone_tts
    models:
      - provider: cartesia
        api_key: ${CARTESIA_API_KEY}
        language_map:
          en:
            language: en
            voice: f786b574-daa5-4673-aa0c-cbe3e8534c02

channels:
  inspector:
    enabled: true
    asr: phone_asr
    tts: phone_tts
```

| Provider | ASR `provider` | TTS `provider` | API key environment variable |
| - | - | - | - |
| Deepgram | `deepgram` | `deepgram` | `DEEPGRAM_API_KEY` |
| Deepgram Flux | `deepgram_flux` | `deepgram_flux` | `DEEPGRAM_API_KEY` |
| Azure AI Speech | `azure` | `azure` | `AZURE_SPEECH_API_KEY` |
| Speechmatics Agent STT | `speechmatics` | — | `SPEECHMATICS_API_KEY` |
| Cartesia | — | `cartesia` | `CARTESIA_API_KEY` |
| Rime | — | `rime` | `RIME_API_KEY` |
| ElevenLabs | `elevenlabs` | `elevenlabs` | `ELEVENLABS_API_KEY` |
| Gradium | `gradium` | `gradium` | `GRADIUM_API_KEY` |
| Inworld | `inworld` | `inworld` | `INWORLD_API_KEY` |

Set the API key in your environment or `.env`. If a key is missing, Mantle logs
the missing variable and the voice conversation can't start. In the Inspector, you
see a "Voice isn't set up yet" warning.

Configure Speechmatics Agent STT as its own model group:

```yaml integrations.yml theme={null}
model_groups:
  - id: phone_asr
    models:
      - provider: speechmatics
        api_key: ${SPEECHMATICS_API_KEY}
        region: eu
        language_map:
          en:
            language: en
  - id: phone_tts
    models:
      - provider: deepgram
        api_key: ${DEEPGRAM_API_KEY}

channels:
  inspector:
    enabled: true
    asr: phone_asr
    tts: phone_tts
```

Gradium supports real-time speech recognition and synthesis over WebSockets. Set
it up as two model groups, with the recognition language and synthesis voice in
`language_map`:

```yaml integrations.yml theme={null}
model_groups:
  - id: phone_asr
    models:
      - provider: gradium
        api_key: ${GRADIUM_API_KEY}
        language_map:
          en:
            language: en
  - id: phone_tts
    models:
      - provider: gradium
        api_key: ${GRADIUM_API_KEY}
        language_map:
          en:
            voice: YTpq7expH9539ERJ

channels:
  inspector:
    enabled: true
    asr: phone_asr
    tts: phone_tts
```

Gradium uses `GRADIUM_API_KEY` for both providers. Its default synthesis voice
is `YTpq7expH9539ERJ`; set a different `voice` for each language as needed.

## Defaults

If the Inspector omits a side, and `voice.asr` or `voice.tts` does not set it
either, that side still works with only `DEEPGRAM_API_KEY`, using Deepgram's
earlier models:

| | Provider | Model |
| - | - | - |
| Speech recognition | `deepgram` | `nova-3`, English |
| Speech synthesis | `deepgram` | `aura-2-thalia-en` |

Those defaults are English, even if `agent.language` is another language. For
another language, set [`language_map`](https://mantle.rasa.com/reference/integrations-yml#language_map)
on the model group. Telephony channels have no defaults, so set both sides on
the channel or in `agent.yml`.

## Speech recognition settings

Settings other than `provider`, `api_key`, and `language_map` go on the ASR
model, next to `provider`.

| Provider | Settings |
| - | - |
| `deepgram` | `endpoint` (full WebSocket URL, default `wss://api.deepgram.com/v1/listen`), `endpointing` (ms of silence before a transcript is final, default `400`), `smart_format` (default `true`), `utterance_end_ms` (default `1000`) |
| `deepgram_flux` | `endpoint` (full WebSocket URL, default `wss://api.deepgram.com/v2/listen`), `eot_threshold` (0.5–0.9, default `0.7`), `eot_timeout_ms` (500–10000, default `5000`). See [Deepgram Flux for speech recognition](https://mantle.rasa.com/reference/integrations-yml.md#deepgram-flux-for-speech-recognition). |
| `azure` | One of `speech_region`, `speech_host`, or `speech_endpoint` (required) |
| `speechmatics` | `region` (`global`, `eu`, `us`, or `au`; default `global`), `enable_partials` (default `true`), `diarization` (`none` or `speaker`; default `none`), `emit_sentences` (default `false`), `domain` (optional) |
| `elevenlabs` | `endpoint` (full WebSocket URL), `commit_strategy` (`vad` or `manual`, default `vad`), `include_timestamps` (default `false`), `include_language_detection` (default `false`) |
| `gradium` | `model_name` (default `default`), `delay_in_frames` (7–55, default `16`), and `json_config` for additional Gradium transcription settings |
| `inworld` | `endpoint` (default `wss://api.inworld.ai/stt/v1/transcribe:streamBidirectional`). Streaming uses the `inworld/inworld-stt-1` model and LINEAR16 audio; μ-law audio is converted to PCM16. Set `language` and `model` in each `language_map` entry when needed. |

## Speech synthesis settings

Settings other than `provider`, `api_key`, and `language_map` go on the TTS
model, next to `provider`.

| Provider | Settings |
| - | - |
| `deepgram` | `endpoint` (full URL, default `wss://api.deepgram.com/v1/speak`) |
| `deepgram_flux` | `endpoint` (default `wss://api.deepgram.com/v2/speak`), `speed` (0.5–1.5 in 0.05 steps), `expressivity` (-2 to 2, default `0`). Every `language_map` entry needs a `flux-` model; the default is `flux-alexis-en`. |
| `azure` | `speech_region` (default `eastus`), `endpoint`, `ws_endpoint`. Default voice `en-US-JennyNeural`. |
| `cartesia` | `model_id` (default `sonic-3.6`), `version`, `endpoint` |
| `rime` | `model_id` (default `coda`), `endpoint`, `speed_alpha` (default `1.0`), `segment` (default `immediate`), `no_text_normalization` (default `false`). Default voice `astra`. |
| `elevenlabs` | `endpoint` (base WebSocket URL, default `wss://api.elevenlabs.io/v1/text-to-speech`), `model_id` (default `eleven_multilingual_v2`), `stability`, `similarity_boost`. Each `language_map` entry must include an ElevenLabs voice ID the API key can access; streaming synthesis includes word-level timestamps. |
| `gradium` | `model_name` (default `default`), `pronunciation_id`, and `json_config` for voice settings. Default voice ID: `YTpq7expH9539ERJ`. |
| `inworld` | `endpoint` (default `wss://api.inworld.ai/tts/v1/voice:streamBidirectional`), `model_id` (default `inworld-tts-2`), `auto_mode` (default `true`), `auto_mode_strategy` (`SENTENCE_BOUNDARY` by default; `CLIENT_SEGMENTED` is also supported). Each `language_map` entry must include an Inworld voice ID. Sentence-boundary mode supports `inworld-tts-2` and `inworld-tts-2-flash`. |

For example, select an Inworld voice and language in `language_map`:

```yaml integrations.yml theme={null}
channels:
  browser_audio:
    enabled: true
    server_url: localhost:5005
    tts:
      name: inworld
      language_map:
        en:
          language: en-US
          voice: Ashley
```

For Inworld speech recognition, select the provider and language in `language_map`:

```yaml integrations.yml theme={null}
channels:
  browser_audio:
    enabled: true
    server_url: localhost:5005
    asr:
      name: inworld
      language_map:
        en:
          language: en
          model: inworld/inworld-stt-1
```

All speech synthesis engines also accept `timeout` (seconds, default `30`).
Synthesized replies are cached in memory, so a repeated line, such as a
greeting, plays without a new provider request.

## Regional endpoints

To keep audio in a region, use the provider's regional endpoint.
For Deepgram, use a full WebSocket URL for both speech recognition and speech
synthesis. Include the API path for the selected engine. Existing ASR configs
with a host-only endpoint continue to work, but use a full URL for new configs:

```yaml integrations.yml theme={null}
model_groups:
  - id: phone_asr
    models:
      - provider: deepgram
        api_key: ${DEEPGRAM_API_KEY}
        endpoint: wss://api.eu.deepgram.com/v1/listen
  - id: phone_tts
    models:
      - provider: deepgram_flux
        api_key: ${DEEPGRAM_API_KEY}
        endpoint: wss://api.eu.deepgram.com/v2/speak
        language_map:
          en:
            model: flux-alexis-en
```

Check your provider's documentation for the hosts available to your account.

To change the default Deepgram host without editing `integrations.yml`, set
`DEEPGRAM_BASE_URL`, for example `wss://api.eu.deepgram.com`. Rasa adds the API
path for each engine. An `endpoint` in `integrations.yml` still wins.

For Speechmatics, `region` selects the Agent STT endpoint. The default `global`
endpoint routes to the nearest region; set `eu`, `us`, or `au` to pin a region.

## Troubleshoot setup

| Symptom | Check first |
| - | - |
| The Inspector says "Voice isn't set up yet" | Usually missing ASR or TTS keys: set them in the environment of the process running `rasa inspect`, or in the project's `.env`, and restart. Any other voice startup error shows the same warning, so check the server log for the cause. |
| Chat works, but the caller is not heard | Check microphone permission and ASR connection errors, then whether a final transcript reaches the agent. Chat does not test the speech provider. |
| The agent produces text but no speech | Check the TTS connection, model or voice identifier, and provider access for your account. |
| A language uses the wrong voice or recognizer | Match `language_map` keys to the agent's language codes and set the provider's `language`, `model`, and `voice` values explicitly where required. The Inspector's implicit defaults are English. |

Keep keys out of `integrations.yml`, shared logs, and source control. When
switching providers, keep the other groups in `model_groups`. Speech settings
belong on the speech model group. The channel, or `voice.asr` and `voice.tts`
in `agent.yml`, names that group. Both are read from the live project; restart
to pick them up. Changing skills, or which group `orchestrator.model_group`
names, needs `rasa train`. See
[`integrations.yml`](https://mantle.rasa.com/reference/integrations-yml) for configuration lifecycles.

Use [Test and monitor voice](https://mantle.rasa.com/voice/testing.md) to distinguish recognition, turn
handling, tool, and playback failures before tuning a provider.

## See also

* [`integrations.yml` → Voice channels](https://mantle.rasa.com/reference/integrations-yml#voice-channels): model group ids, `language_map`, `interruptions`, and `silence_timeout`
* [`agent.yml` → `asr` and `tts`](https://mantle.rasa.com/reference/agent-yml#asr-and-tts): the project-wide speech group ids
* [Custom speech providers](https://mantle.rasa.com/reference/custom-speech-providers.md): use a provider that is not built in
* [Rasa Community voice tuning walkthrough](https://rasa.community/library/tutorials/voice-ai-agent/09-voice-deep-dive/): a worked Deepgram Flux example with spoken formatting and recognition recovery; check its tested version before reusing its settings


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.