Spoken language identification

VoiceThere can identify the language the caller is speaking and deliver a user_language speech event to your agent. Use it to switch prompts, pick a TTS voice, or keep session state in the caller’s language. Detection uses Whisper tiny on VoiceThere (ISO 639-1 codes such as en, de, es).

How it works

• At the start of each new user utterance, the voice pipeline buffers inbound speech and runs spoken-language identification once enough audio is buffered. On VoiceThere Cloud the window is 1500 ms (1.5 seconds) by default and identification runs while the caller is still talking; the SDK and self-hosted runners default to 1000 ms when unset.
• The confident ISO code is emitted as user_language. The event is re-emitted only if the detected language changes later in the session.
• Both event.language and event.text carry the ISO code.
• Detection alone does not change STT or TTS unless you enable LID auto-switch or call setVoiceLanguage from your agent (see the request shape).
• When conversation history is on, each stored utterance can include an ISO language on turn metadata (not a separate user_language row).

Agent handler

Handle every speech event with onSpeechEvent, or use the convenience hook onUserLanguage in @voicethere/agent:

import { defineAgent, agentLog } from "@voicethere/agent";

export default defineAgent({
  onUserLanguage({ sessionId, language }) {
    agentLog("info", `detected language ${language} for ${sessionId}`);
    // Route system prompts or TTS voice from the ISO 639-1 code.
  },
});

The voice-starter template logs user_language. See Agent templates.

Identification timing

On VoiceThere Cloud, languageId.timing decides when identification runs (runner SHERPA_LID_TIMING):

• early (default): once per utterance, as soon as languageId.minSpeechMs of speech is buffered, while the caller is still talking.
• end_of_utterance: once per utterance, after the caller stops talking.
• continuous: repeated passes while the caller keeps talking. In your own SDK app the same behavior is languageId.continuous: true in VoiceAgentConfig.

CPU trade-off of continuous: it runs Whisper more often. On a busy runner that can slow Piper TTS and stretch the gap between spoken sentences (a brief first TTS burst such as an echo. prefix is the first thing listeners lose). If agent audio gets choppy, go back to voicethere projects voice-advanced set languageId.timing early. See Switching language mid-conversation for how timing changes the language check hold on finals.

TTS voice ids

These are the VoiceThere text-to-speech voices. Pass one as ttsVendor.voice in setVoiceLanguage (see the request shape), or as the project tts_model_id.

STT model ids

These are the VoiceThere speech-to-text models. Pass one as sttVendor.model in setVoiceLanguage, or as the project stt_model_id. When you omit the model on a switch, the language code is used if a model with that id exists. Whisper tiny is only the language detector. It is not an STT id.

Cloud runners

Voice and Voice+Data cloud deploys enable spoken-language identification automatically using Whisper tiny on VoiceThere, with early identification by default.

Tune identification from the dashboard or CLI under Advanced voice pipeline:

• languageId.enabled — turn off to skip LID while keeping the model on the runner (SHERPA_LID_ENABLED=0).
• languageId.minSpeechMs — integer 1000–5000, cloud default 1500. Lower values emit user_language sooner but Whisper tiny may mis-guess on short or ambiguous audio. Higher values wait for more inbound PCM for a more confident code. An utterance shorter than the window skips identification (language_id_skipped) and its final is released immediately.
• languageId.timing — early (default), end_of_utterance or continuous (SHERPA_LID_TIMING).

See also Advanced voice settings.

Whisper tiny is not an STT picker option. Choose a Zipformer language in STT & TTS vendors as usual; language ID runs alongside that model.

Local development

From a node-webrtc-rust checkout:

npm run download-lid --workspace=@node-webrtc-rust/example-voice-agent-local-sherpa
export SHERPA_LID_MODEL_PATH=.models/sherpa-onnx-whisper-tiny
npm run start:roundtrip-language-id --workspace=@node-webrtc-rust/example-voice-agent-local-sherpa

Local runner: set SHERPA_LID_MODEL_PATH in .env.local (see the runner .env.example).