Voice input and replies
xopc can turn voice messages into text (speech-to-text, or STT) and assistant replies into audio (text-to-speech, or TTS). Availability depends on the selected provider and client.
Choose the experience
| Mode | Use it for | Tools |
|---|---|---|
| Current assistant | Continue the selected Agent's work, ask questions, or delegate by voice | Retains the Agent's enabled tools |
| Natural voice | Natural conversation; some clients label it Native voice | No tools; do not use it to promise task execution |
| Dictation | Turn speech into an editable draft | Inserts text after finishing; does not send it automatically |
Configure and test the voice service first, then call from the same conversation. Ending a call does not cancel a delegated task; use the separate task cancellation action.
Realtime calls
A Chat keeps one conversation across text and repeated calls. Click Voice call in Chat to connect immediately using the saved default. Current assistant retains Agent tools; Native voice · no tools is an explicit alternative under Voice service settings. Hanging up, reconnecting, or changing voice modes keeps the same Chat and its saved history. Only an explicit new Chat or reset starts a new conversation.
In the Web console, open Settings → Models and capabilities → Realtime conversation → Settings. Save a Boson API key or use BOSON_API_KEY from the Gateway environment. You can reveal and copy a key saved in the Gateway. Choose a voice, run a test call, and optionally select Higgs in the shared natural voice model picker. To configure Ada separately, open Ada → Edit profile → Voice calls and choose a call mode, natural voice model, and voice. Ada defaults to Current assistant, which can delegate work by voice. Higgs starts each call with a limited snapshot of the same Chat and its Ada persona. Boson processes the snapshot and call audio. Captions are saved in the existing Chat.
In Current assistant calls, delegated Task questions and selected results can become a separate Agent reply during a quiet gap. Routine progress stays on the Task card. A user utterance takes priority, and an interrupted update remains in the Chat record. Native voice does not run this Task update path.
The call has its own window. Minimize it to keep talking while visiting other pages. Mute microphone disables capture upload and discards unfinished input while assistant playback continues; End call releases the microphone. Dictation buffers text until Finish, applies the configured cleanup, and inserts an editable draft without sending. Cancel leaves the original draft untouched, including cancellation during cleanup. Finalized text remains recoverable after a disconnect. The same Chat cannot run a text response and a call simultaneously.
Choose XOPC hosted or Your API key once under Settings → Capabilities → Voice. This configures dictation, Agent speech and natural conversation together, without changing ordinary message readout. Natural voice defaults to the shared input credential; independent endpoints, keys and instructions are optional advanced settings. If setup is missing, the call window links to Voice settings, with a return link to the original Chat. Capability validation precedes microphone permission; it does not prove a live provider connection will succeed.
With your Alibaba key, natural chat uses the configured compatible realtime model. XOPC Cloud uses the catalog’s public realtime service and available voices. Connections with these two services restore the selected Agent's configured name/instructions and a bounded text projection of the same Chat's history. It has no tools. Older context may be excerpted; full records remain in Chat. Interrupted generated replies remain visible in records, but are omitted from subsequent model context because the exact portion heard is unknown.
Use the composer’s call button to start or continue voice in the same Chat. Network failure, a call time limit or a page reload ends the connection; start again to continue the conversation. Minimize/route navigation does not end it. There is no silent microphone reopening or automatic indefinite connection renewal.
Hosted natural calls require a published conversation route on XOPC Platform. Gateway and clients must all support protocol v3. See the repository's protocol, mobile technical design, and delivery review for implementation and verification limits.
Where else voice works
- Web and desktop Chat can transcribe supported audio attachments.
- Telegram can transcribe voice notes and send audio replies when configured.
- Other channels may support one or both directions depending on their media capabilities.
- An Agent can create speech with the voice tool when TTS is enabled.
Configure speech-to-text
Realtime dictation and conversation
Open Settings → Capabilities → Voice and choose XOPC hosted or Your API key (Alibaba Qwen). Hosted voice requires a signed-in account and available speech/realtime services. Voice names follow the display language; select from the current catalog rather than copying a supplier voice ID. For Alibaba, dictation and conversation share the input credential; existing Edge message readout stays unchanged.
Choose a conversation voice, then use Test voice. The test opens the microphone only after a click, displays a real final transcript, plays a fixed sample through the native streaming speech provider, and asks you to confirm that you heard it. It does not create a chat or call an Agent. Not tested means a route is configured, not that a live connection has succeeded. Testing may incur provider usage. When only input is configured, use Test dictation.
After testing, open Chat and use the microphone for dictation or the call button for voice conversation. Voice assistant mode also requires a working Agent model. Read messages aloud controls ordinary message readout separately. Settings are grouped into Speaking & listening, Input & display, Audio devices, Voice service, and Troubleshooting. Voice/pacing/language are listening preferences; cleanup and message readout are input/display options; providers and fallback live under service. Captions and microphone selection are saved on this browser; output follows the system device.
Audio attachments and message channels
In the Gateway console, open Settings → Capabilities → Voice:
- Open Voice service → Technical settings and enable speech-to-text.
- Choose a provider and model.
- Add the provider credential if it is not already configured.
- Save and upload a short test recording in Chat.
The setup is working when the message shows an accurate transcript and the Agent answers the spoken request.
Audio input contract
Voice uploads use POST /api/voice/transcriptions with multipart form data. WAV, WebM/Opus, Ogg/Opus, MP3, and MP4/M4A are accepted. Provider extensions receive the same audio input contract and can adapt it to a user-managed local or OpenAI-compatible transcription service.
Discussion capture keeps the compressed original recording as recoverable evidence and sends speech-aware WAV segments for live text. Segments close on a pause after at least four seconds, are capped at fifteen seconds, and pure silence is skipped. If live text is incomplete, the original is decoded into bounded chunks and transcribed sequentially instead of loading a long recording into one STT request.
Configure text-to-speech
- Expand Read messages aloud on the same Voice settings page and enable text-to-speech.
- Choose a provider, model, and voice.
- Choose when audio is created:
| Mode | Behavior |
|---|---|
| Off | No automatic audio replies; the voice tool can still be used if enabled |
| Always | Convert eligible text replies to audio |
| Inbound voice | Reply with audio only after the user sends voice |
| Tagged | Create audio only when the response explicitly requests it |
Inbound voice is a good starting point because normal text conversations remain text.
Provider choices
xopc can use supported cloud speech providers and configured speech extensions. STT is disabled by default, and xopc does not bundle a local inference engine or model manager. To keep transcription on your device, install an extension that connects to your own local or OpenAI-compatible service. Cloud providers process audio according to their own policies and may charge per use.
Use environment variables or the credential controls in the UI; do not put real keys into documentation examples. Exact configuration keys are listed in Configuration reference.
Test safely
- Start with a recording under 15 seconds.
- Speak clearly and avoid sensitive content during setup.
- Confirm the selected language and voice.
- Test in local Chat before relying on a messaging channel.
- Check usage limits before enabling automatic TTS for every reply.
Troubleshooting
| Problem | Check |
|---|---|
| Audio uploads but no transcript appears | STT is enabled, the file format is supported, and the provider credential is valid |
| A local STT extension rejects the audio format | Check the extension and local service format requirements |
| A long discussion is still finalizing | Keep the gateway running; saved segments and the original recording resume from durable state |
| Transcript uses the wrong language | Set the provider language when available or choose a more suitable model |
| Text replies work but audio replies do not | TTS is enabled and its trigger matches the current message |
| Telegram voice fails | Local Chat voice works first, then check the Telegram channel logs |
| Long replies are cut off | Shorten the response or increase the configured text limit within provider limits |
| A call reply stops while playing through speakers | Check voice interruption and response errors in Gateway logs. Verify microphone echo cancellation is available and try a headset to distinguish acoustic echo from an unexpected new response |
Use Settings → Logs or xopc logs tail to find the first provider error. Never share recordings or credentials in a support report unless you intend to disclose their contents.
Interaction and validation
Calls allow a short continuation window before answering. The default response pacing uses 1,200 ms of provider silence; common unfinished Chinese/English phrases get additional waiting time. Resuming speech discards a reply that has not yet been shown or played. During audible playback, detected microphone activity first lowers the volume; a finalized user utterance can then interrupt when barge-in is enabled. Speech matching the current playback is ignored. Natural voice requests the next upstream reply only after the turn settles, so an unconfirmed detection cannot replace the current audio. With barge-in disabled, the next reply waits until playback is acknowledged. This is pause/continuation and playback-aware interruption handling, not full acoustic speaker identification. The policy runs on the gateway and applies to mobile, web and desktop; deploy the updated gateway for connected clients to receive it.
Stop reply immediately clears playback and detaches the current voice rendering. A durable Agent task continues in the Chat; use the separate Cancel task action to abort it. Neither action sends a new message or undoes completed tool effects. Tool progress and explicit clarification/connector approval controls appear in the call. Ambient speech does not answer a pending clarification. Calls opened from a task retain its existing task status and detail link.
Calls use the product modes Current assistant (assistant) and Natural voice · no tools (natural). Session requests may omit mode to use the configured default; clients cannot select an internal provider engine. Active calls keep their original mode.
Run node scripts/voice-browser-smoke.mjs for production-component checks with Chrome synthetic microphone input and a fake gateway. Set XOPC_VOICE_SMOKE_BROWSER to another Chrome/Chromium executable if needed. This does not measure real acoustic quality. See delivery and audio acceptance.
Technical reference
See Voice runtime reference for latency metrics, segmentation, and connection preparation.