# Live avatar API documentation | Sequencer Canonical page: https://sequencer.media/live-avatar-docs API base URL: https://api.sequencer.media/v1 Authentication: x-api-key header, server only. Build real-time video avatars, voice-only chat, and text conversations with the Sequencer REST API. Video uses LiveKit. Voice-only and text sessions use a WebSocket. MCP is an optional assistant-driven setup path. ## Contents - [Overview](https://sequencer.media/live-avatar-docs#overview) - [Build with AI / MCP](https://sequencer.media/live-avatar-docs#tools) - [API quickstart](https://sequencer.media/live-avatar-docs#api-quickstart) - [Create an avatar](https://sequencer.media/live-avatar-docs#create-avatar) - [Choose a voice](https://sequencer.media/live-avatar-docs#voices) - [Prepare for video](https://sequencer.media/live-avatar-docs#prepare) - [Your backend](https://sequencer.media/live-avatar-docs#sessions) - [Video avatar](https://sequencer.media/live-avatar-docs#video-chat) - [Voice-only chat](https://sequencer.media/live-avatar-docs#voice-chat) - [Text chat](https://sequencer.media/live-avatar-docs#text-chat) - [Your own voice agent](https://sequencer.media/live-avatar-docs#audio-stream) - [Manage sessions](https://sequencer.media/live-avatar-docs#manage) - [All API endpoints](https://sequencer.media/live-avatar-docs#endpoints) ## Build with AI / MCP For assistant-driven setup, connect Sequencer to [ChatGPT](https://sequencer.media/mcp?setup=chatgpt-web#installation) or [Claude](https://sequencer.media/mcp?setup=claude-custom-connector#installation). Full setup: https://sequencer.media/mcp. MCP server: https://mcp.sequencer.media/. Starter prompt: Use Sequencer to create a live avatar from my attached portrait. Choose a warm English voice, keep replies short, and prepare it for a conversation. Share the avatar link and quote the cost before starting a call. Remote client configuration: ```json { "mcpServers": { "sequencer": { "url": "https://mcp.sequencer.media/" } } } ``` Local proxy configuration: ```json { "mcpServers": { "sequencer": { "command": "npx", "args": ["-y", "@sequencer.media/mcp"] } } } ``` The local proxy displays a pairing code. Enter it in Sequencer Settings > Connected Devices. ### MCP tool reference | Tool | Parameters | Purpose | | --- | --- | --- | | list_live_avatars | workspaceId | Find existing avatars before creating one. | | list_live_avatar_voices | workspaceId, search?, limit? | Find a real preset voice and its voiceId. | | create_live_avatar | workspaceId, name, avatarImageUrl, systemPrompt, … | Save an avatar draft. Supply a catalog voice ID or custom voice reference. | | get_live_avatar | workspaceId, characterId | Read settings, active version, and preparation progress. | | update_live_avatar | workspaceId, characterId, changed fields | Update supplied fields. Read the avatar first. | | prepare_live_avatar | workspaceId, characterId | Prepare the saved avatar for live video. | | get_live_avatar_readiness | workspaceId, characterId | Check preparation and runtime readiness. | | quote_live_avatar_session | workspaceId, characterId, maxSessionSeconds | Read the USD reservation for a bounded conversation. | | start_live_avatar_session | workspaceId, characterId, maxSessionSeconds, maxCostUsd | Start an approved, bounded voice and video conversation. | | send_live_avatar_message | workspaceId, avatarSessionId, message | Send text into the active voice conversation. | | stop_live_avatar_session | workspaceId, avatarSessionId | End the call and settle connected-time billing. | | link_live_avatar_to_worldstream | workspaceId, characterId, liveWorldId, expectedVersionId | Connect an avatar to an existing Worldstream draft. | ## API quickstart 1. Create an API key at https://sequencer.media/account?tab=api-keys. 2. List workspaces. Use workspaces[].id as SEQUENCER_WORKSPACE_ID. 3. Select a preset voice, create an avatar and save liveCharacter.id as SEQUENCER_CHARACTER_ID. 4. Prepare the avatar for video. Voice-only and text sessions skip video preparation. 5. Start a session on your authenticated backend with explicit channels and a duration limit. 6. Return session connection credentials to the browser. Close media connections and end the API session when finished. Keep API keys in server environment variables. Every session, message and end route in your app must authorize the signed-in user. Session credentials belong to the user who started that session. ### JavaScript ```javascript // Node.js 20+, on your server const response = await fetch('https://api.sequencer.media/v1/workspaces', { headers: { 'x-api-key': process.env.SEQUENCER_API_KEY } }); const data = await response.json(); if (!response.ok) throw new Error(data.error || 'Request failed'); console.log(data.workspaces); // Select a workspace's id ``` ### Python ```python # pip install requests import os, requests response = requests.get( "https://api.sequencer.media/v1/workspaces", headers={"x-api-key": os.environ["SEQUENCER_API_KEY"]}, timeout=30, ) response.raise_for_status() print(response.json()["workspaces"]) ``` ### cURL ```bash export SEQUENCER_API_KEY="sk_your_api_key" export SEQUENCER_API_BASE="https://api.sequencer.media/v1" curl --fail-with-body "$SEQUENCER_API_BASE/workspaces" \ -H "x-api-key: $SEQUENCER_API_KEY" ``` ### Environment variables ```bash # Server environment, or your local shell for the cURL examples. export SEQUENCER_API_KEY="YOUR_API_KEY" export SEQUENCER_API_BASE="https://api.sequencer.media/v1" export APP_ORIGIN="https://your-app.example" # Set SEQUENCER_WORKSPACE_ID after listing workspaces. # Set SEQUENCER_CHARACTER_ID after creating your avatar. ``` ## Choose a voice List the catalog and pass voices[].voiceId as providerVoiceId. Filter returned metadata for language and delivery. The REST catalog has no MCP-style search parameter. ```bash # Use an id from workspaces[].id, not a workspace name or URL slug. export SEQUENCER_WORKSPACE_ID="your_workspace_id" curl --fail-with-body "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-character-voices/presets" \ -H "x-api-key: $SEQUENCER_API_KEY" # Select voices[].voiceId from the returned catalog. export SEQUENCER_VOICE_ID="voice_id_from_catalog" ``` Preview a voice: GET /workspaces/{workspaceId}/live-character-voices/presets/{voiceId}/preview. Returns audio bytes, not JSON. For a custom voice, use voiceSource: custom with a stable voiceReferenceUrl and voiceDescription: ```jsonc // POST /workspaces/{workspaceId}/live-characters // Or PATCH /workspaces/{workspaceId}/live-characters/{characterId} { "voiceSource": "custom", "voiceReferenceUrl": "https://your-cdn.example/maya-voice.wav", "voiceDescription": "A warm, casual American English voice. Relaxed pace and playful delivery." } ``` ## Create and configure an avatar ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Content-Type: application/json" \ -d "{ \"name\": \"Maya\", \"avatarImageUrl\": \"https://your-cdn.example/maya.jpg\", \"voiceSource\": \"preset\", \"providerVoiceId\": \"$SEQUENCER_VOICE_ID\", \"systemPrompt\": \"You are Maya, a friendly, playful guide. Speak casually and keep replies to one or two sentences.\", \"runtimeProvider\": \"sequencer\", \"animaMode\": \"fast\", \"language\": \"en\", \"maxSessionMinutes\": 5 }" # Save liveCharacter.id from the response. export SEQUENCER_CHARACTER_ID="your_character_id" ``` - avatarImageUrl: stable HTTPS portrait URL. - voiceSource: preset or custom. Use a catalog providerVoiceId for a preset. - systemPrompt: role, tone and response instructions. - knowledgeText: approved facts and reference material. - openingLines: optional greetings. - language: language code such as en. - orientation: square, vertical or horizontal. - animaMode: fast or max (when enabled). - allowedOrigins: web app origins permitted to start sessions. - maxSessionSeconds or maxSessionMinutes: character duration limit. - cerebro, avatarAnimationSettings and moderationSettings: advanced personality, animation and moderation configuration. See the full configuration reference at https://sequencer.media/docs?tab=api#voice-agents. Save liveCharacter.id from the response. Read the avatar before editing and PATCH only changed fields: ```bash curl --fail-with-body -X PATCH \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "systemPrompt": "You are Maya. Be warm, witty, and concise. Ask one natural follow-up question at a time.", "knowledgeText": "Support hours: Monday to Friday, 9am to 5pm Pacific.", "openingLines": ["Hey, what brings you here?", "Hi! What are we getting into today?"], "cerebro": { "personalityTraits": { "extraversion": 75, "playfulness": 80 } }, "maxSessionMinutes": 5 }' ``` Direct portrait and voice file uploads require the Sequencer interface. API integrations pass hosted avatarImageUrl or voiceReferenceUrl when creating or updating. ## Prepare for video ```bash curl --fail-with-body --max-time 900 -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/prepare" \ -H "x-api-key: $SEQUENCER_API_KEY" # Get the latest configuration and preparation status. curl --fail-with-body \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID" \ -H "x-api-key: $SEQUENCER_API_KEY" ``` Wait for liveCharacter.status: active and liveCharacter.runtimeStatus: ready. While preparation is queued or running, poll GET avatar. Inspect runtimeError before retrying a failed job. Image, voice and mode changes can require another preparation. Preparation confirms saved assets; test actual speech and playback in a short session. Voice-only and text chat do not need video preparation. A voice-only session prepares its lightweight voice as needed. ## Your backend Choose the channels for sessionMode: conversation: | Experience | text | voice | video | Connection | | --- | --- | --- | --- | --- | | Video avatar | true | true | true | LiveKit audio and video | | Voice only | true | true | false | WebSocket audio and transcripts | | Text only | true | false | false | WebSocket transcripts | Set SEQUENCER_API_KEY, SEQUENCER_WORKSPACE_ID, SEQUENCER_CHARACTER_ID and APP_ORIGIN (your web app origin). The following helper starts regular sessions and ends them: ```javascript // avatar-api.js, server only, Node.js 20+ const BASE = 'https://api.sequencer.media/v1'; const key = process.env.SEQUENCER_API_KEY; const workspace = process.env.SEQUENCER_WORKSPACE_ID; const character = process.env.SEQUENCER_CHARACTER_ID; if (!key || !workspace || !character) throw new Error('Missing avatar configuration'); const wsPath = '/workspaces/' + encodeURIComponent(workspace); export async function avatarApi(path, { method = 'GET', body, origin } = {}) { const response = await fetch(BASE + wsPath + path, { method, headers: { 'x-api-key': key, 'Content-Type': 'application/json', // For sessions, set this to your configured web app origin. ...(origin ? { Origin: origin } : {}) }, ...(body ? { body: JSON.stringify(body) } : {}) }); const data = await response.json(); if (!response.ok) { const error = new Error(data.error || 'Avatar API request failed'); error.status = response.status; error.retryAfter = response.headers.get('Retry-After'); throw error; } return data; } export function startAvatar(userId, channels, conversationHistory = [], startToken) { return avatarApi('/live-characters/' + encodeURIComponent(character) + '/sessions', { method: 'POST', origin: process.env.APP_ORIGIN, // e.g. https://your-app.example body: { launchMode: 'webrtc', sessionMode: 'conversation', channels, endUserId: userId, maxSessionSeconds: 300, saveHistory: false, ...(conversationHistory.length ? { conversationHistory } : {}), ...(startToken ? { startToken } : {}) } }); } export function endAvatar(sessionId) { return avatarApi('/live-character-sessions/' + encodeURIComponent(sessionId) + '/end', { method: 'POST', body: { reason: 'USER_CLOSED' } }); } ``` Mount these Express routes on your app. Provide your own authentication middleware, session store and history store. The authenticated user ID comes from your backend, not the request body. Browser examples below call these routes: ```javascript // Express example, mount these on your own backend. import express from 'express'; import { startAvatar, endAvatar } from './avatar-api.js'; import { requireSignedInUser, sessionStore, historyStore } from './your-app-auth.js'; const router = express.Router(); router.use(express.json(), requireSignedInUser); const modes = { text: { text: true, voice: false, video: false }, voice: { text: true, voice: true, video: false }, video: { text: true, voice: true, video: true } }; router.post('/session', async (req, res, next) => { try { const channels = modes[req.body.mode]; if (!channels) return res.status(400).json({ error: 'Unknown mode' }); // req.user.id comes from your authentication middleware. // historyStore is your own database, scoped by authenticated user and avatar. const history = await historyStore.recent({ userId: req.user.id, workspaceId: process.env.SEQUENCER_WORKSPACE_ID, characterId: process.env.SEQUENCER_CHARACTER_ID, limit: 60, maxBytes: 65536 }); const session = await startAvatar(req.user.id, channels, history); try { await sessionStore.save(session.sessionId, { userId: req.user.id }); } catch (error) { await endAvatar(session.sessionId); throw error; } res.json(session); // Session credentials only, never your API key. } catch (error) { next(error); } }); router.post('/session/:id/end', async (req, res, next) => { try { const owned = await sessionStore.get(req.params.id); if (owned?.userId !== req.user.id) return res.sendStatus(404); res.json(await endAvatar(req.params.id)); } catch (error) { next(error); } }); export default router; // app.use('/api/avatar', router) // Add your app's error handler to return error.status and a safe message. ``` ## Video avatar Start a conversation with text, voice and video enabled: ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/sessions" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Origin: https://your-app.example" \ -H "Content-Type: application/json" \ -d '{ "launchMode": "webrtc", "sessionMode": "conversation", "channels": { "text": true, "voice": true, "video": true }, "endUserId": "visitor_123", "saveHistory": false, "maxSessionSeconds": 300 }' ``` Returns sessionId, livekitUrl, livekitClientToken, channels and billing. Connect audio and video with LiveKit: ```javascript // HTML:
// import { Room, RoomEvent, Track } from 'https://sequencer.media/realtime/client-v1.js'; let room, session, starting = false, ending = false; const container = document.querySelector('#avatar'); async function endCall() { if (ending) return; ending = true; try { await room?.disconnect(); // Also unpublishes the microphone. container.replaceChildren(); if (session) await fetch('/api/avatar/session/' + session.sessionId + '/end', { method: 'POST' }); room = session = undefined; } finally { ending = false; } } document.querySelector('#start').onclick = async () => { if (session || starting || ending) return; starting = true; room = new Room(); room.on(RoomEvent.TrackSubscribed, track => { if (track.kind !== Track.Kind.Video && track.kind !== Track.Kind.Audio) return; const element = track.attach(); if (track.kind === Track.Kind.Video) { element.playsInline = true; element.style.width = '100%'; } container.appendChild(element); }); room.on(RoomEvent.TrackUnsubscribed, track => { track.detach().forEach(element => element.remove()); }); room.on(RoomEvent.Disconnected, () => { void endCall().catch(console.error); }); try { const response = await fetch('/api/avatar/session', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ mode: 'video' }) }); if (!response.ok) throw new Error('Could not start video session'); session = await response.json(); await room.connect(session.livekitUrl, session.livekitClientToken); await room.startAudio(); await room.localParticipant.setMicrophoneEnabled(true); } catch (error) { await endCall(); throw error; } finally { starting = false; } }; document.querySelector('#end').onclick = endCall; // Provide an Enable audio button that calls room.startAudio() if autoplay is blocked. // For typed input, use your backend's messages route from Send a text turn. ``` Start from a user click. Request microphone access before allocating a paid session where practical. The player imports the public LiveKit-compatible client at https://sequencer.media/realtime/client-v1.js. With a bundler, use livekit-client. Attach audio and video tracks, publish the microphone and handle blocked autoplay. End the API session after disconnecting, including on errors and navigation. The legacy embed endpoint is an HTML status page, not a complete LiveKit player. ## Voice-only chat Use the same avatar personality and voice with video disabled. No LiveKit player or video preparation is needed. ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/sessions" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Origin: https://your-app.example" \ -H "Content-Type: application/json" \ -d '{ "launchMode": "webrtc", "sessionMode": "conversation", "channels": { "text": true, "voice": true, "video": false }, "endUserId": "visitor_123", "saveHistory": false, "maxSessionSeconds": 300 }' ``` The response includes sessionId and conversationStream: {url, token}. In the browser: 1. Request microphone access from a user click over HTTPS or localhost. 2. Open conversationStream.url and send {token: conversationStream.token, sampleRate: 48000} as the first JSON message. 3. Wait for ready. Send {type: microphone_start, sampleRate: audioContext.sampleRate} and wait for microphone_ready. 4. Send mono little-endian PCM16 binary audio frames. Do not send encoded MediaRecorder/WebM chunks. 5. Play audio events: base64 PCM16 in pcm, sample rate in sampleRate. Display transcript events and replace streaming messages by id. Clear scheduled audio on interrupt. 6. Stop the microphone, close the WebSocket and AudioContext, and end the API session on End, errors and navigation. ### WebSocket helper: conversation.js ```javascript // conversation.js, browser export function connectConversation(session, onEvent, onClose = () => {}) { return new Promise((resolve, reject) => { const socket = new WebSocket(session.conversationStream.url); let ready = false; const timer = setTimeout(() => { reject(new Error('Conversation connection timed out')); socket.close(); }, 15000); socket.onopen = () => socket.send(JSON.stringify({ token: session.conversationStream.token, sampleRate: 48000 })); socket.onmessage = ({ data }) => { const event = JSON.parse(data); if (event.type === 'ready') { ready = true; clearTimeout(timer); resolve(socket); } if (event.type === 'error' && !ready) { clearTimeout(timer); reject(new Error(event.message)); socket.close(); } onEvent(event); }; socket.onerror = () => { if (!ready) { clearTimeout(timer); reject(new Error('WebSocket connection failed')); } }; socket.onclose = () => { clearTimeout(timer); if (!ready) reject(new Error('Connection closed before ready')); onClose(); }; }); } ``` ### Microphone worklet: public/pcm-capture.js ```javascript // public/pcm-capture.js, served from your app's origin class PcmCapture extends AudioWorkletProcessor { process(inputs) { const channels = inputs[0]; if (!channels?.length) return true; const count = channels[0].length; const pcm = new ArrayBuffer(count * 2); const view = new DataView(pcm); for (let i = 0; i < count; i++) { let mono = 0; for (const channel of channels) mono += channel[i] / channels.length; mono = Math.max(-1, Math.min(1, mono)); view.setInt16(i * 2, mono < 0 ? mono * 32768 : mono * 32767, true); } this.port.postMessage(pcm, [pcm]); return true; } } registerProcessor('pcm-capture', PcmCapture); ``` ### Browser audio client Call startVoiceChat() from a click. Wire the returned cleanup function to your End button and navigation cleanup. It reuses conversation.js above and your authenticated backend routes. ```javascript // Start from a click. Reuse connectConversation from the Text chat guide. import { connectConversation } from './conversation.js'; export async function startVoiceChat() { const context = new AudioContext({ sampleRate: 48000 }); await context.resume(); const activeAudio = new Set(); let nextAudio = 0, microphoneReady = false, session, socket, mic, capture, closing = false; const clearAudio = () => { for (const source of activeAudio) source.stop(); activeAudio.clear(); nextAudio = 0; }; const cleanup = async () => { if (closing) return; closing = true; microphoneReady = false; capture?.disconnect(); mic?.getTracks().forEach(track => track.stop()); socket?.close(); clearAudio(); await context.close(); if (session) await fetch('/api/avatar/session/' + session.sessionId + '/end', { method: 'POST' }); }; try { // Request permission before allocating a paid session. mic = await navigator.mediaDevices.getUserMedia({ audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true } }); await context.audioWorklet.addModule('/pcm-capture.js'); const response = await fetch('/api/avatar/session', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ mode: 'voice' }) }); if (!response.ok) throw new Error('Could not start voice session'); session = await response.json(); socket = await connectConversation(session, event => { if (event.type === 'microphone_ready') microphoneReady = true; if (event.type === 'transcript') console.log(event.role, event.text, event.final); if (event.type === 'error') console.error(event.message); if (event.type === 'interrupt') clearAudio(); if (event.type !== 'audio') return; const bytes = Uint8Array.from(atob(event.pcm), char => char.charCodeAt(0)); const view = new DataView(bytes.buffer); const buffer = context.createBuffer(1, bytes.length / 2, event.sampleRate); const samples = buffer.getChannelData(0); for (let i = 0; i < samples.length; i++) samples[i] = view.getInt16(i * 2, true) / 32768; const source = context.createBufferSource(); source.buffer = buffer; source.connect(context.destination); const when = Math.max(context.currentTime + 0.02, nextAudio); nextAudio = when + buffer.duration; activeAudio.add(source); source.onended = () => activeAudio.delete(source); source.start(when); }, () => { void cleanup().catch(console.error); }); socket.send(JSON.stringify({ type: 'microphone_start', sampleRate: context.sampleRate })); const input = context.createMediaStreamSource(mic); capture = new AudioWorkletNode(context, 'pcm-capture'); capture.port.onmessage = ({ data }) => { if (microphoneReady && socket.readyState === WebSocket.OPEN) { if (socket.bufferedAmount > 192000) { socket.close(); return; } socket.send(data); // Mono little-endian PCM16 binary, not encoded WebM. } }; const silent = context.createGain(); silent.gain.value = 0; input.connect(capture).connect(silent).connect(context.destination); return cleanup; // Call on your End button, including on errors. } catch (error) { await cleanup(); throw error; } } ``` The ready event includes messages. transcript events contain id, role, text and final. audio events contain id, pcm and sampleRate. Handle error events and connection closure. microphone_ready permits sending input audio; interrupt clears pending playback. ## Text chat ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/sessions" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Origin: https://your-app.example" \ -H "Content-Type: application/json" \ -d '{ "launchMode": "webrtc", "sessionMode": "conversation", "channels": { "text": true, "voice": false, "video": false }, "endUserId": "visitor_123", "saveHistory": false, "maxSessionSeconds": 300 }' ``` Use conversationStream with the same conversation.js helper. After ready, send {type: text_turn, id: crypto.randomUUID(), text: message}. Receive transcript events and replace messages by id as they stream. ```javascript // HTML: //
//
import { connectConversation } from './conversation.js'; let session, socket, starting = false, ending; const messages = new Map(); function upsert(message) { messages.set(message.id, message); const lines = [...messages.values()].map(m => m.role + ': ' + m.text); document.querySelector('#transcript').textContent = lines.join('\n'); } async function endChat() { if (ending) return ending; const current = session; session = undefined; socket?.close(); socket = undefined; ending = current ? fetch('/api/avatar/session/' + current.sessionId + '/end', { method: 'POST' }) : Promise.resolve(); try { await ending; } finally { ending = undefined; } } document.querySelector('#start').onclick = async () => { if (session || starting || ending) return; starting = true; try { const response = await fetch('/api/avatar/session', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ mode: 'text' }) }); if (!response.ok) throw new Error('Could not start session'); session = await response.json(); messages.clear(); socket = await connectConversation(session, event => { if (event.type === 'ready') event.messages.forEach(upsert); if (event.type === 'transcript') upsert(event); // Replace by id as text streams. if (event.type === 'error') console.error(event.message); }, () => { void endChat().catch(console.error); }); } catch (error) { await endChat(); throw error; } finally { starting = false; } }; document.querySelector('#chat').onsubmit = event => { event.preventDefault(); const input = document.querySelector('#message'); const text = input.value.trim(); if (!text || socket?.readyState !== WebSocket.OPEN) return; socket.send(JSON.stringify({ type: 'text_turn', id: crypto.randomUUID(), text })); input.value = ''; }; document.querySelector('#end').onclick = endChat; ``` ## Your own voice agent Use sessionMode: audio_stream to drive avatar video with your own agent's speech: ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/sessions" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Origin: https://your-app.example" \ -H "Content-Type: application/json" \ -d '{ "launchMode": "webrtc", "sessionMode": "audio_stream", "channels": { "text": false, "voice": true, "video": true }, "endUserId": "visitor_123", "saveHistory": false, "maxSessionSeconds": 300 }' ``` Prepare the avatar for video. Connect to LiveKit using the video guide, then publish your speech as a microphone track on the connected room. Track is exported by the same LiveKit client. ```javascript // Browser: connect to LiveKit using the Connect with LiveKit guide. // Feed your own agent's speech into a Web Audio destination. const context = new AudioContext(); await context.resume(); const output = context.createMediaStreamDestination(); // Connect your speech-producing AudioNode to output. // Do not also connect that speech to speakers, or it will play twice. const speechTrack = output.stream.getAudioTracks()[0]; await room.localParticipant.publishTrack(speechTrack, { source: Track.Source.Microphone, dtx: false }); // Keep this track alive through pauses. Let returned audio finish before ending. // On End: disconnect the room, close the AudioContext, and call End Session. ``` Keep the track alive through pauses. Let returned audio finish, disconnect the room, close your AudioContext and end the API session. audio_stream does not run the built-in conversation agent or accept text turns. ## Manage sessions ### Send a message ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-character-sessions/{sessionId}/messages" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "message": "Hey Maya, what should we try first?" }' ``` Video returns an acknowledgement; lightweight sessions can return message and messageId. Receive streamed answers through WebSocket transcripts or LiveKit playback. ### End and cancel ```bash curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-character-sessions/{sessionId}/end" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "reason": "USER_CLOSED" }' # For an abandoned start that has not returned a sessionId: curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-character-session-starts/{startToken}/cancel" \ -H "x-api-key: $SEQUENCER_API_KEY" ``` Supply your own unique startToken on the start request if you need to cancel a pending start. If a late response returns a sessionId, end that session as well. Closing a player alone is insufficient to settle billing promptly. ### History ```jsonc // Default: your backend owns and supplies history for the authenticated user. { "saveHistory": false, "endUserId": "visitor_123", "conversationHistory": [ { "role": "user", "content": "My dog is named Luna." }, { "role": "assistant", "content": "How is Luna doing?" } ] } // Add the usual channels and session fields. Save final transcript events in // your backend and supply the relevant recent messages when starting again. // Maximum: 60 messages, 16000 characters per message, 64 KiB total UTF-8 content. // New API conversations use client-managed history. Your backend chooses // its retention policy. Hosted history creation is reserved for Sequencer's // own products. Existing legacy hosted conversation IDs can still resume. ``` Save transcript events in your own backend, scoped by authenticated user, workspace and avatar. Preserve your app's conversation ID across calls. Maximum: 60 messages, 16000 characters per message and 64 KiB total UTF-8 content. New API sessions use client-managed history. Existing legacy hosted conversation IDs can still resume. ### Billing and bounded video calls Session start reserves funds; ending reconciles usage and returns unused reservation. Text, voice and video have different rates. Read billing.reservedUsd and billing.usdPerMinute on regular session responses. POST /live-characters/{characterId}/sessions/quote is specifically a video conversation quote. Pass maxSessionSeconds (30 to 3600, within the avatar limit), display reservedUsd, then pass an approved maxCostUsd and maxSessionSeconds to /sessions with video conversation channels. /bounded-sessions is also available for prepared avatars and adds avatarSessionId, usable in session paths. maxCostUsd is supported only for video conversation channels, not text or voice-only sessions. ```javascript // Server only. Reuse avatar-api.js from Your backend. import { avatarApi } from './avatar-api.js'; const characterPath = '/live-characters/' + encodeURIComponent(process.env.SEQUENCER_CHARACTER_ID); // Show reservedUsd to the user before starting a video call. const quote = await avatarApi(characterPath + '/sessions/quote', { method: 'POST', body: { maxSessionSeconds: 60 } }); console.log(quote.reservedUsd); // Invoke from an authenticated handler after the amount is approved. export function startQuotedVideo(userId, approvedMaxCostUsd) { return avatarApi(characterPath + '/sessions', { method: 'POST', origin: process.env.APP_ORIGIN, body: { launchMode: 'webrtc', sessionMode: 'conversation', channels: { text: true, voice: true, video: true }, endUserId: userId, saveHistory: false, maxSessionSeconds: quote.maxSessionSeconds, maxCostUsd: approvedMaxCostUsd } }); } // Store returned sessionId against this user. End it when finished. // /bounded-sessions is also available for prepared video avatars; // it returns avatarSessionId alongside the connection fields. ``` ### Animations ```bash # Upload a source clip and describe when to use it. curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/custom-animations/upload" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -F "file=@wave.mp4" -F "name=Wave" -F "trigger=Wave when greeting a visitor" # Prepare the returned customAnimation.id. curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-characters/$SEQUENCER_CHARACTER_ID/custom-animations/{animationId}/prepare" \ -H "x-api-key: $SEQUENCER_API_KEY" # Once its status is ready, trigger it in an active video conversation. curl --fail-with-body -X POST \ "$SEQUENCER_API_BASE/workspaces/$SEQUENCER_WORKSPACE_ID/live-character-sessions/{sessionId}/animations/trigger" \ -H "x-api-key: $SEQUENCER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "animationId": "your_animation_id" }' ``` Read GET avatar for customAnimations progress. Prepare a motion before triggering it in a compatible live video conversation. ## All avatar API endpoints All paths use the API origin https://api.sequencer.media. The full /v1 prefix is shown below. Send x-api-key on workspace routes except the legacy embed page, which uses its session token. ### Workspaces & characters #### GET /v1/workspaces List workspaces. Request: No body. Requires an API key. Response: workspaces[].id and workspace metadata. Use the id in subsequent paths. #### GET /v1/workspaces/{workspaceId}/live-characters List avatars. Request: No body. Response: liveCharacters[] with saved configuration and preparation status. #### POST /v1/workspaces/{workspaceId}/live-characters Create an avatar. Request: JSON: name, avatarImageUrl, systemPrompt, voiceSource, providerVoiceId (preset) or voiceReferenceUrl and voiceDescription (custom). Set runtimeProvider to sequencer. Optional knowledgeText, openingLines, language, orientation, animaMode, allowedOrigins, maxSessionSeconds, cerebro and avatarAnimationSettings. Response: 201. liveCharacter.id, configuration and status. Save the id as characterId. #### GET /v1/workspaces/{workspaceId}/live-characters/{characterId} Get an avatar. Request: No body. Response: liveCharacter, including runtimeStatus, prepStatus, runtimeError, activeVersionId and customAnimations. For video, wait for status: active and runtimeStatus: ready. #### PATCH /v1/workspaces/{workspaceId}/live-characters/{characterId} Update an avatar. Request: JSON: only changed creation fields. status can be active, paused or archived. Image, voice and mode changes can require new video preparation. Response: Updated liveCharacter. #### POST /v1/workspaces/{workspaceId}/live-characters/{characterId}/prepare Prepare video assets. Request: No body required. Poll GET avatar for progress. Response: liveCharacter, prepared and prepJobId. Queued work can outlive the request. Skip video preparation for text and voice-only sessions. #### DELETE /v1/workspaces/{workspaceId}/live-characters/{characterId} Archive an avatar. Request: No body. Requires write access. Response: Archives the avatar. New sessions cannot start for an archived character. #### POST /v1/workspaces/{workspaceId}/live-characters/move/preview Preview a workspace move. Request: JSON: characterIds (array). Requires write access to the source workspace. Response: Affected characters, worlds, counts and canMoveCharactersOnly. Review dependencies before moving. #### POST /v1/workspaces/{workspaceId}/live-characters/move Move avatars. Request: JSON: characterIds, targetWorkspaceId, includeWorlds (boolean). Requires write access to both workspaces. Preview dependencies first. Response: Moved characterIds, sourceWorkspaceId, targetWorkspaceId, movedCount, movedWorldCount and warnings. ### Voices & preparation #### GET /v1/workspaces/{workspaceId}/live-character-voices/presets List preset voices. Request: No body. Filter the returned catalog in your application. Response: voices[].voiceId and voice metadata. Pass voiceId as providerVoiceId when saving an avatar. #### GET /v1/workspaces/{workspaceId}/live-character-voices/presets/{voiceId}/preview Preview a voice. Request: voiceId from the catalog. Response: Audio bytes with the appropriate Content-Type. This response is not JSON. #### POST /v1/workspaces/{workspaceId}/live-character-voices/provision Provision generated voice audio. Request: JSON: sourceMediaId, voiceReferenceUrl, voiceDescription (at least 20 characters), language (optional). sourceMediaId must identify completed audio in this workspace and its URL must match. For a normal custom voice, set voiceReferenceUrl on the avatar instead. Response: 200 when ready, otherwise 202. Voice draft status and providerVoiceId when available. #### POST /v1/workspaces/{workspaceId}/live-character-visual-description Describe a portrait. Request: JSON: avatarImageUrl (or sourceImageUri). Requires write access. Response: characterVisualDescription generated from the image. #### POST /v1/workspaces/{workspaceId}/live-character-runtime/prepare Check or warm the runtime. Request: JSON: characterId, statusOnly, ttlSeconds and waitSeconds (optional). statusOnly: true inspects the cache. Warming can incur costs and requires write access. Response: 200 if ready, otherwise 202. ready, status and cache or setup estimates. This is separate from preparing the saved avatar. ### Sessions & playback #### POST /v1/workspaces/{workspaceId}/live-characters/{characterId}/sessions Start a session. Request: JSON: launchMode: webrtc, sessionMode: conversation or audio_stream, channels: {text, voice, video}, endUserId and maxSessionSeconds. Optional startToken, initialText, dynamicVariables, conversationHistory and saveHistory: false. Forward your configured web app Origin header. For a cost cap, maxCostUsd is supported on video conversation sessions only. Response: 201. sessionId, channels, billing and connection credentials. Video returns livekitUrl and livekitClientToken. Text and voice-only return conversationStream.url and conversationStream.token. #### POST /v1/workspaces/{workspaceId}/live-characters/{characterId}/sessions/quote Quote a bounded video call. Request: JSON: maxSessionSeconds (integer, 30 to 3600, within the avatar limit). This quote is for video conversation sessions. Response: characterId, maxSessionSeconds, currency: usd, reservedUsd, ready and preparationRequired. #### POST /v1/workspaces/{workspaceId}/live-characters/{characterId}/bounded-sessions Start a bounded video call. Request: JSON: maxSessionSeconds and maxCostUsd covering the quote. Optional endUserId and legacy conversationId. Requires a prepared avatar. Uses text, voice and video channels. Response: 201. Video session connection fields plus avatarSessionId and maxCostUsd. Use avatarSessionId for message, metrics and end paths. #### POST /v1/workspaces/{workspaceId}/live-character-session-starts/{startToken}/cancel Cancel a pending start. Request: Use the startToken you supplied when starting. No body required. Response: Cancellation status. If a start completes with a sessionId, end that session too. #### POST /v1/workspaces/{workspaceId}/live-character-sessions/{sessionId}/messages Send a text turn. Request: JSON: message. Up to 4,000 characters. Requires text channels and conversation mode. Receive replies through WebSocket transcripts or LiveKit playback. Response: Video returns an acknowledgement. Lightweight sessions can return message and messageId. Use WebSocket transcript events for streamed replies. #### GET /v1/workspaces/{workspaceId}/live-character-sessions/{sessionId}/metrics Read session metrics. Request: Optional query: live=true for current runtime diagnostics. Response: Session usage, billing and available runtime diagnostics. #### POST /v1/workspaces/{workspaceId}/live-character-sessions/{sessionId}/end End a session. Request: JSON: reason (optional, defaults to USER_CLOSED). Close browser audio/video connections too. Response: Ends the session and reconciles billing. Unused reservation is returned. #### POST /v1/workspaces/{workspaceId}/live-character-presets/{presetId}/sessions Start a preset avatar. Request: A supported presetId and the same body as Start a session. Forward the configured app Origin. Response: 201. A new session with the selected preset and the requested channels. #### GET /v1/workspaces/{workspaceId}/live-character-presets/{presetId}/status Check a preset avatar. Request: A supported presetId. No body. Response: Preset availability and preparation status. #### GET /v1/workspaces/{workspaceId}/live-character-sessions/{sessionId}/embed Read the legacy embed page. Request: Query: token, the session embed capability. This is an HTML status page. Use LiveKit to implement working live video playback. Response: HTML. A token is required. Video is only attached when a direct playback URL is enabled; this is not a ready-made LiveKit player. ### Conversation history #### GET /v1/workspaces/{workspaceId}/avatar-conversations/{conversationId} Read legacy hosted history. Request: Optional query: endUserId, limit and cursor. Scope access to the original user. New API integrations use their own history store and supply conversationHistory at session start. Response: Authorized hosted conversation and paginated messages. Existing hosted IDs can resume; new API sessions use client-managed history. #### DELETE /v1/workspaces/{workspaceId}/avatar-conversations/{conversationId} Delete legacy hosted history. Request: Optional query: endUserId. Requires authorized access to this conversation. Response: 202. Schedules history deletion. #### GET /v1/workspaces/{workspaceId}/worldstream-avatar-conversations/{conversationId} Read Worldstream history. Request: Signed-in viewer access to their own Worldstream conversation. Optional query: limit and cursor. Response: Viewer-scoped hosted conversation history. #### DELETE /v1/workspaces/{workspaceId}/worldstream-avatar-conversations/{conversationId} Delete Worldstream history. Request: Signed-in viewer access to their own Worldstream conversation. Response: 202. Schedules viewer-scoped history deletion. ### Animations & Worldstreams #### POST /v1/workspaces/{workspaceId}/live-characters/{characterId}/custom-animations/upload Upload an animation. Request: multipart/form-data: file (video clip), name and trigger. Optional animationId to replace a clip. Response: 202. customAnimation.id and preparation status. #### PATCH /v1/workspaces/{workspaceId}/live-characters/{characterId}/custom-animations/{animationId} Update an animation. Request: JSON: name and/or trigger guidance. Response: Updated custom animation metadata. #### POST /v1/workspaces/{workspaceId}/live-characters/{characterId}/custom-animations/{animationId}/prepare Prepare an animation. Request: Optional JSON: force (boolean). Read the avatar for animation progress. Response: 200 when prepared, otherwise 202. #### DELETE /v1/workspaces/{workspaceId}/live-characters/{characterId}/custom-animations/{animationId} Delete an animation. Request: No body. Response: Removes the animation from the character. #### POST /v1/workspaces/{workspaceId}/live-character-sessions/{sessionId}/animations/trigger Play an animation. Request: JSON: animationId of a prepared motion. Requires a compatible active video conversation. Response: Animation trigger acknowledgement. #### POST /v1/workspaces/{workspaceId}/live-worlds/{liveWorldId}/avatar-links Link an avatar to a Worldstream. Request: JSON: characterId and expectedVersionId from the current Worldstream draft. Response: Updated draft with the avatar link. A stale expectedVersionId is rejected. ## Errors Check HTTP status and the JSON error. 401/403: verify key, workspace membership, write access and allowedOrigins. 429: honor Retry-After. Preparation failure: inspect runtimeError. Video cap rejection: refresh the quote and use an approved cap covering reservedUsd. Missing playback: verify connection credentials, subscribed tracks, browser audio permissions and microphone access. End any allocated session after a connection failure. ## Instructions for AI agents 1. Use the REST API for hand-coded app integrations. Keep x-api-key on the server and authorize your own application routes. 2. List workspaces and existing avatars first. Use real returned IDs and a providerVoiceId from the voice catalog. 3. Create or update the avatar with an HTTPS image and voice configuration. Prepare video assets for video sessions. 4. Choose channels explicitly. Video uses LiveKit; voice-only and text use conversationStream over WebSocket. 5. For voice input, authenticate the WebSocket, wait for ready and microphone_ready, and send mono little-endian PCM16. Honor the sampleRate on returned audio events. 6. For bounded video calls, quote first and pass approved maxCostUsd and maxSessionSeconds. Text and voice-only sessions use the regular session endpoint. 7. Store session ownership on your server. Persist transcripts in your own history store and supply conversationHistory with saveHistory: false. 8. Close media connections and end API sessions on End, errors and navigation. Report preparation, mock checks and live playback verification separately. 9. Use MCP only when assistant-driven tools are requested. Do not invent REST readiness or voice-search endpoints from MCP tool names. ## Related documentation - [Live avatars](https://sequencer.media/live-avatar) - [Full API configuration reference](https://sequencer.media/docs?tab=api#voice-agents) - [MCP setup](https://sequencer.media/mcp) - [Worldstream API](https://sequencer.media/worldstream-api) - [Agent documentation index](https://sequencer.media/llms.txt)