The WebSocket

Every conversation with an agent happens on one WebSocket. Tokens arrive as the model writes them, and a speaking agent’s reply is spoken as it is written: the first sentence is audio while the third is still being generated. It also takes audio in, so a spoken turn is one connection rather than an upload followed by a request. There is no request-and-response chat endpoint.

OpenAPI describes requests and responses, and this is neither, so the protocol is not in the API reference. Everything it needs is on this page.

Most apps never write it by hand: the client SDK, and the React hooks over it, speak it for you — the queue behind a running turn, reconnecting by the close code, a new key when one is refused, each recording’s words by its recording_id. This page is the protocol itself, for another language or a look underneath.

JavaScript

const ws = new WebSocket(`wss://api.lumavox.ai/v1/agent/stream?session_id=${sessionId}`, "ino.v2");
ws.onopen = () =>
  ws.send(JSON.stringify({ type: "auth", key: token, session_id: sessionId }));
ws.onmessage = ({ data }) => console.log(data);

Before you connect

You need two things from your own server and the API:

1. An ephemeral token for the person who is talking, minted on your server with POST /v1/ephemeral-keys. See End users and their keys. Never your API key: an API key on this socket is closed with 4002.
2. A session id, from POST /v1/agent/sessions called with that token. The socket joins a conversation; it does not create one. See Conversations and memory.

Connect and authenticate

Connect to wss://api.lumavox.ai/v1/agent/stream?session_id=<the session's id> with the subprotocol ino.v2, then send an auth frame as the first message:

auth

{ "type": "auth", "key": "<ephemeral token>", "session_id": "<the session's id>" }

The credential goes in a frame rather than the URL or a header because a browser cannot set an Authorization header on a WebSocket, and a credential in a query string ends up in every proxy log on the way. The URL and the frame must name the same session, or the socket closes with 4009. With no auth frame within 10 seconds, or a different frame first, it closes with 4001.

An app that runs tools of its own names them in the same frame, as tools: up to 50 names. The agent is offered only those of its app tools named here; leave tools out and it is offered none.

auth, with the app's tools

{ "type": "auth", "key": "<ephemeral token>", "session_id": "<the session's id>", "tools": ["open_page"] }

The answer is a ready frame:

ready

{
  "type": "ready",
  "seq": 1,
  "protocol": "ino.v2",
  "session_id": "0193b1f4-8e02-7c3a-a1d5-6b7e90c4f218",
  "agent_id": "0193b1e2-6c1d-7a4e-9f20-5d8a3c2b1e07",
  "end_user_id": "eu_3kQ9x7MzT2pL8vNc",
  "speaks": true,
  "audio":   { "codec": "pcm_s16le", "sample_rate": 24000, "channels": 1 },
  "hearing": { "codec": "pcm_s16le", "sample_rate": 16000, "channels": 1 },
  "emotion": true,
  "relation": null,
  "greeting": true,
  "idle_seconds": 300,
  "edge_version": "0.220.3"
}

Field	Meaning
speaks	Whether the agent replies with audio.
audio	The format replies come back in.
hearing	The format a recording is assumed to be in when user_audio.start names no rate. A different number from audio, and neither can be inferred from the other.
emotion	Whether replies report a mood.
relation	The standing relationship with this person, with the same fields as GET /v1/agent/relations, so “you have spoken twelve times” needs no second request. null when the agent keeps none, which is not the same as never having met.
greeting	true when the agent is about to speak first.
idle_seconds	How long the connection may be idle before it is closed with 4007.

When greeting is true, a turn is already running: the agent’s opening line, streamed and spoken like any other reply. Wait for turn.done before you send anything. A client that treats ready as “idle” has its first message refused with an error frame, because the turn starts the moment ready is written. When greeting is false, the connection is idle.

The connection is pinned to that conversation. Nothing can change it; a different conversation is a different socket.

What you send

Nine verbs, each a JSON text frame with a type. The connection accepts no others.

Verb	What it does
auth	The first frame. Carries the credential (key) and the session_id, and the app tools this connection runs, tools. Sending it again later closes the socket with 4009.
user_message	Take a turn by typing: text, 1 to 32,000 characters. speech (true or false) overrides the agent’s own switch for this turn; thoughts: true asks for thinking summaries.
user_audio.start	Open a recording. Optional rate, language (2 to 8 characters), model, recording_id (your name for the recording, 1 to 64 characters, echoed on its transcript and on an error about it), barge_in (true for a hands-free recording that may start over the agent’s voice: see Talking over the agent), plus the same speech and thoughts.
user_audio.end	Close it. The recording is transcribed and becomes the turn, unless transcribe_only is true, in which case the words come back on transcript and no turn is taken, or discard is true, in which case it is thrown away without a word and a running turn goes on.
cancel	Stop the turn in flight, and discard any open recording.
mute	Stop speaking, but let the turn finish. Answered with muted after the last audio.end.
ping	Liveness: send exactly {"type":"ping"} after 30 seconds with nothing from the server. Answered with pong, and does not count as activity.
tool.confirmed	The person’s answer to a tool.confirm: call_id and yes (true or false), plus the same speech and thoughts. It is a turn of theirs — “Yes.” or “No.” in the transcript — and on a yes the tool runs before the agent replies. A call that is no longer waiting is answered with an error, not_found.
tool.result	Your app’s answer to a tool.call: its call_id, then result (any JSON) or error (what went wrong, 1 to 2,000 characters), which is the answer when you send it. One for a call that is over — answered, timed out, or never made — is ignored.

user_message

{ "type": "user_message", "text": "What should I see on my first morning?" }

A connection takes one turn at a time. A message sent while a turn is running is refused with an error frame, and the turn carries on.

thoughts only shows anything for an agent whose thinking_summaries is auto. The summaries arrive as thought.delta and are never part of the reply or the transcript.

What arrives

Every JSON frame but pong carries type and seq. seq rises by one with every frame on the connection, so a gap is a bug, not a lost frame. Everything produced inside a turn also carries turn, the id from that turn’s turn.start. A frame always has every field its event names, null where one does not apply, and its times and units are written as everywhere else in the API (Conventions).

A frame you send may hold only the fields its verb takes: anything else — a typo, a field from another verb — closes the socket with 4009 and a reason naming it.

Event	When, and what it carries
ready	After auth. See above.
turn.start	A turn began. turn, session_id.
heard.partial	While a recording is open: the transcript so far, text. Show it as the person speaks.
barge_in	A barge_in recording heard the person’s own words: text (what was heard), its recording_id, and the turn it stopped (null when none was running). Stop playing the agent’s voice. Once per recording.
heard	A spoken turn’s transcript, before the reply: text, and audio_seconds of audio.
transcript	The answer to user_audio.end with transcribe_only: text, audio_seconds, request_id, and the recording_id user_audio.start named (null when it named none). Not part of any turn.
knowledge	The agent looked something up in its knowledge: via (first, before the reply, or tool, when the model asked), outcome (found, none or failed) and entries, each with entry_id, title and score.
mood	How the agent feels as it replies: mood and intensity (0 to 1, or null), when the agent reports moods. Usually once per turn; a reply that continues after a tool call may report a new one.
thought.delta	A piece of a thinking summary, text, when you asked for thoughts.
text.delta	A piece of the reply as the model writes it, text.
text.chunk	A piece of the reply cut for speech: chunk (its index) and text, the words as spoken. A speaking agent’s audio for it arrives in the audio.start window with the same chunk.
tool.start, tool.end	The model called a tool: name, kind (platform for a knowledge search or recording a change in the relationship, webhook or app for one of your tools) and call_id (your tool’s call, call_…; null for the platform’s own). tool.end adds ok and ms, how long it took. Always a pair.
tool.call	The model called one of your app’s tools, which auth named: call_id, name, arguments, action_only and timeout_ms. Run it and answer with tool.result within timeout_ms, or the call fails. An action_only call is not waited for: the reply goes on at once, and no answer is wanted.
tool.confirm	A tool marked ask_first was called and has not run: call_id, name and arguments. The agent asks the person in its own words; show Yes and No if you like, and answer with tool.confirmed. It also runs if, in the person’s next turn, the agent calls it again with the same arguments. It waits the platform’s agents.tools.confirm_for_minutes (Limits), and one turn.
audio.start	An audio window opens: chunk, codec, sample_rate, channels. The binary frames that follow belong to it.
audio.end	The window closes: chunk, bytes, audio_seconds.
muted	After mute, once the last window has closed.
usage	The model and its token counts for the reply: model, input_tokens (with what a built-in tool read inside the call, which is billed as input), output_tokens, thought_tokens, cached_tokens, generation_seconds. A scripted opening line makes no model call and sends none.
turn.done	The turn finished. See below.
interrupted	The turn ended early: message (what was written before it stopped), mood, mood_intensity, and reason: the client’s cancel, or the failure its error named.
relation	The relationship moved on this turn, which is rare. turn and the relationship’s new fields. Arrives just after turn.done.
compacted	The conversation was compacted: session_id, after_seq (the last message the summary covers) and trigger, automatic or manual. Its next turn starts from the summary and the messages after it; the transcript is unchanged, so there is nothing to fetch again. Sent to every socket on the conversation, after a reply and never inside one, and not part of any turn.
error	Something went wrong: code, message, and turn, stage, chunk, request_id and recording_id, each null when it does not apply. See Errors on the socket.
pong	The answer to ping: exactly {"type":"pong"}, with no seq, so it never counts in the sequence. Nothing else is sent unprompted to keep a connection alive.

How a turn ends

Exactly one of turn.done or interrupted ends every turn, including failing ones. Treat either as “the connection is idle again”.

turn.done carries the whole reply, so a client that ignores every delta is still correct:

turn.done

{
  "type": "turn.done",
  "seq": 42,
  "turn": "…",
  "session_id": "0193b1f4-8e02-7c3a-a1d5-6b7e90c4f218",
  "message": "Start in Alfama. It's quiet before ten, and the light is lovely.",
  "model": "gemini-3.1-flash-lite",
  "input_tokens": 1834,
  "output_tokens": 21,
  "mood": "cheerful",
  "mood_intensity": 0.6,
  "truncated": false,
  "chunks_spoken": 2,
  "heard": null,
  "heard_seconds": null,
  "tool_calls": [],
  "sources": [],
  "timings": { "model_first_text_ms": 640, "first_audio_sent_ms": 910, "total_ms": 1720 }
}

truncated says whether the reply was cut short. heard and heard_seconds repeat a spoken turn’s transcript. tool_calls lists the calls of your tools the reply made, in order: call_id, name, kind, arguments, result (what it answered, as the model read it, cut at the platform’s agents.tools.result_max_bytes, Limits), ok, error, ms and round, as the transcript keeps them. sources lists the web pages it read with the agent’s url_context switch on, each a url and whether it could be retrieved (Built-in tools). timings says where the turn’s time went, in milliseconds from when the message was received (abbreviated above).

The order of a turn

turn.start → heard (a spoken turn) → knowledge, mood, thought.delta, text.delta, text.chunk and audio windows, interleaved → usage → up to the platform’s agents.tools.max_rounds rounds of tool.start / tool.end pairs (with a tool.call between them for your app’s tool, a tool.confirm for a call that asks first, and a tool’s filler line spoken while it runs) and the text that follows them → the remaining audio windows → muted (if you sent mute) → turn.done. On a failure: error, then interrupted.

Audio out

Audio arrives as binary frames: raw PCM, 16-bit signed little-endian, mono, with no header. Each binary frame belongs to the audio.start before it, and only one window is open at a time, so play in wire order. Sentences may finish being generated out of order; they never arrive out of order.

Read sample_rate from audio.start (ready.audio carries the same number at connect). An agent’s voice may be at 8000, 16000, 22050, 24000 or 44100 Hz, and playing one as another is the wrong speed, silently.

Speaking a turn

Send user_audio.start, then the audio as binary frames, then user_audio.end. You decide when the person stopped talking: there is no voice activity detection on the server.

JavaScript

ws.send(JSON.stringify({ type: "user_audio.start", rate: 16000 }));
for (const chunk of pcmChunks) ws.send(chunk); // ArrayBuffer of 16-bit little-endian mono PCM
ws.send(JSON.stringify({ type: "user_audio.end" }));

Send the rate. Raw PCM has no header, so that number is the only thing that says what your samples are, and it is used both for what the transcriber hears and for what you are billed. Without it, the recording is read at the agent’s own default, announced as ready.hearing.sample_rate, which is whatever the agent’s owner configured and not necessarily what you are recording. Fifteen seconds of 24 kHz speech read as 16 kHz is 22.5 seconds of it, at the wrong pitch and half again the price. Accepted rates: 8000, 16000, 22050, 24000 and 44100.

language hints the language (detected per recording when unset) and model picks the transcription model (scribe_v2). Both override the agent’s stt_config for this recording.

One recording is capped at 10 MB. Anything longer is a recording rather than a turn, and belongs on POST /v1/transcriptions. Past the cap, or binary audio with no user_audio.start before it, the socket closes with 4011.

While the person speaks, heard.partial frames carry the transcript so far. After user_audio.end, heard carries the final words, and the turn continues like a typed one.

If a turn is still running when you send user_audio.end, you get an error and the recording stays open: wait for turn.done and send user_audio.end again, or cancel to discard it.

Transcribing without answering

Send user_audio.end with {"transcribe_only": true}:

user_audio.end

{ "type": "user_audio.end", "transcribe_only": true }

The words come back on transcript and go no further: no turn is taken, nothing is stored, and the agent never sees them. Send them yourself as a user_message when you decide the person has finished.

This is what a client with its own voice activity detection wants: three fragments of one sentence become three transcripts you join, rather than three turns that each answer a third of a thought. It may be sent while a turn is running, up to three at a time per connection, and is charged exactly as a spoken turn’s transcription is. A silent recording comes back as an empty text.

Name each recording. The recordings in flight are transcribed at once, so their transcripts arrive in the order they finish, not the order you sent them: join them in arrival order and a sentence can come out scrambled. Give user_audio.start a recording_id, and the transcript for that recording, or the error that failed it, carries it back:

user_audio.start

{ "type": "user_audio.start", "rate": 16000, "recording_id": "take-2" }

transcript

{ "type": "transcript", "seq": 18, "text": "on my first morning", "audio_seconds": 1.4, "request_id": "…", "recording_id": "take-2" }

Interrupting and silencing

cancel stops the turn. What was already generated is still billed and still stored, and the turn ends with interrupted rather than turn.done. You may open a recording straight away: user_audio.start is accepted while a turn is still unwinding, because opening a recording is not taking a turn. A user_message sent while a cancelled turn unwinds is held and taken as soon as it ends.

mute stops the voice and lets the turn finish. Text deltas keep arriving and the turn still ends on turn.done. Use it when somebody has read the reply and does not want to sit through it; cancel would throw away the rest of the answer as well. Sentences not yet spoken are never generated, so this stops the charge as well as the playback. It is answered with muted after the last audio.end, cannot be undone, and applies to that turn only.

Talking over the agent

A hands-free app keeps the microphone open and opens a recording whenever it hears the person, the agent’s voice playing or not. Open those with barge_in: true:

user_audio.start

{ "type": "user_audio.start", "rate": 16000, "barge_in": true }

Sound alone never stops the agent: a cough is loud, and so is whatever of the agent’s own voice the device’s echo cancellation lets through. Lower its volume while you wait. The first words heard that the agent did not just say are the person: barge_in arrives, the running turn is cancelled as cancel would cancel it (it ends with interrupted), and the recording goes on, to become the next turn when you send user_audio.end. Stop playing the agent’s voice on barge_in. It arrives whether or not a turn was still running, since a device plays a reply for a while after the last of it was sent.

When no words come, the person did not speak: close the recording with {"type": "user_audio.end", "discard": true}. Nothing of it is transcribed, the turn goes on, and no error comes back.

Errors on the socket

An error frame does not end the connection, and inside a turn it does not end the turn: the turn still ends with turn.done or interrupted.

error

{
  "type": "error",
  "seq": 7,
  "code": "conflict",
  "message": "A turn is already running on this connection. Wait for turn.done, or send cancel.",
  "turn": null,
  "stage": null,
  "chunk": null,
  "request_id": null,
  "recording_id": null
}

code is the REST API’s, from the same table:

code	Meaning
conflict	A message or a recording’s end while a turn is running, or a reply another connection on the session got in first. Nothing changed.
not_found	A tool.confirmed for a call that is no longer waiting: answered already, expired, or dropped by a later turn.
invalid_request	user_audio.end with nothing recorded.
validation_failed	A rate or model the socket does not accept.
rate_limited	Three transcribe_only recordings already in flight: the recording is still open.
unavailable	With stage: "speech": the reply cannot be spoken — the agent has no voice, or nothing in it could be said. The text turn carries on.
provider_failed	The speech or transcription provider failed: with stage: "speech" (and the chunk it lost), the text turn carries on; for a transcribe_only recording, with its request_id.
insufficient_credits, internal, …	The turn failed, followed by interrupted.

An error about a recording carries its recording_id; one inside a turn, its turn.

Close codes

Codes are in the 4000 range, and each one is a different next action. The close frame’s reason is the specific cause when there is one, and otherwise the text below.

Code	Reason	Final
4001	No credentials were sent. Send auth as the first frame, within 10 seconds.	no
4002	Invalid, revoked or expired credentials, or an API key instead of a token.	yes
4003	This token has expired. Mint another and reconnect.	no
4004	This token was revoked, or has spent its credit limit.	yes
4005	Your organisation has no credits left.	no
4006	Too many turns, or too many connections on one token. Slow down and reconnect.	no
4007	Closed after a period with no activity (idle_seconds, 300). ping does not count as activity.	no
4008	That conversation is not available: unknown, deleted, or its agent or person was deleted.	yes
4009	The frame could not be understood, or the session in the URL and the auth frame differ.	no
4010	This capability is not available, for example the agent’s voice cannot be used. Fix it in the console.	yes
4011	Unaccountable audio: too much for one turn, or no user_audio.start before it.	no
4013	The connection failed on our side. Reconnect.	no
4014	This end user already has 3 open connections.	no

4002, 4004, 4008 and 4010 are final: reconnecting with the same credential fails the same way. Everything else is worth retrying, with a new token where the token was the problem (4003).

Limits

Limit	Value
Time to send auth	10 seconds
Idle before close	300 seconds without activity
ping, from you	after 30 seconds with nothing from the server; a minute with no pong is a dead connection: close it and reconnect
user_message text	1 to 32,000 characters
One recording	10 MB
Recording rates	8000, 16000, 22050, 24000, 44100 Hz
A recording’s recording_id	1 to 64 characters
transcribe_only recordings in flight	3 per connection
Open connections per end user	3
Turns at a time	1 per connection
App tools named on auth	50
A tool.result’s error	1 to 2,000 characters

Your plan may also limit how many turns a token takes a minute and an hour, and how many connections one token holds; past either, the socket closes with 4006.

Scopes and billing

A token holds one scope, agent, and it covers everything on this socket, including speaking to the agent and hearing it speak back, even though the token cannot call /v1/tts or /v1/stt directly. transcribe_only grants nothing new: the words of a spoken turn already come back on heard.

Every turn is charged to the token: the model’s tokens for the reply, and for a spoken turn its transcription, and for a speaking agent its speech, on separate usage rows. When the token has spent its limit, the reply in progress still finishes, and the socket then closes with 4004. See What a turn costs.

A minimal browser client

This client connects, waits for the greeting, sends one message and plays the reply. Create the AudioContext from a click: browsers start audio only after a user gesture.

JavaScript

function connect({ token, sessionId, onText }) {
  const ws = new WebSocket(`wss://api.lumavox.ai/v1/agent/stream?session_id=${sessionId}`, "ino.v2");
  ws.binaryType = "arraybuffer";

  const audio = new AudioContext();
  let rate = 24000;
  let playAt = 0;
  let idle = false;

  function play(buffer) {
    const samples = new Int16Array(buffer, 0, buffer.byteLength >> 1);
    const clip = audio.createBuffer(1, samples.length, rate);
    const channel = clip.getChannelData(0);
    for (let i = 0; i < samples.length; i++) channel[i] = samples[i] / 32768;
    const source = audio.createBufferSource();
    source.buffer = clip;
    source.connect(audio.destination);
    playAt = Math.max(playAt, audio.currentTime);
    source.start(playAt);
    playAt += clip.duration;
  }

  ws.onopen = () =>
    ws.send(JSON.stringify({ type: "auth", key: token, session_id: sessionId }));

  ws.onmessage = ({ data }) => {
    if (typeof data !== "string") return play(data);
    const frame = JSON.parse(data);
    switch (frame.type) {
      case "ready":
        rate = frame.audio.sample_rate;
        idle = !frame.greeting;
        break;
      case "audio.start":
        rate = frame.sample_rate;
        break;
      case "text.delta":
        onText(frame.text);
        break;
      case "turn.done":
      case "interrupted":
        idle = true;
        break;
      case "error":
        console.warn(frame.code, frame.message);
        break;
    }
  };

  ws.onclose = ({ code, reason }) => console.log("closed", code, reason);

  return {
    say(text) {
      if (!idle) return false;
      idle = false;
      ws.send(JSON.stringify({ type: "user_message", text }));
      return true;
    },
    cancel: () => ws.send(JSON.stringify({ type: "cancel" })),
    mute: () => ws.send(JSON.stringify({ type: "mute" })),
  };
}