SDKs

Client SDK

Talk to agents from a browser, a React Native app or Node with an end user's key — a conversation that keeps itself connected, and their memory.

@lumavox/client is the agent’s side of the API for your users’ devices: a conversation on the WebSocket that keeps itself connected, and the REST routes for the person’s conversations, what is remembered about them and how your agents feel about them. It holds an ephemeral key for one person, never your API key. It has no dependencies, and runs in browsers, in React Native (Hermes) and in Node 22 or later. On React, the hooks sit on top of it.

npm install @lumavox/client

Make a client that gets its key from your server, open a conversation, and talk:

import { createClient } from "@lumavox/client";

const agents = createClient({
  baseUrl: "https://api.lumavox.ai",
  // Your server mints the key with your API key: POST /v1/ephemeral-keys.
  getToken: async () => (await fetch("/api/agent-key", { method: "POST" })).text(),
});

const session = await agents.sessions.create({ agent_id: "agt_..." });
const conversation = agents.connect(session.id, {
  onText: ({ text }) => render(text),
});

const outcome = await conversation.send("What should I see on my first morning?");

The key

createClient({
  getToken,                 // ({ reason, status?, code? }) => the key, or a promise of it
  baseUrl: "https://api.lumavox.ai",
  socketUrl,                // "wss://api.lumavox.ai/v1/agent/stream" by default
  timeoutMs: 30_000,        // per attempt
  maxRetries: 2,
  fetch,                    // the runtime's own by default
  WebSocket,                // the runtime's own by default
});

getToken is your way to a key — usually a call to your own server, which mints one with @lumavox/server’s ephemeralKeys.create. The SDK asks once and keeps the key. It asks again only when the platform refuses it — a 401, or the socket closed with 4002 to 4005 — with reason: "refused" and the status or code, so your policy can decide whether to mint, and once: calls refused together share one new key, since every mint sets credit aside, and a refusal of the fresh key is final. It never asks for a 402, 403 or 429. Throw from getToken to stop. Call clearToken() at sign-out.

Sessions, memory and relations

Resource	Methods
`sessions`	`create`, `list` (every one), `page`, `get`, `history` (the whole transcript, newest first), `delete`, `compact`, `extract`
`facts`	`list`, `page` (`{ include_replaced: true }` after the options lists the replaced ones too), `create` (tell the agents something; `created` says whether it was new), `update` (correct one’s words), `delete`, `deleteAll`
`relations`	`list`, `page`

Bodies and answers are the API’s own JSON, as the reference shows them; the SDK’s options go last on every call: { timeoutMs, signal, maxRetries, requestId }. list() is every item, an async iterator reading page after page by each page’s next_cursor; page() is one page, { data, next_cursor }, as the route answers it — pass next_cursor back as cursor for the next. A session’s transcript is a page inside it: get answers the newest messages, oldest first, and messages.next_cursor reads further back. A delete answers true, and false for what is already gone. The key names the person, so no call takes an end_user_id. Errors are @lumavox/server’s: a PlatformError with status, code, requestId, issues and retryAfter, and a class per status — AuthenticationError, PermissionError, NotFoundError, RateLimitError and the rest.

for await (const session of agents.sessions.list()) {
  console.log(session.id, session.title);
}

const latest = await agents.sessions.get(session.id, { limit: 50 });
if (latest.messages.next_cursor) {
  const older = await agents.sessions.get(session.id, { cursor: latest.messages.next_cursor });
  console.log(older.messages.data.length);
}

A conversation

connect(sessionId, handlers, options) answers at once. It asks for the key, dials, authenticates in the first frame, and from then on keeps the conversation going:

What you send is queued behind the agent’s greeting, a running turn, a reconnect or a quiet spell, and goes when it can. send answers how the turn ended — done, interrupted, or dropped with a reason (closed, cancelled, refused) and the platform’s message for a refusal — and never rejects. A message the platform never acknowledged is sent again once after a reconnect; one it took is never sent twice.
It reconnects by the close code: a new key once for 4002 to 4005; 5 seconds, doubling, after a rate or connection limit (4006, 4014); half a second, doubling, after anything else, each delay varied by up to a fifth so a fleet of devices does not return at once; final after five in a row, or after 4008 and 4010. After 4007 — closed for quiet — it rests, and dials again on the next send, startAudio or reconnect().
It notices a dead connection: after 30 seconds with nothing from the platform it sends ping, and a minute with nothing at all — no pong — is a socket that is gone, a phone that changed networks: it dials again. You never need to ping yourself.
snapshot is what it knows now — state, ready, the turn (its text, chunks, mood, tool, knowledge, heard, done, interrupted), caption, relation and queued — a new object on every change, with subscribe(fn).

Handler	Given
`onReady`, `onTurnStart`, `onHeard`, `onHeardPartial`, `onKnowledge`, `onMood`, `onThought`, `onText`, `onChunk`, `onToolStart`, `onToolEnd`, `onToolConfirm`, `onUsage`, `onMuted`, `onAudioStart`, `onAudioEnd`, `onTranscript`, `onCompacted`, `onDone`, `onInterrupted`, `onError`	the frame, as the WebSocket page shows it
`onAudio`	`{ pcm, chunk, turn, sample_rate, channels }`: a reply’s PCM with its window’s format
`onRelation`	the relationship, from `ready` and each time it moves
`onState`	`connecting`, `ready`, `busy`, `reconnecting`, `dormant`, `closed`
`onDisconnect`	every socket that closed, recovered from or not: `{ code, reason, ready }`
`onClose`	only when it will not reconnect: `{ code, reason, final }`
`onWarning`	a gap in the frames, a frame it could not read, audio past 10 MB
`onEvent`	every frame, including any this version does not know

Speech you cannot play is not paid for: with no onAudio handler, every turn asks for no speech. connect’s { speech: true } asks for it anyway, and { thoughts: true } for thinking summaries. cancel() stops the turn and drops what is queued; mute() stops the voice and lets the reply finish; reconnect() dials now — when your app comes back to the foreground, say; close() ends it, and every pending promise answers dropped.

A tool marked “ask first” sends tool.confirm (call_id, name, arguments) and does not run: the agent asks the person in its own words. confirmTool(callId, yes) answers with their tap — a turn of theirs, queued as send is, which runs the tool first on a yes. See Tools.

Your app’s tools

An agent’s app tools run here: give connect a handler for each, by the tool’s name. The conversation names them when it authenticates, so the agent is offered those and no others; runs the handler when the model calls one, with the model’s arguments; and sends back what it answers — any JSON, or nothing — or, when it throws, the error’s message, which the model is told.

const conversation = agents.connect(session.id, {}, {
  tools: {
    open_page: async ({ page }) => {
      render(`Opening ${String(page)}…`);
      return { opened: page };
    },
  },
});

A handler has the tool’s timeout_ms to answer, 10 seconds by default. An action-only tool’s handler runs and nothing is sent back: the reply goes on without waiting. A call whose connection dropped before its handler answered is over, and its answer is not sent on the next one. onEvent sees each tool.call as it comes.

Speaking

conversation.startAudio({ rate: 16_000 });           // one of RECORDING_RATES
conversation.sendAudio(pcm);                          // Int16Array, or an ArrayBuffer of 16-bit little-endian PCM
const turn = await conversation.endAudio();          // a spoken turn, answered as send's
const words = await conversation.endAudio({ transcribeOnly: true });

The rate is required: raw PCM has no header, so it is the only thing that says what your samples are. A recording past 10 MB is refused here, rather than closing the socket. Each recording carries its own id, which the platform echoes, so endAudio({ transcribeOnly: true }) answers that recording’s words — { status: "done", text, audioSeconds }, or failed with the error frame, which names it as recording_id — however the transcripts of several finish.

createClient captures and plays nothing itself, so it adds no audio library to your app: your recorder hands it samples, your player takes onAudio’s. floatToPcm16, pcm16ToFloat, resample (a 48 kHz microphone is not an accepted rate), concatPcm and pcmSeconds convert between the two and the wire. AgentAudio is the shape of a player and recorder together, which the React hooks drive.

Audio in a browser

A website gets a player and a recorder on Web Audio from its own import, so createClient carries none of it (about 8.5 KB minified):

import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio();
startButton.addEventListener("click", () => {
  void audio.prime(); // in the click, before any await
});

const conversation = agents.connect(session.id, {
  onAudio: (frame) => audio.play(frame.pcm, frame),
});

Call prime() from the click that starts the conversation: browsers start sound only from a person’s gesture, and a player made outside one stays silent while it looks like it is playing. Replies play gaplessly as they arrive, each frame at its own rate, on an audio context at the device’s own rate; stop() silences them at once — on cancel(), say. startCapture(onPcm) opens the microphone with the browser’s echo cancellation, noise suppression and gain control, and records at 24 kHz whatever you ask, answering { rate } to open the recording at; stopCapture() closes it, the last samples delivered before it settles. With the React hooks, hand it to useAgent instead: it plays, stops and records through it for you.

The clock: a face, subtitles

To draw against the voice — a face lip-synced, subtitles revealed chunk by chunk — read the player’s clock. onScheduled tells every frame as it is queued, before it plays: its 16-bit samples (pcm, sample_rate, channels), its chunk and turn as the socket sent them, and startsAt and duration, in seconds on the clock now() reads. Draw against heardTime(), what the listener hears at this moment: the clock less the device’s output latency, which Bluetooth makes longer.

import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio({ leadInMs: 350 }); // time to read each frame before it plays

audio.onScheduled(({ pcm, sample_rate, chunk, startsAt }) => {
  lipsync.add(pcm, sample_rate, startsAt); // your model, ahead of the sound
  subtitles.reveal(chunk, startsAt); // the text.chunk of the same number
});
audio.onStopped(() => lipsync.clear()); // a cancel, a mute, a barge-in: what was queued never plays

function draw() {
  face.show(lipsync.at(audio.heardTime()));
  requestAnimationFrame(draw);
}
requestAnimationFrame(draw);

A reply’s first frame waits leadInMs before it plays — 60 ms unless set, and again after any gap in the voice; frames after it follow back to back. It is the time a model has to read each frame first, and it adds as much to the first word heard.
onStopped fires when stop() drops audio still to play: a cancel, a mute, a barge-in, a new turn. onDrained fires then too, and when a reply plays out.
onCapture((pcm, rate) => …) is every frame the microphone captures while it is open — push-to-talk’s, and hands-free’s between utterances, which are never sent — for a level meter or an orb.
heardTime() takes the browser’s outputLatency and baseLatency off the clock, read each time; Safari says neither, so there it is the clock. now() is 0 before prime().
Each on… answers its unsubscribe, and a handler that throws is rethrown on its own: it never stops the voice or the recording.

The microphone and the speaker

import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio({
  input: savedMicrophone, // an id from inputs(); the default when it is gone
  processing: { noiseSuppression: false }, // each on unless false
});

const microphones = await audio.inputs(); // [{ id, label }]
audio.setInput(microphones[1]?.id); // from the next recording

if (audio.canChooseOutput) {
  const [speaker] = await audio.outputs();
  if (speaker) await audio.setOutputDevice(speaker.id); // true when it took
}

The microphone is the system’s default unless chosen. setInput(id) and setProcessing({ … }) take effect from the next recording; one unplugged since it was chosen records from the default rather than failing the press.
Ids and labels are empty until the person has allowed the microphone: list them after the first recording.
The browser’s echo cancellation, noise suppression and gain control are each on unless set false. With echo cancellation off, hands-free is refused: the agent would hear itself.
The speaker can be chosen in Chrome and Edge only: elsewhere canChooseOutput is false, outputs() is [] and setOutputDevice answers false — hide the choice. A speaker that took is kept for a player made later, after dispose() and prime() again.

Hands-free

listen leaves the microphone open, and every utterance becomes a turn: a detector hears when the person talks, the 300 ms before it are sent first so no syllable is lost, and 700 ms of silence ends it — a wait added to every reply (endAfterMs sets it). Nothing is sent between utterances, so silence is never billed.

import { listen } from "@lumavox/client";
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio();
startButton.addEventListener("click", async () => {
  void audio.prime(); // in the click, before any await
  const outcome = await listen(conversation, audio, {
    onState: (state) => render(state), // listening, hearing, paused, stopped
  });
  if (!outcome.ok) render(outcome.reason); // push-to-talk still works
});

Talking over the agent lowers its voice at once and opens the recording; the first words that are not the agent’s own stop it (onBargeIn) — about two seconds in, as long as the transcriber takes to say them — and what the person says becomes the next turn. Sound alone never stops it: talking that brings no words within 3 seconds (wordsWithinMs) — a cough, the agent’s own voice leaking back — is thrown away, the voice comes back up, and nothing new opens until the room is quiet again. While the agent speaks, the built-in detector asks for more before it hears talking.

It refuses where the agent would hear itself: listen answers { ok: false, reason } when the audio cannot keep the agent’s voice out of the microphone. createWebAudio() allows it where the browser cancels echo — Chrome, Edge and Firefox — and refuses it on an iPhone or iPad, whose browsers’ echo cancellation is broken. There, offer half-duplex. With headphones on, or an adapter of your own that cannot say, pass assumeNoEcho: true. The built-in detector is loudness (loudness()), never a model; pass your own as detector — anything with push(frame, rate, speaking) answering "talking" or "silent" for each 20 ms frame.

Half-duplex

Tap once and talk for the whole conversation, where echo cannot be cancelled: with halfDuplex: true the microphone stays open, but nothing is sent while the agent’s voice is heard and talking never stops it — so the agent never hears itself, on any browser and any phone, and listen never refuses for echo.

import { listen } from "@lumavox/client";
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio();
startButton.addEventListener("click", () => {
  void audio.prime();
  void listen(conversation, audio, { halfDuplex: true }); // never refused for echo
});

Heard means heard: the player’s clock (heardTime()), not when the audio arrived — a reply is queued seconds ahead of its sound. An adapter of your own with no clock is taken at its word, from a reply’s first audio to onDrained.
After the voice, 300 ms more (afterVoiceMs) before the microphone listens again, so the room’s echo of the last word does not open a recording. It adds nothing to a reply: words begun in it keep their start. A word over before it ends — a quick “yes” under 0.3 s — is not heard; with headphones on, set it lower.
The agent’s voice has the floor: something said when its voice starts is cut there and sent as it is, answered after the reply.
No barge-in: to cut the agent short, give the person a stop button (conversation.cancel()).
The rest is hands-free’s: the detector, the pre-roll, 700 ms of silence to end an utterance, and nothing sent between utterances.

Runtimes

Browsers: the native WebSocket and fetch.
React Native: binary frames are asked for as ArrayBuffers, text frames read as strings, query strings built by hand. Each binary frame still reaches JavaScript as base64, a cost per audio frame no SDK can remove — measure it on a low-end Android. The audio for it, with the hooks, is @lumavox/react-native.
Node 22 or later: the global WebSocket and fetch; on an older Node, pass WebSocket from the ws package.