# Client SDK

> Talk to agents from a browser, a React Native app or Node with an end user's key — a conversation that keeps itself connected, and their memory.

`@lumavox/client` is the agent's side of the API for your users' devices: a conversation on [the WebSocket](https://docs.lumavox.ai/agents/websocket.md) that keeps itself connected, and the REST routes for the person's conversations, what is remembered about them and how your agents feel about them. It holds an [ephemeral key](https://docs.lumavox.ai/agents/end-users.md) for one person, **never your API key**. It has **no dependencies**, and runs in browsers, in React Native (Hermes) and in Node 22 or later. On React, [the hooks](https://docs.lumavox.ai/sdks/react.md) sit on top of it.

```bash title="npm"
npm install @lumavox/client
```

Make a client that gets its key from your server, open a conversation, and talk:

```ts title="TypeScript"
import { createClient } from "@lumavox/client";

const agents = createClient({
  baseUrl: "https://api.lumavox.ai",
  // Your server mints the key with your API key: POST /v1/ephemeral-keys.
  getToken: async () => (await fetch("/api/agent-key", { method: "POST" })).text(),
});

const session = await agents.sessions.create({ agent_id: "agt_..." });
const conversation = agents.connect(session.id, {
  onText: ({ text }) => render(text),
});

const outcome = await conversation.send("What should I see on my first morning?");
```

## The key

```ts title="TypeScript"
createClient({
  getToken,                 // ({ reason, status?, code? }) => the key, or a promise of it
  baseUrl: "https://api.lumavox.ai",
  socketUrl,                // "wss://api.lumavox.ai/v1/agent/stream" by default
  timeoutMs: 30_000,        // per attempt
  maxRetries: 2,
  fetch,                    // the runtime's own by default
  WebSocket,                // the runtime's own by default
});
```

`getToken` is your way to a key — usually a call to your own server, which mints one with `@lumavox/server`'s `ephemeralKeys.create`. The SDK asks once and keeps the key. It asks again only when the platform refuses it — a `401`, or the socket closed with `4002` to `4005` — with `reason: "refused"` and the status or code, so your policy can decide whether to mint, and **once**: calls refused together share one new key, since every mint sets credit aside, and a refusal of the fresh key is final. It never asks for a `402`, `403` or `429`. Throw from `getToken` to stop. Call `clearToken()` at sign-out.

## Sessions, memory and relations

| Resource | Methods |
|---|---|
| `sessions` | `create`, `list` (every one), `page`, `get`, `history` (the whole transcript, newest first), `delete`, `compact`, `extract` |
| `facts` | `list`, `page` (`{ include_replaced: true }` after the options lists the replaced ones too), `create` (tell the agents something; `created` says whether it was new), `update` (correct one's words), `delete`, `deleteAll` |
| `relations` | `list`, `page` |

Bodies and answers are the API's own JSON, as the [reference](https://docs.lumavox.ai/reference/agent.md) shows them; the SDK's options go last on every call: `{ timeoutMs, signal, maxRetries, requestId }`. **`list()` is every item**, an async iterator reading page after page by each page's `next_cursor`; **`page()` is one page**, `{ data, next_cursor }`, as the route answers it — pass `next_cursor` back as `cursor` for the next. A session's transcript is a page inside it: `get` answers the newest messages, oldest first, and `messages.next_cursor` reads further back. A delete answers `true`, and `false` for what is already gone. The key names the person, so no call takes an `end_user_id`. Errors are `@lumavox/server`'s: a `PlatformError` with `status`, `code`, `requestId`, `issues` and `retryAfter`, and a class per status — `AuthenticationError`, `PermissionError`, `NotFoundError`, `RateLimitError` and the rest.

```ts title="TypeScript"
for await (const session of agents.sessions.list()) {
  console.log(session.id, session.title);
}

const latest = await agents.sessions.get(session.id, { limit: 50 });
if (latest.messages.next_cursor) {
  const older = await agents.sessions.get(session.id, { cursor: latest.messages.next_cursor });
  console.log(older.messages.data.length);
}
```

## A conversation

`connect(sessionId, handlers, options)` answers at once. It asks for the key, dials, authenticates in the first frame, and from then on keeps the conversation going:

- **What you send is queued** behind the agent's greeting, a running turn, a reconnect or a quiet spell, and goes when it can. `send` answers how the turn ended — `done`, `interrupted`, or `dropped` with a `reason` (`closed`, `cancelled`, `refused`) and the platform's `message` for a refusal — and **never rejects**. A message the platform never acknowledged is sent again once after a reconnect; one it took is never sent twice.
- **It reconnects by the close code**: a new key once for `4002` to `4005`; 5 seconds, doubling, after a rate or connection limit (`4006`, `4014`); half a second, doubling, after anything else, each delay varied by up to a fifth so a fleet of devices does not return at once; final after five in a row, or after `4008` and `4010`. After `4007` — closed for quiet — it rests, and dials again on the next `send`, `startAudio` or `reconnect()`.
- **It notices a dead connection**: after 30 seconds with nothing from the platform it sends `ping`, and a minute with nothing at all — no `pong` — is a socket that is gone, a phone that changed networks: it dials again. You never need to `ping` yourself.
- **`snapshot`** is what it knows now — `state`, `ready`, the `turn` (its `text`, `chunks`, `mood`, `tool`, `knowledge`, `heard`, `done`, `interrupted`), `caption`, `relation` and `queued` — a new object on every change, with `subscribe(fn)`.

| Handler | Given |
|---|---|
| `onReady`, `onTurnStart`, `onHeard`, `onHeardPartial`, `onKnowledge`, `onMood`, `onThought`, `onText`, `onChunk`, `onToolStart`, `onToolEnd`, `onToolConfirm`, `onUsage`, `onMuted`, `onAudioStart`, `onAudioEnd`, `onTranscript`, `onCompacted`, `onDone`, `onInterrupted`, `onError` | the frame, as the [WebSocket](https://docs.lumavox.ai/agents/websocket.md#what-arrives) page shows it |
| `onAudio` | `{ pcm, chunk, turn, sample_rate, channels }`: a reply's PCM with its window's format |
| `onRelation` | the relationship, from `ready` and each time it moves |
| `onState` | `connecting`, `ready`, `busy`, `reconnecting`, `dormant`, `closed` |
| `onDisconnect` | every socket that closed, recovered from or not: `{ code, reason, ready }` |
| `onClose` | only when it will not reconnect: `{ code, reason, final }` |
| `onWarning` | a gap in the frames, a frame it could not read, audio past 10 MB |
| `onEvent` | every frame, including any this version does not know |

**Speech you cannot play is not paid for**: with no `onAudio` handler, every turn asks for no speech. `connect`'s `{ speech: true }` asks for it anyway, and `{ thoughts: true }` for thinking summaries. `cancel()` stops the turn and drops what is queued; `mute()` stops the voice and lets the reply finish; `reconnect()` dials now — when your app comes back to the foreground, say; `close()` ends it, and every pending promise answers `dropped`.

**A tool marked "ask first"** sends `tool.confirm` (`call_id`, `name`, `arguments`) and does not run: the agent asks the person in its own words. `confirmTool(callId, yes)` answers with their tap — a turn of theirs, queued as `send` is, which runs the tool first on a yes. See [Tools](https://docs.lumavox.ai/agents/tools.md).

### Your app's tools

An agent's [app tools](https://docs.lumavox.ai/agents/tools.md#app-tools) run here: give `connect` a handler for each, by the tool's name. The conversation names them when it authenticates, so the agent is offered those and no others; runs the handler when the model calls one, with the model's arguments; and sends back what it answers — any JSON, or nothing — or, when it throws, the error's message, which the model is told.

```ts title="TypeScript"
const conversation = agents.connect(session.id, {}, {
  tools: {
    open_page: async ({ page }) => {
      render(`Opening ${String(page)}…`);
      return { opened: page };
    },
  },
});
```

A handler has the tool's `timeout_ms` to answer, 10 seconds by default. An action-only tool's handler runs and nothing is sent back: the reply goes on without waiting. A call whose connection dropped before its handler answered is over, and its answer is not sent on the next one. `onEvent` sees each `tool.call` as it comes.

## Speaking

```ts title="TypeScript"
conversation.startAudio({ rate: 16_000 });           // one of RECORDING_RATES
conversation.sendAudio(pcm);                          // Int16Array, or an ArrayBuffer of 16-bit little-endian PCM
const turn = await conversation.endAudio();          // a spoken turn, answered as send's
const words = await conversation.endAudio({ transcribeOnly: true });
```

**The rate is required**: raw PCM has no header, so it is the only thing that says what your samples are. A recording past 10 MB is refused here, rather than closing the socket. **Each recording carries its own id**, which the platform echoes, so `endAudio({ transcribeOnly: true })` answers that recording's words — `{ status: "done", text, audioSeconds }`, or `failed` with the error frame, which names it as `recording_id` — however the transcripts of several finish.

`createClient` captures and plays nothing itself, so it adds no audio library to your app: your recorder hands it samples, your player takes `onAudio`'s. `floatToPcm16`, `pcm16ToFloat`, `resample` (a 48 kHz microphone is not an accepted rate), `concatPcm` and `pcmSeconds` convert between the two and the wire. `AgentAudio` is the shape of a player and recorder together, which [the React hooks](https://docs.lumavox.ai/sdks/react.md) drive.

## Audio in a browser

A website gets a player and a recorder on Web Audio from its own import, so `createClient` carries none of it (about 8.5 KB minified):

```ts title="TypeScript"
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio();
startButton.addEventListener("click", () => {
  void audio.prime(); // in the click, before any await
});

const conversation = agents.connect(session.id, {
  onAudio: (frame) => audio.play(frame.pcm, frame),
});
```

**Call `prime()` from the click that starts the conversation**: browsers start sound only from a person's gesture, and a player made outside one stays silent while it looks like it is playing. Replies play gaplessly as they arrive, each frame at its own rate, on an audio context at the device's own rate; `stop()` silences them at once — on `cancel()`, say. `startCapture(onPcm)` opens the microphone with the browser's echo cancellation, noise suppression and gain control, and records at 24 kHz whatever you ask, answering `{ rate }` to open the recording at; `stopCapture()` closes it, the last samples delivered before it settles. With [the React hooks](https://docs.lumavox.ai/sdks/react.md#audio), hand it to `useAgent` instead: it plays, stops and records through it for you.

### The clock: a face, subtitles

To draw against the voice — a face lip-synced, subtitles revealed chunk by chunk — read the player's clock. `onScheduled` tells every frame as it is queued, before it plays: its 16-bit samples (`pcm`, `sample_rate`, `channels`), its `chunk` and `turn` as the socket sent them, and `startsAt` and `duration`, in seconds on the clock `now()` reads. **Draw against `heardTime()`**, what the listener hears at this moment: the clock less the device's output latency, which Bluetooth makes longer.

```ts title="TypeScript"
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio({ leadInMs: 350 }); // time to read each frame before it plays

audio.onScheduled(({ pcm, sample_rate, chunk, startsAt }) => {
  lipsync.add(pcm, sample_rate, startsAt); // your model, ahead of the sound
  subtitles.reveal(chunk, startsAt); // the text.chunk of the same number
});
audio.onStopped(() => lipsync.clear()); // a cancel, a mute, a barge-in: what was queued never plays

function draw() {
  face.show(lipsync.at(audio.heardTime()));
  requestAnimationFrame(draw);
}
requestAnimationFrame(draw);
```

- **A reply's first frame waits `leadInMs` before it plays** — 60 ms unless set, and again after any gap in the voice; frames after it follow back to back. It is the time a model has to read each frame first, and **it adds as much to the first word heard**.
- **`onStopped`** fires when `stop()` drops audio still to play: a cancel, a mute, a barge-in, a new turn. `onDrained` fires then too, and when a reply plays out.
- **`onCapture((pcm, rate) => …)`** is every frame the microphone captures while it is open — push-to-talk's, and hands-free's between utterances, which are never sent — for a level meter or an orb.
- `heardTime()` takes the browser's `outputLatency` and `baseLatency` off the clock, read each time; Safari says neither, so there it is the clock. `now()` is `0` before `prime()`.
- Each `on…` answers its unsubscribe, and **a handler that throws is rethrown on its own**: it never stops the voice or the recording.

### The microphone and the speaker

```ts title="TypeScript"
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio({
  input: savedMicrophone, // an id from inputs(); the default when it is gone
  processing: { noiseSuppression: false }, // each on unless false
});

const microphones = await audio.inputs(); // [{ id, label }]
audio.setInput(microphones[1]?.id); // from the next recording

if (audio.canChooseOutput) {
  const [speaker] = await audio.outputs();
  if (speaker) await audio.setOutputDevice(speaker.id); // true when it took
}
```

- **The microphone** is the system's default unless chosen. `setInput(id)` and `setProcessing({ … })` take effect from the next recording; one unplugged since it was chosen records from the default rather than failing the press.
- **Ids and labels are empty until the person has allowed the microphone**: list them after the first recording.
- **The browser's echo cancellation, noise suppression and gain control** are each on unless set `false`. **With echo cancellation off, hands-free is refused**: the agent would hear itself.
- **The speaker** can be chosen in Chrome and Edge only: elsewhere `canChooseOutput` is `false`, `outputs()` is `[]` and `setOutputDevice` answers `false` — hide the choice. A speaker that took is kept for a player made later, after `dispose()` and `prime()` again.

## Hands-free

`listen` leaves the microphone open, and every utterance becomes a turn: a detector hears when the person talks, the 300 ms before it are sent first so no syllable is lost, and **700 ms of silence ends it — a wait added to every reply** (`endAfterMs` sets it). Nothing is sent between utterances, so silence is never billed.

```ts title="TypeScript"
import { listen } from "@lumavox/client";
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio();
startButton.addEventListener("click", async () => {
  void audio.prime(); // in the click, before any await
  const outcome = await listen(conversation, audio, {
    onState: (state) => render(state), // listening, hearing, paused, stopped
  });
  if (!outcome.ok) render(outcome.reason); // push-to-talk still works
});
```

**Talking over the agent** lowers its voice at once and opens the recording; the **first words that are not the agent's own** stop it (`onBargeIn`) — about two seconds in, as long as the transcriber takes to say them — and what the person says becomes the next turn. Sound alone never stops it: talking that brings no words within 3 seconds (`wordsWithinMs`) — a cough, the agent's own voice leaking back — is thrown away, the voice comes back up, and nothing new opens until the room is quiet again. While the agent speaks, the built-in detector asks for more before it hears talking.

**It refuses where the agent would hear itself**: `listen` answers `{ ok: false, reason }` when the audio cannot keep the agent's voice out of the microphone. `createWebAudio()` allows it where the browser cancels echo — Chrome, Edge and Firefox — and refuses it on an iPhone or iPad, whose browsers' echo cancellation is broken. There, offer half-duplex. With headphones on, or an adapter of your own that cannot say, pass `assumeNoEcho: true`. The built-in detector is loudness (`loudness()`), never a model; pass your own as `detector` — anything with `push(frame, rate, speaking)` answering `"talking"` or `"silent"` for each 20 ms frame.

### Half-duplex

**Tap once and talk for the whole conversation, where echo cannot be cancelled**: with `halfDuplex: true` the microphone stays open, but **nothing is sent while the agent's voice is heard** and talking never stops it — so the agent never hears itself, on any browser and any phone, and `listen` never refuses for echo.

```ts title="TypeScript"
import { listen } from "@lumavox/client";
import { createWebAudio } from "@lumavox/client/web-audio";

const audio = createWebAudio();
startButton.addEventListener("click", () => {
  void audio.prime();
  void listen(conversation, audio, { halfDuplex: true }); // never refused for echo
});
```

- **Heard means heard**: the player's clock (`heardTime()`), not when the audio arrived — a reply is queued seconds ahead of its sound. An adapter of your own with no clock is taken at its word, from a reply's first audio to `onDrained`.
- **After the voice, 300 ms more** (`afterVoiceMs`) before the microphone listens again, so the room's echo of the last word does not open a recording. It adds nothing to a reply: words begun in it keep their start. **A word over before it ends — a quick "yes" under 0.3 s — is not heard**; with headphones on, set it lower.
- **The agent's voice has the floor**: something said when its voice starts is cut there and sent as it is, answered after the reply.
- **No barge-in**: to cut the agent short, give the person a stop button (`conversation.cancel()`).
- The rest is hands-free's: the detector, the pre-roll, 700 ms of silence to end an utterance, and nothing sent between utterances.

## Runtimes

- **Browsers**: the native `WebSocket` and `fetch`.
- **React Native**: binary frames are asked for as `ArrayBuffer`s, text frames read as strings, query strings built by hand. Each binary frame still reaches JavaScript as base64, a cost per audio frame no SDK can remove — measure it on a low-end Android. The audio for it, with the hooks, is [`@lumavox/react-native`](https://docs.lumavox.ai/sdks/react-native.md).
- **Node 22 or later**: the global `WebSocket` and `fetch`; on an older Node, pass `WebSocket` from the `ws` package.