# React Native

> Voice conversations with agents in a React Native app on iOS, Android and the web — the client, the hooks, and push-to-talk, half-duplex and hands-free audio.

`@lumavox/react-native` is one install for a React Native app, on iOS, Android and the web: [the client](https://docs.lumavox.ai/sdks/client.md), [the hooks](https://docs.lumavox.ai/sdks/react.md), and the audio they lack — a player for the agent's voice and a recorder for the person's, for **push-to-talk**, **half-duplex** — the microphone left open for the whole conversation — and **hands-free**, talking over the agent to stop it (on a phone in early access). On a phone the audio runs on [`react-native-audio-api`](https://github.com/software-mansion/react-native-audio-api), installed beside it because React Native links only the native code your app lists itself; in a browser, on Web Audio.

```bash title="npm"
npm install @lumavox/react-native react-native-audio-api
```

- **React Native 0.76 or later, on the New Architecture**, with `react-native-audio-api` 0.13 (0.13.6 or later) — or, for hands-free on a phone, its early-access build ([below](#hands-free-on-a-phone-early-access)).
- **iOS 17 and Android 12 (API 31) or later**: supported and tested; older phones are not.
- **Expo needs a development build** (`npx expo run:ios`, `npx expo run:android`), never Expo Go: the audio is native code.

## Expo

List the plugin in `app.json` and build again:

```json title="app.json"
{
  "expo": {
    "plugins": [["@lumavox/react-native", { "microphonePermission": "Hold to talk to your coach." }]]
  }
}
```

**Never list `react-native-audio-api`'s own plugin as well.** This one runs it with every option set: background audio off (it is on by default there), and Android's `RECORD_AUDIO` and `MODIFY_AUDIO_SETTINGS` permissions listed (its own list lacks the first). That plugin runs once, so a second listing's options would be dropped without a word; listed before this one, the build stops and says so.

| Option | |
|---|---|
| `microphonePermission` | what iOS shows when the app first asks for the microphone — your app's words |
| `androidPermissions` | more Android permissions, added to the two above |
| `disableFFmpeg` | `true` leaves out FFmpeg, which decodes compressed audio files — a conversation's raw audio never needs it, and the app is smaller |
| `disableStaticExternalLibs` | `react-native-audio-api`'s own switch, passed through |

## Bare React Native

After the install, `cd ios && pod install`. Then, by hand, what the plugin would do. iOS's `Info.plist` says why the microphone is asked for — and has **no** `audio` background mode:

```xml title="Info.plist"
<key>NSMicrophoneUsageDescription</key>
<string>Hold to talk to your coach.</string>
```

Android's `AndroidManifest.xml` asks for the microphone, and for the audio settings a call-like route needs:

```xml title="AndroidManifest.xml"
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
```

## Push-to-talk

Make the client once, put it in a provider, and give `useAgent` the audio. Pressing starts a recording — the agent's voice stops, so it never reaches the microphone — and releasing sends it as the person's turn:

```tsx title="App.tsx"
import { AgentProvider, createAudio, createClient, useAgent } from "@lumavox/react-native";
import { useEffect, useMemo } from "react";
import { Pressable, Text } from "react-native";

const agents = createClient({
  baseUrl: "https://api.lumavox.ai",
  // Your server mints the key with your API key: POST /v1/ephemeral-keys.
  getToken: async () => (await fetch(keyServer, { method: "POST" })).text(),
});

export default function App() {
  return (
    <AgentProvider client={agents}>
      <Talk agentId="agt_..." />
    </AgentProvider>
  );
}

function Talk({ agentId }: { agentId: string }) {
  const audio = useMemo(() => createAudio(), []);
  useEffect(() => () => audio.dispose(), [audio]);
  const agent = useAgent({ agentId }, { audio });
  return (
    <Pressable
      onPressIn={() => {
        void audio.prime(); // in the press, before any await: the web needs it
        void agent.startRecording();
      }}
      onPressOut={() => void agent.stopRecording()}
    >
      <Text>{agent.recording ? "Listening…" : agent.speaking ? "Speaking…" : "Hold to talk"}</Text>
    </Pressable>
  );
}
```

The audio is yours: make it once per screen and dispose of it when the screen goes — `useAgent` never does. Everything else is [the hooks'](https://docs.lumavox.ai/sdks/react.md#useagent): `turn`, `speaking`, `recording`, `cancel()`, `stopRecording({ discard: true })` for a press that slid off the button, `stopRecording({ transcribeOnly: true })` for a dictation's words with no turn.

- **Phones record at 16 kHz**, mono, 100 ms a buffer; a phone that records at another rate is resampled to it. A browser records at 24 kHz.
- **The agent plays from the loudspeaker.** Headphones and Bluetooth take over when connected. `createAudio({ output: "earpiece" })`, or `audio.setOutput("earpiece")`, plays it at the ear like a call — on iPhones; Android stays on the loudspeaker for now.
- **Bluetooth earbuds' microphone** is used on iPhones, like a call — and like a call, the agent sounds at call quality while they are on. Android records from the phone for now.
- **The microphone permission** is asked on the first press. To ask earlier, at a moment your app explains, call `requestMicrophone()` — it answers whether it was given.

## Interruptions

A phone call, Siri or another app's audio **pauses** the conversation: the reply stops where it is and goes on when the audio comes back, its words still arriving meanwhile; a recording in progress is **thrown away**, since its speaker was cut off mid-sentence. **Leaving the screen** does the same: there is no background audio in this release, so the conversation pauses with the app and goes on when it is back.

```tsx title="TSX"
const agent = useAgent({ agentId }, {
  audio,
  onAudioInterrupted: () => setNote("Paused for your call"),
  onAudioResumed: () => setNote(null),
});
// agent.interrupted: true while the phone has the audio
```

The audio comes back by itself when the system says it may. When it does not — another app went on playing — `agent.interrupted` stays `true` until the person presses to talk again, or you call `audio.resume()`, which rejects while the audio is still taken. On Android, a conversation on screen asks for the audio focus, so other apps' music pauses.

## The clock: a face, subtitles

The audio has a clock to draw against — a 3D character's mouth, subtitles revealed chunk by chunk — on a phone and on the web alike. `onScheduled` tells every frame of the reply as it is queued, before it plays: its 16-bit samples (`pcm`, `sample_rate`), its `chunk` and `turn`, and `startsAt` and `duration` in seconds on the clock `now()` reads. Draw against `heardTime()`, what the person hears at this moment:

```tsx title="TSX"
import { createAudio } from "@lumavox/react-native";
import { useEffect, useMemo } from "react";

function Character() {
  // 350 ms for your model to read each frame before it plays.
  const audio = useMemo(() => createAudio({ leadInMs: 350, outputLatencyMs: 0 }), []);
  useEffect(() => () => audio.dispose(), [audio]);
  useEffect(() => audio.onScheduled(({ pcm, sample_rate, startsAt }) => lipsync.add(pcm, sample_rate, startsAt)), [audio]);
  useEffect(() => audio.onStopped(() => lipsync.clear()), [audio]); // a cancel, a barge-in: never played
  // Each frame the face is drawn: lipsync.at(audio.heardTime())
  return null;
}
```

- **A reply's first frame waits `leadInMs`** — and again after any gap in the voice — so your model has that long to read it; frames after it follow back to back. **It adds as much to the first word heard.** None on a phone unless set (a buffer of silence played first), 60 ms on the web.
- **`outputLatencyMs`: what the phone's sound takes to reach the ear**, which `heardTime()` takes off the clock. `react-native-audio-api` 0.13 cannot say it, so it is `0` unless you set it — **Bluetooth earbuds add about 150 to 250 ms**. Its early-access build says the phone's own, unless you set it; on the web the browser says its own.
- **`onStopped`** fires when `stop()` drops audio still to play — a cancel, a mute, a barge-in, a new turn; `onDrained` then too, and when a reply plays out. **`onCapture((pcm, rate) => …)`** is every frame the microphone captures while it is open, for a level meter.
- On a phone, a reply at another sample rate — the agent's voice changed — makes a new player, whose clock starts again; what was queued is stopped, and `onStopped` says so. A browser keeps its clock.
- Each `on…` answers its unsubscribe, and a handler that throws is rethrown on its own: it never stops the voice.

## The web

The same code runs in a browser, on [the client's Web Audio](https://docs.lumavox.ai/sdks/client.md#audio-in-a-browser): React Native's bundler picks it for the web. **`audio.prime()` must run in the press**, before any `await` — browsers start sound only from a gesture, and a player started outside one is silent. There are no interruptions to tell and no earpiece there. **A browser lets the app choose the microphone, its processing and the speaker** — `createAudio({ input, processing })`, `inputs()`, `setInput`, `setProcessing`, and `outputs()` and `setOutputDevice` where `canChooseOutput` (Chrome and Edge): [the client's page](https://docs.lumavox.ai/sdks/client.md#the-microphone-and-the-speaker) has them. On a phone the system chooses: `inputs()` and `outputs()` answer `[]`, `canChooseOutput` is `false`, and `setOutput` picks the loudspeaker or the earpiece. It needs Metro's package `exports`, on by default from React Native 0.79 (Expo SDK 53); before, set `resolver.unstable_enablePackageExports` in `metro.config.js`. On iOS Safari, a page's audio can stay quieter once the microphone has opened (WebKit's bug 218012).

## Hands-free

Two ways to talk without holding a button, both `agent.listen()` [from the hooks](https://docs.lumavox.ai/sdks/react.md#hands-free): the microphone stays open for the whole conversation, and each thing the person says becomes a turn after 700 ms of silence.

| | Half-duplex | Hands-free |
|---|---|---|
| Start it | `agent.listen({ halfDuplex: true })` | `agent.listen()` |
| Talk over the agent to stop it | no — nothing is heard while it speaks | yes |
| On a phone | every phone, either build of `react-native-audio-api` | [the early-access build](#hands-free-on-a-phone-early-access) |
| The agent's voice | as in push-to-talk | like a speakerphone call while it listens |
| In a browser | every browser | where the browser cancels echo: Chrome, Edge, Firefox |

### Half-duplex

**Half-duplex runs on every phone**: tap once and talk for the whole conversation. The microphone stays open, but **nothing is heard while the agent speaks** — so it never hears itself through the loudspeaker, and needs no echo cancellation. The agent's voice cannot be talked over, so give the person a stop button (`agent.cancel()`).

```tsx title="TSX"
const agent = useAgent({ agentId }, { audio });
// Tap once: the microphone stays open; nothing is heard while the agent speaks.
void agent.listen({ halfDuplex: true });
// agent.listening and agent.hearing say how it goes; agent.stopListening() ends it
```

- **The agent speaking is what the person hears**, on the player's clock — set `outputLatencyMs` with Bluetooth earbuds on, so it ends when they stop sounding.
- **The microphone waits 300 ms after the voice ends** (`afterVoiceMs`), so the room's echo of its last word does not open a recording. Words begun in it keep their start; a word over before it ends — a quick "yes" under 0.3 s — is not heard.
- **The phone shows the microphone in use** for the whole conversation, as any app that keeps it open: the orange dot on an iPhone, the green one on Android.
- [The client's page](https://docs.lumavox.ai/sdks/client.md#half-duplex) has the rest.

### Hands-free on a phone (early access)

**Talking over the agent to stop it needs the phone to cancel its echo**: the agent's voice, from the loudspeaker, must be taken out of what the microphone hears, or the agent answers itself. `react-native-audio-api` cancels echo from its 1.0, which is not released yet — so hands-free on a phone runs, **in early access**, on one build of it, named exactly:

```bash title="npm"
npm install @lumavox/react-native react-native-audio-api@1.0.0-nightly-6fd4c68-20261102
```

- **That build, and no other.** On 0.13, or a later test build of the library, `listen()` answers `false` and `agent.error` says why — on a later build, naming the one it runs on. Everything else — push-to-talk, half-duplex, the clock — runs on that build and on 0.13 alike, so you can move back with `npm install react-native-audio-api@0.13`.
- **Build the app again** after changing it: `npx expo prebuild --clean`, then `npx expo run:ios` and `npx expo run:android`; bare, `cd ios && pod install`. The plugin above works with both builds — still never list the library's own. This build's own plugin writes its build options into your `package.json`, under `"react-native-audio-api"`, as it runs.

Ask the audio first, then listen — hands-free where it may, half-duplex where it may not:

```tsx title="TSX"
const agent = useAgent({ agentId }, { audio });

async function talk() {
  const may = await audio.handsFree();
  // On "no", may.reason says why, in words to show.
  await agent.listen(may.ok ? {} : { halfDuplex: true });
}
```

- **Talking over the agent** lowers its voice at once; it stops when the person's first words are heard — about two seconds in — and what they say becomes the next turn. Sound with no words — a cough, the agent's own voice — never stops it: its voice comes back up.
- **While it listens, the phone's sound is a call's.** On an iPhone, Apple's voice processing takes the agent's voice out of the microphone; the agent plays from the loudspeaker (the earpiece with `output: "earpiece"`), a little quieter and narrower, as a speakerphone does, and other apps' sound is lowered. On Android, the phone's call audio: its echo canceller, the loudspeaker, and the call's volume, which the volume buttons then set.
- **Bluetooth earbuds** take the agent's voice and the person's, by the earbuds' microphone, on both.
- **`stopListening()` gives the phone its usual sound back.** On an iPhone the voice has a short gap as listening starts and stops, while the phone's audio restarts; on Android a reply playing finishes first.
- **Where a phone's echo cancelling is poor**, the agent's own voice can still reach the microphone: it lowers its voice now and then for no one, and comes back up. Offer half-duplex as a setting.
- **The microphone shows in use** for the whole conversation, as in half-duplex. Nothing is sent or billed between the person's turns.

With headphones on, the agent's voice never reaches the microphone, so hands-free runs on 0.13 too, at the usual sound:

```tsx title="TSX"
const agent = useAgent({ agentId }, { audio });
// Headphones on: the agent's voice never reaches the microphone.
void agent.listen({ assumeNoEcho: true });
// agent.listening and agent.hearing say how it goes; agent.error, why it refused
```

## Not in this release

- **Hands-free on a phone's loudspeaker on `react-native-audio-api` 0.13**: it needs [the early-access build](#hands-free-on-a-phone-early-access) until the library's 1.0 is released. Half-duplex and push-to-talk work on both.
- **Background audio** — a conversation going on with the screen off, as a call does — will be your app's choice, off by default. With it come the stores' rules: App Review checks that background audio is used as meant and that recording shows; Google Play asks every app targeting Android 14 or later to declare its microphone foreground service, with a video of it.

**Each audio frame reaches JavaScript as base64** inside React Native's `WebSocket`, a cost no SDK can remove: keep the screen's renders light while the agent speaks, or a long render is a gap in its voice.