OpenAI Realtime API

OpenAI Realtime API

The OpenAI Realtime API is the low-latency path for live audio. Unlike the request-based TTS/STT APIs above, a realtime session keeps a connection open, accepts audio or text as it arrives, and streams events back immediately. OneRingAI exposes the GA API through the same connector-first authentication used everywhere else in the library.

Use the Realtime API for:

• speech-to-speech agents that reason, answer, and call tools;
• live captions or call transcription without an assistant response;
• continuous speech-to-speech translation;
• browser/mobile audio over WebRTC;
• server media pipelines over WebSocket; and
• telephony through OpenAI SIP controls or VoiceBridge with Twilio Media Streams.

Choose a Session Type and Transport

OpenAIRealtimeAgentSession is the preferred voice-agent surface. It binds a normal Agent to a Realtime transport and preserves Agent context plugins, memory refresh, identity-filtered tools, permission policies, tool hooks, limits, metrics, parallel function results, and provider-hosted MCP approvals. Its transport contract can be implemented by WebSocket, a WebRTC data channel, or a SIP/server-control bridge.

OpenAIRealtimeSession is the lower-level server-side WebSocket client. It handles connector authentication, session creation, JSON serialization, endpoint selection, and the protocol differences between conversation, transcription, and translation sessions. It intentionally leaves audio capture/playback to the host and exposes every server event so applications are not limited by a closed event union as OpenAI evolves the protocol.

OpenAIRealtimeAPI contains the REST operations: minting WebRTC credentials, exchanging an SDP offer with /v1/realtime/calls, and controlling SIP calls. It does not put a long-lived OpenAI API key in the browser.

Authentication and Setup

Create a normal OpenAI connector. The connector may use the default OpenAI base URL or an explicit compatible proxy URL.

import {
  Connector,
  OpenAIRealtimeAPI,
  OpenAIRealtimeSession,
  Vendor,
} from 'oneringai';

const openai = Connector.create({
  name: 'openai',
  vendor: Vendor.OpenAI,
  auth: {
    type: 'api_key',
    apiKey: process.env.OPENAI_API_KEY!,
  },
});

Keep the standard API key on a trusted server. For an identifiable end user, pass a stable, privacy-preserving safetyIdentifier (for example, a one-way hash of an internal user id). The library sends it through the OpenAI-Safety-Identifier header.

Server WebSocket Voice Agent

For a tool-using agent, use the agent-aware session. contextSync: 'per_turn' is the default: it disables automatic VAD response creation, waits for the input transcript, refreshes Agent memory/plugin instructions and the dynamic tool set, then requests the response. If transcription fails, it still continues with the original audio so the turn cannot deadlock.

import {
  Agent,
  OpenAIRealtimeAgentSession,
} from 'oneringai';

const agent = Agent.create({
  connector: 'openai',
  model: 'gpt-realtime-2.1',
  instructions: 'Be concise and confirm consequential actions.',
  userId: 'user-123',
  tools: [lookupOrder, createTicket],
  permissions: { onApprovalRequired: approveLocalTool },
  context: {
    model: 'gpt-realtime-2.1',
    features: { memory: true },
    plugins: { memory: { memory } },
  },
});

const voice = new OpenAIRealtimeAgentSession({
  agent,
  safetyIdentifier: hashUserId('user-123'),
  session: {
    reasoning: { effort: 'low' },
    audio: {
      input: {
        format: { type: 'audio/pcm', rate: 24000 },
        turn_detection: { type: 'semantic_vad', eagerness: 'auto' },
      },
      output: {
        format: { type: 'audio/pcm', rate: 24000 },
        voice: 'marin',
      },
    },
    tools: [{
      type: 'mcp',
      server_label: 'crm',
      connector_id: 'conn_crm',
      require_approval: 'always',
    }],
  },
  approveMCP: async (request) => authorizeHostedTool(request),
});

voice.on('audio', playPcm24);
voice.on('transcript:input', showUserTranscript);
voice.on('transcript:output', showAssistantTranscript);
voice.on('usage', recordCumulativeUsage);
voice.on('error', reportVoiceError);

await voice.connect();
voice.appendAudio(pcm24Chunk);

// Text and image turns use the same managed Agent context.
await voice.sendText('Summarize the current order status.');
await voice.sendImage('https://example.com/damaged-package.jpg');

await voice.close();
agent.destroy();

Missing approveMCP handlers reject MCP approval requests by default. Local function calls always execute through Agent.executeExternalToolCall(), so Agent hooks, connector identity, permission policy, circuit breakers, hard timeouts, and metrics behave the same way as in agent.run().

The agent-aware session decodes base64 response.output_audio.delta events and emits their bytes through audio. A custom WebRTC/media transport that already does this must set emitsOutputAudioFromEvents: true to avoid duplicate audio. Normal close() completes the external Agent execution, closing after agent.cancel() reports cancelled, and an unrequested transport close reports failed regardless of its WebSocket close code.

Use the raw WebSocket session when the host intentionally owns context and tool execution itself:

The default model is gpt-realtime-2.1. Audio sent to appendAudio() may be a Buffer or an already-base64-encoded string. PCM audio is signed 16-bit little endian, mono, at 24 kHz; PCMU and PCMA use their native 8 kHz telephony format.

const realtime = new OpenAIRealtimeSession({
  connector: openai,
  model: 'gpt-realtime-2.1',
  safetyIdentifier: hashUserId('user-123'),
  session: {
    instructions: 'Be concise, friendly, and confirm important numbers.',
    reasoning: { effort: 'low' },
    output_modalities: ['audio'],
    audio: {
      input: {
        format: { type: 'audio/pcm', rate: 24000 },
        noise_reduction: { type: 'near_field' },
        turn_detection: {
          type: 'semantic_vad',
          eagerness: 'auto',
          create_response: true,
          interrupt_response: true,
        },
      },
      output: {
        format: { type: 'audio/pcm', rate: 24000 },
        voice: 'marin',
      },
    },
  },
});

realtime.on('event', (event) => {
  switch (event.type) {
    case 'response.output_audio.delta':
      playPcm24(Buffer.from(event.delta, 'base64'));
      break;
    case 'response.output_audio_transcript.delta':
      process.stdout.write(event.delta);
      break;
    case 'error':
      console.error(event.error);
      break;
  }
});

realtime.on('error', console.error); // socket, parsing, and connection errors
realtime.on('close', (code, reason) => console.log('closed', code, reason));

await realtime.connect();

// Text turns are useful for testing or mixed text/audio applications.
realtime.sendText('Introduce yourself in one short sentence.');
realtime.createResponse();

// Stream microphone/server audio as it arrives.
realtime.appendAudio(pcm24Chunk);

// Only commit/request manually when automatic VAD is disabled.
realtime.commitAudio();
realtime.createResponse();

For a new Realtime WebSocket, connect() resolves after session.created. For a callId sideband connection to an existing WebRTC or SIP call, OpenAI does not replay that creation event; the library waits for the socket to open, sends an initial session.update, and resolves after session.updated acknowledges the control channel. Register event listeners before connecting so no early events are missed.

Session Helpers and Events

The wrapper keeps the protocol accessible instead of hiding it behind a narrow abstraction:

Every server message is emitted through the typed event subscription. At runtime it is also emitted under its exact event-type string for consumers using the underlying EventEmitter dynamically, except API error payloads: Node reserves the error event for Error objects, so OpenAI error payloads remain on the raw event stream. Prefer the typed event listener shown above.

OpenAIRealtimeServerEvent and OpenAIRealtimeClientEvent have a typed type and event_id, plus an open index signature. Narrow on event.type before reading event-specific fields.

Audio, VAD, and Interruption Configuration

Supported audio formats are:

type OpenAIRealtimeAudioFormat =
  | { type: 'audio/pcm'; rate: 24000 }
  | { type: 'audio/opus'; rate?: 24000 }
  | { type: 'audio/pcmu' }
  | { type: 'audio/pcma' };

type GrokRealtimeAudioFormat =
  | { type: 'audio/pcm'; rate: 8000 | 16000 | 22050 | 24000 | 32000 | 44100 | 48000 }
  | { type: 'audio/opus'; rate?: 24000 }
  | { type: 'audio/pcmu' }
  | { type: 'audio/pcma' };

OpenAIRealtimeAudioFormat accepts only OpenAI's 24 kHz PCM contract, while GrokRealtimeAudioFormat carries xAI's full PCM range. REST and WebSocket OpenAI boundaries—including every updateSession() call—also validate the rate at runtime to protect JavaScript and type-escape callers. The xAI-specific session accepts its wider rate set in both initial configuration and updates.

Turn detection accepts server_vad, semantic_vad, or null:

// Silence-based VAD
const serverVAD = {
  type: 'server_vad' as const,
  threshold: 0.5,
  prefix_padding_ms: 400,
  silence_duration_ms: 500,
  idle_timeout_ms: 15_000,
  create_response: true,
  interrupt_response: true,
};

// Model-based end-of-turn detection
const semanticVAD = {
  type: 'semantic_vad' as const,
  eagerness: 'auto' as const, // low | medium | high | auto
  create_response: true,
  interrupt_response: true,
};

Set turn_detection: null when your application owns the turn boundary. Append audio, call commitAudio(), and then createResponse(). When users interrupt playback, cancel the response, stop local playback, and use truncateItem() to keep server conversation state aligned with what the user actually heard. VoiceBridge performs this barge-in sequence automatically.

Input noise reduction supports { type: 'near_field' }, { type: 'far_field' }, or null. Output speed is supported from 0.25 to 1.5.

Advanced Session Configuration

The session type covers reasoning, tools, tracing, stored prompts, token limits, parallel tool calls, and retention-ratio truncation:

const advanced = new OpenAIRealtimeSession({
  connector: 'openai',
  model: 'gpt-realtime-2.1',
  session: {
    reasoning: { effort: 'low' },
    max_output_tokens: 2048,
    parallel_tool_calls: true,
    prompt: {
      id: 'pmpt_voice_support',
      version: '3',
      variables: { company: 'Acme' },
    },
    tracing: {
      workflow_name: 'phone-support',
      group_id: 'call-42',
      metadata: { region: 'eu' },
    },
    truncation: {
      type: 'retention_ratio',
      retention_ratio: 0.8,
      token_limits: { post_instructions: 12_000 },
    },
    tools: [{
      type: 'mcp',
      server_label: 'support',
      connector_id: 'conn_support_mcp',
      allowed_tools: { read_only: true },
      require_approval: 'never',
      defer_loading: true,
    }],
  },
});

For a function tool sent directly to OpenAI, use the flattened Realtime shape:

const lookupTool = {
  type: 'function' as const,
  name: 'lookup_order',
  description: 'Look up an order by id',
  parameters: {
    type: 'object',
    properties: { orderId: { type: 'string' } },
    required: ['orderId'],
    additionalProperties: false,
  },
};

Raw OpenAIRealtimeSession surfaces OpenAI tool-call events but deliberately does not execute function tools. Use OpenAIRealtimeAgentSession or VoiceBridge when local ToolFunctions should execute with the full Agent lifecycle. Remote MCP tools remain provider-executed; the agent-aware surfaces answer approval requests through approveMCP and reject them when no handler is configured.

Voice-Agent Models

Realtime voices are alloy, ash, ballad, coral, echo, marin, sage, shimmer, verse, and cedar; custom { id: 'voice_…' } references are also accepted by the session type. OpenAI recommends marin or cedar for best quality, and VoiceBridge defaults to marin. Select the voice before the model first emits audio; OpenAI does not allow changing it later in the same session.

Realtime Transcription

Use a transcription session when the application needs streaming text without an assistant response. The connection uses intent=transcription; the actual transcription model belongs in session.audio.input.transcription.model. OneRingAI applies that distinction automatically.

const transcription = new OpenAIRealtimeSession({
  connector: 'openai',
  session: {
    type: 'transcription',
    audio: {
      input: {
        format: { type: 'audio/pcm', rate: 24000 },
        noise_reduction: { type: 'near_field' },
        transcription: {
          model: 'gpt-live-transcribe',
          prompt: 'Customer support call about account AC-42.',
          keywords: ['AC-42', 'premium plan', 'billing'],
          languages: ['en', 'fr'],
          delay: 'low',
        },
        turn_detection: null,
      },
    },
  },
});

transcription.on('event', (event) => {
  if (event.type === 'conversation.item.input_audio_transcription.delta') {
    process.stdout.write(event.delta);
  }
  if (event.type === 'conversation.item.input_audio_transcription.completed') {
    console.log('\nfinal:', event.transcript);
  }
});

await transcription.connect();
transcription.appendAudio(pcm24Chunk);
transcription.commitAudio();

gpt-live-transcribe is the recommended streaming model. The registry also includes gpt-realtime-whisper; both are priced at $0.017 per audio minute. Use languages for multiple expected languages and do not combine it with the singular language field. With automatic VAD enabled, the server can commit turns; with turn_detection: null, the host commits them explicitly.

Completion events from separate turns are not guaranteed to arrive in order. Use item_id and content_index to reconcile partial and final transcripts. Live transcription does not provide word-level timestamps, speaker labels, or confidence scores; use a compatible file-transcription workflow when those are required.

Realtime Translation

Translation is a dedicated continuous protocol, not a voice-agent prompt. It uses gpt-realtime-translate and /v1/realtime/translations. Do not call createResponse() or wait for a conversation turn: keep appending source audio, including silence between phrases, and consume translated output as it arrives.

const translation = new OpenAIRealtimeSession({
  connector: 'openai',
  model: 'gpt-realtime-translate', // also selects realtime/translations
  session: {
    audio: {
      output: { language: 'es' },
    },
  },
});

translation.on('event', (event) => {
  if (event.type === 'session.input_transcript.delta') {
    updateSourceCaption(event.delta);
  }
  if (event.type === 'session.output_transcript.delta') {
    updateSpanishCaption(event.delta);
  }
  if (event.type === 'session.output_audio.delta') {
    playPcm24(Buffer.from(event.delta, 'base64'));
  }
  if (event.type === 'session.closed') {
    translation.close();
  }
});

await translation.connect();
translation.appendAudio(pcm24Chunk);

// At end of source media, flush pending translation before closing the socket.
translation.closeTranslation();

The WebSocket session.update translation shape contains only the target output language. The WebRTC client-secret shape additionally requires model: 'gpt-realtime-translate'; the library exposes separate TypeScript types so the two wire contracts cannot be accidentally mixed.

Use one translation session per output language. For two-way interpretation, keep participant tracks separate and create one session per direction. For WebSocket media, source audio is base64 24 kHz PCM16. For browser media, WebRTC handles capture and playback as media tracks.

Browser and Mobile WebRTC Credentials

Create ephemeral credentials on your server and return only the short-lived secret to the client. expiresAfterSeconds must be between 10 and 7,200.

const api = new OpenAIRealtimeAPI('openai');

// Voice-agent or transcription credential.
const voiceSecret = await api.createClientSecret({
  session: {
    type: 'realtime',
    model: 'gpt-realtime-2.1-mini',
    instructions: 'Answer briefly.',
    audio: { output: { voice: 'marin' } },
  },
  expiresAfterSeconds: 600,
  safetyIdentifier: hashUserId('user-123'),
});

// Translation has a separate credential endpoint and session shape.
const translationSecret = await api.createTranslationClientSecret({
  session: {
    model: 'gpt-realtime-translate',
    audio: { output: { language: 'de' } },
  },
  safetyIdentifier: hashUserId('user-123'),
});

// Trusted-server WebRTC setup can exchange an SDP offer directly.
const { sdp: sdpAnswer, callId } = await api.createWebRTCCallWithMetadata({
  sdp: browserOffer.sdp!,
  session: {
    model: 'gpt-realtime-2.1',
    audio: { output: { voice: 'marin' } },
  },
  safetyIdentifier: hashUserId('user-123'),
});

// Apply sdpAnswer in the peer. A server-owned Agent can join by call ID:
const serverAgentSession = new OpenAIRealtimeAgentSession({ agent, callId });
await serverAgentSession.connect();
// Send `voiceSecret.value` or `translationSecret.value` only to the intended client.

createWebRTCCall() remains source-compatible with earlier OneRingAI releases and returns only the SDP answer string. Use createWebRTCCallWithMetadata() when the caller also needs the opaque call ID for sideband control or hangup.

The browser-safe OpenAIRealtimeWebRTCPeer creates an RTCPeerConnection, adds its microphone track, creates the oai-events data channel, and exchanges SDP through a host callback. It intentionally accepts no provider credential. The full Agent can remain in an Electron main process and consume the bridged data channel through OpenAIRealtimeChannelTransport. Use /v1/realtime/calls for a voice-agent SDP offer and /v1/realtime/translations/calls for translation.

// Renderer or browser-safe bundle
import { OpenAIRealtimeWebRTCPeer } from 'oneringai/realtime-browser';

const peer = new OpenAIRealtimeWebRTCPeer({
  exchangeSdp: (offer, { signal }) => appBackend.createRealtimeCall({ offer, signal }),
  releaseCall: (callId) => appBackend.hangupRealtimeCall({ callId }),
  onRemoteTrack: (streamOrTrack) => {
    if (typeof MediaStream !== 'undefined' && streamOrTrack instanceof MediaStream) {
      audioElement.srcObject = streamOrTrack;
    }
  },
});
await peer.connect();

// On explicit shutdown, wait until both local media and the provider call end.
await peer.closeAndRelease();

exchangeSdp must return { sdp, callId }; the old string-only answer is not accepted because it cannot support reliable cleanup. Before the first message subscriber, the peer retains at most 256 events or 1 MiB by default. Before connect(), OpenAIRealtimeChannelTransport independently retains at most 256 events or 1 MiB. Configure these with maxPendingMessages, maxPendingMessageBytes, maxPendingEvents, and maxPendingEventBytes. Overflow closes or fails the session instead of dropping an older ordered event. Bridges that implement their own channel should preserve the same ordering and may use getSessionCreatedMessage() for compatibility with creation-event replay.

Never send the connector's standard API key to a browser or mobile application. Mint client secrets per session and associate the safety identifier when the secret is created.

SIP Call Control

OpenAIRealtimeAPI also wraps server-side controls for an OpenAI SIP call. The callId normally comes from the incoming-call webhook event.

const api = new OpenAIRealtimeAPI('openai');

await api.acceptCall(callId, {
  model: 'gpt-realtime-2.1',
  instructions: 'You are Acme telephone support.',
  audio: { output: { voice: 'cedar' } },
});

// Reject an unanswered call (486 = busy, for example).
await api.rejectCall(callId, 486);

// Transfer an accepted call.
await api.referCall(callId, 'sip:human-agent@example.com');

// End an accepted call.
await api.hangupCall(callId);

Webhook signature validation, replay protection, routing policy, and mapping the incoming payload to callId remain host responsibilities. Accept/reject/refer/ hangup calls are authenticated through the selected connector.

VoiceBridge Realtime Telephony

For Twilio Media Streams, use VoiceBridge with pipeline: 'realtime'. It creates a fresh full Agent per call, translates agent instructions and enabled local tools into the Realtime session, streams native PCMU in both directions, executes tool calls through the full Agent lifecycle, refreshes context before every response, maintains a transcript and cumulative usage, and handles interruption timing and truncation.

import {
  TwilioAdapter,
  VoiceBridge,
} from 'oneringai';

const bridge = VoiceBridge.create({
  agent: {
    connector: 'openai',
    model: 'gpt-realtime-2.1',
    instructions: 'Be a concise and helpful phone support agent.',
    tools: [lookupOrder, createTicket],
    permissions: {
      onApprovalRequired: approveVoiceTool,
    },
  },
  pipeline: 'realtime',
  voice: 'marin',
  speed: 1,
  turnDetection: 'semantic_vad',
  semanticVADEagerness: 'auto',
  noiseReduction: 'far_field',
  inputTranscription: true,
  transcriptionModel: 'gpt-4o-transcribe',
  interruptible: true,
  greeting: 'Hello, this is Acme support. How can I help?',
  greetingOutbound: 'Hello, I am calling from Acme support.',
  safetyIdentifier: hashUserId('caller-123'),
  maxConcurrentCalls: 20,
  maxCallDuration: 3600,
  realtime: {
    reasoning: { effort: 'low' },
    tracing: { workflow_name: 'twilio-support' },
    truncation: { type: 'retention_ratio', retention_ratio: 0.8 },
  },
  approveMCP: async (request, session) =>
    authorizeHostedVoiceTool(session.callId, request),
  hooks: {
    onCallStart: async (session) => isAllowedCaller(session.from),
    onInterrupt: async (session) => audit('interrupt', session.sessionId),
    onCallEnd: async (session, summary) => saveCallSummary(session, summary),
    onError: async (error, session) => reportCallError(error, session),
  },
});

const twilio = TwilioAdapter.createStandalone({
  connector: 'twilio',
  port: 3000,
  publicUrl: 'https://voice.example.com',
});

bridge.on('transcript', (session, entry) => {
  console.log(session.callId, entry.role, entry.text);
});

bridge.attach(twilio);
await twilio.start();

When the effective Agent depends on caller or tenant context, replace the static agent object with a per-call factory:

const bridge = VoiceBridge.create({
  agentFactory: async (session) => resolveAuthorizedAgent(session),
  pipeline: 'realtime',
  voice: 'marin',
});

The factory must return a new Agent. VoiceSession owns and destroys it.

Realtime beforeAgentResponse and afterAgentResponse hooks are observational: the model has already consumed or generated audio concurrently, so returned text cannot rewrite that audio. Use agent instructions, tools, or session controls to change realtime behavior. onCallStart, onInterrupt, onError, and onCallEnd retain their normal lifecycle semantics.

VoiceBridge realtime configuration:

With input transcription enabled, the OpenAI bridge refreshes Agent context and tools before every response. Setting inputTranscription: false intentionally falls back to initial-only context synchronization and preserves OpenAI's normal server/manual response creation.

When turnDetection: 'none', VoiceBridge uses local EnergyVAD, commits the audio buffer, and requests each response itself. The existing vad and silenceTimeout options configure that local detector. Server and semantic VAD do not use EnergyVAD.

On barge-in, the pipeline cancels the active model response, clears queued telephony playback, and truncates the assistant conversation item at the media timestamp actually reached by the caller. Tool calls and results are added to the bridge transcript as tool_use and tool_result entries.

Tool Security

There are two distinct tool paths:

1. Local OneRingAI tools are supplied through agent.tools. OpenAIRealtimeAgentSession and VoiceBridge execute them through the normal Agent lifecycle, including permission policy, user rules, before/approve/after hooks, circuit breakers, tool context, limits, cancellation, events, and metrics.
2. Provider-hosted MCP tools are supplied through realtime.tools. They run through OpenAI's Realtime tool path. Configure connector authorization and allowed tools explicitly; approval requests fail closed unless approveMCP returns an affirmative decision.

Do not place long-lived secrets directly in model-visible tool definitions. Prefer an OpenAI-managed connector reference (connector_id) or resolve credentials on the trusted server. This field is an OpenAI connector id, not a OneRingAI connector-registry name. Treat third-party MCP data movement as an explicit trust decision.

Realtime Cost Estimation

Realtime voice-agent models have separate text, audio, and image token prices. Select the modality when calculating cost. Translation is billed by audio duration rather than tokens.

import { calculateCost } from 'oneringai';

const audioTokenCost = calculateCost(
  'gpt-realtime-2.1',
  12_000,
  4_000,
  {
    modality: 'audio',
    cachedInputTokens: 2_000,
  },
);

const translationCost = calculateCost(
  'gpt-realtime-translate',
  0,
  0,
  { audioMinutes: 15 },
); // $0.51 at $0.034/minute

If gpt-realtime-translate is used without audioMinutes, calculateCost() returns null. Image output cost is also null when the registry has no image output price. Inspect getModelInfo(model).features.pricing for the exact modality metadata used by the calculator.

Production Checklist and Boundaries

• Use WebRTC for browser/mobile capture and playback; use WebSocket when a trusted server already owns raw audio.
• Never expose the standard OpenAI connector token to a client. Mint a new ephemeral secret for each WebRTC session.
• Set safetyIdentifier to a stable privacy-preserving value per end user.
• Benchmark VAD, noise reduction, reasoning effort, latency, interruption, and reconnect behavior with real microphones, accents, network conditions, and telephony audio.
• Handle the raw error event payload and socket-level error separately.
• Observe backpressure and do not queue unbounded raw audio; the WebSocket transport rejects new audio frames after maxBufferedAmountBytes (1 MiB by default).
• Recreate or hand off the session when session:expiring fires. The default warning is at 55 minutes, before OpenAI's hard 60-minute session limit.
• To carry semantic context across that boundary, first close the old OpenAIRealtimeAgentSession (releasing its external-execution lease), then call agent.rolloverContext({ reason: 'realtime-session-expiring' }), and create the replacement session with the same Agent. The rollover preserves eight recent user turns exactly by default, never splits tool-call pairs, preserves plugin state and the full history journal, and checkpoints when session storage is configured. If the Agent's configured model cannot make a normal text generation call, pass a trusted summarize callback backed by a text-capable Agent or server proxy.
• Stop local playback when cancelling a response; keep server conversation state aligned with truncateItem().
• For translation, send closeTranslation(), continue reading until session.closed, and only then close the socket.
• Reconcile transcription turns by item_id, not event arrival order.
• Apply normal OneRingAI permission policies to every local tool used by a phone or voice agent.
• Validate SIP webhook signatures and make accept/refer decisions on a trusted server.
• Record consent and disclosures where call recording, transcription, synthetic speech, or automated calling laws require them.

The library's WebSocket client transports raw audio but does not capture a microphone or play sound. The REST helper mints credentials; the separate browser-safe OpenAIRealtimeWebRTCPeer manages RTCPeerConnection through a host SDP callback. SIP helpers control an existing OpenAI call but do not provision phone numbers or receive webhooks. VoiceBridge supplies those media and lifecycle integrations for supported telephony adapters.

xAI Realtime Voice Agent API

xAI's Voice Agent API is wire-compatible with the OpenAI Realtime protocol, so GrokRealtimeSession, RealtimePipeline, and VoiceBridge select xAI behavior from a Vendor.Grok connector. grok-voice-latest currently resolves through the registry to grok-voice-think-fast-2.0.

import {
  Connector,
  GrokRealtimeAPI,
  GrokRealtimeSession,
  Vendor,
} from 'oneringai';

Connector.create({
  name: 'xai',
  vendor: Vendor.Grok,
  auth: { type: 'api_key', apiKey: process.env.XAI_API_KEY! },
});

const voice = new GrokRealtimeSession({
  connector: 'xai',
  model: 'grok-voice-latest',
  reasoningEffort: 'high', // handshake option: 'high' or 'none'
  session: {
    instructions: 'Be concise and helpful.',
    voice: 'eve',
    turn_detection: { type: 'server_vad' },
    resumption: { enabled: true },
    audio: {
      input: {
        format: { type: 'audio/pcm', rate: 32000 },
        transport: 'binary',
        transcription: {
          language_hint: 'en',
          keyterms: ['OneRingAI'],
        },
      },
      output: {
        format: { type: 'audio/pcm', rate: 32000 },
        transport: 'binary',
        speed: 1,
      },
    },
  },
});

voice.on('event', (event) => {
  if (event.type === 'response.output_audio.delta') playBase64(event.delta);
});
await voice.connect();

Unlike OpenAI's Realtime session shape, xAI requires voice and turn_detection directly under session. Transcription uses session.audio.input.transcription.language_hint (not language). RealtimePipeline and VoiceBridge translate their normalized options to this xAI layout automatically.

Set audio.input.transport or audio.output.transport to binary to use raw WebSocket frames. appendAudio(Buffer) then sends binary input directly and the session emits the typed audio event for binary assistant frames. JSON remains the default and is the transport used by RealtimePipeline.

xAI accepts PCM sample rates of 8,000, 16,000, 22,050, 24,000, 32,000, 44,100, and 48,000 Hz for both input and output. PCMU/PCMA keep their native telephony rate. GrokRealtimeSessionConfig represents that range; the OpenAI session type and runtime boundary remain fixed at 24,000 Hz.

Resume a disconnected conversation by constructing a new session with the previous conversationId. For inbound SIP, pass callId; the model query is then omitted because the call already owns its session.

Browser clients must use an ephemeral secret instead of exposing the team API key:

const api = new GrokRealtimeAPI('xai');
const secret = await api.createClientSecret(300);

await api.referCall('call_123', 'tel:+14155550123');
await api.hangupCall('call_123');

xAI also supports MCP, hosted web/X/collections search, custom functions, pronunciation replacement, language hints, keyterms, Opus/PCM/G.711 codecs, custom voices, session resumption, and SIP. Provider-specific session fields can be sent through the typed session shape where represented or via raw protocol events with send().

xAI currently accepts server_vad or disabled turn detection. Do not reuse an OpenAI semantic_vad configuration for a Grok session: RealtimePipeline rejects it during initialization so an unsupported session update never reaches the provider.

Public Realtime Exports

The package root exports:

• runtime classes: OpenAIRealtimeAgentSession, OpenAIRealtimeSession, OpenAIRealtimeAPI, OpenAIRealtimeChannelTransport, GrokRealtimeSession, GrokRealtimeAPI, RealtimePipeline, and VoiceBridge;
• session/API types: OpenAIRealtimeSessionOptions, OpenAIRealtimeAgentSessionOptions, OpenAIRealtimeAgentTransport, OpenAIRealtimeChannelTransportOptions, CreateRealtimeClientSecretOptions, and CreateRealtimeTranslationClientSecretOptions, and CreateRealtimeWebRTCCallOptions, OpenAIRealtimeWebRTCCall, and RealtimeMessageChannel;
• configuration types: OpenAIRealtimeSessionConfig, OpenAIRealtimeTranscriptionSessionConfig, OpenAIRealtimeTranslationSessionConfig, and OpenAIRealtimeTranslationClientSessionConfig, plus GrokRealtimeSessionConfig;
• audio types: OpenAIRealtimePCMSampleRate, RealtimePCMSampleRate, OpenAIRealtimeAudioFormat, and GrokRealtimeAudioFormat;
• protocol types: OpenAIRealtimeClientEvent, OpenAIRealtimeServerEvent, OpenAIRealtimeClientSecret, audio/VAD types, function/MCP tool types, voices, and model ids; and
• telephony types: RealtimePipelineConfig, VoiceBridgeConfig, VoiceAgentFactory, VoicePipelineEvents, TranscriptMessage, and adapter/session types.

The browser-only oneringai/realtime-browser subpath exports OpenAIRealtimeWebRTCPeer, OpenAIRealtimeWebRTCPeerOptions, its event types, and the shared Realtime channel/event types without loading the Node.js SDK entry point.

Run the complete server-side example with:

npm run example:realtime

OpenAI reference material: Realtime overview, WebSocket, WebRTC, transcription, translation, and SIP.