Advanced Inference

Advanced Inference

Advanced inference is the provider-neutral layer for capabilities that go beyond a basic interactive text request:

• prompt/context caching;
• asynchronous provider batch jobs;
• tools executed by the model provider;
• detailed cache, reasoning, tool, tier, and processing-mode telemetry;
• explicit host authorization for retention-sensitive or third-party features.

The API is normalized, but support is not assumed to be uniform. Every request is checked against the concrete provider and model before provider inference. Unknown model families fail closed for provider-hosted tools and asynchronous batches.

Four API surfaces

Advanced options appear on four related surfaces. Agent-facing calls use camelCase; the low-level provider/batch contract uses snake_case.

RunOptions take precedence over agent-level values. vendorOptions are shallow-merged; advanced objects and arrays are replaced as whole values. Pass nativeTools: [] or promptCache: { mode: 'off' } to suppress an agent-level request for one context-managed call. Direct calls intentionally use only DirectCallOptions for per-call inference settings; they do not inherit the agent's instructions, temperature, or advanced request defaults.

const agent = Agent.create({
  connector: 'openai-main',
  model: 'gpt-5.4',
  promptCache: { mode: 'auto', key: 'assistant-v4' },
  dataHandling: { allowProviderCaching: true },
});

// Uses the agent defaults.
await agent.run('Analyze the attached policy');

// Replaces the cache policy for this call.
await agent.run('Handle a sensitive one-off request', {
  promptCache: { mode: 'off' },
});

Capability discovery

Always discover executable behavior from the agent/provider pair before enabling a feature in a UI or workflow:

const capabilities = agent.getAdvancedCapabilities();

if (capabilities.promptCaching.mode !== 'unsupported') {
  console.log('TTL choices:', capabilities.promptCaching.ttlModes);
}

if (capabilities.nativeTools.includes('file_search')) {
  // Show vector-store selection in the UI.
}

const batch = agent.getBatchProvider();
if (capabilities.batch.supported && batch) {
  console.log('Maximum requests:', capabilities.batch.maxRequests);
  console.log('Completion window:', capabilities.batch.completionWindow);
}

The returned contract is:

interface AdvancedTextCapabilities {
  promptCaching: {
    mode: 'unsupported' | 'implicit' | 'request_controlled' | 'explicit_resource';
    ttlModes: Array<'short' | 'extended'>;
    reportsCacheUsage: boolean;
    explicitBreakpoints?: boolean;
  };
  batch: {
    supported: boolean;
    cancellable: boolean;
    maxRequests?: number;
    completionWindow?: string;
  };
  structuredOutput: {
    jsonObject: 'native' | 'prompt';
    jsonSchema: 'native' | 'prompt';
    nativeWithTools: boolean;
  };
  nativeTools: Array<
    | 'web_search' | 'x_search' | 'web_fetch' | 'code_execution'
    | 'file_search' | 'remote_mcp' | 'computer_use' | 'hosted_shell'
    | 'apply_patch' | 'tool_search' | 'image_generation'
  >;
  nativeToolOptions: { remoteMcpApproval: boolean };
  responsesExtensions?: {
    asyncToolCalling: boolean;
    midTurnSteering: boolean;
    configurationUpdates: boolean;
    misalignmentMonitoring: boolean;
  };
  dataHandling: {
    promptCaching: 'none' | 'provider_managed';
    batch: 'none' | 'provider_retained';
    remoteMcp: 'none' | 'third_party';
  };
}

Model.features.promptCaching, batchAPI, and structuredOutput describe the model catalog. getAdvancedCapabilities() is more restrictive: it also verifies that the chosen adapter has a wire mapping and usage/result conversion for that model family.

Provider orientation matrix

This matrix is explanatory and can become stale as vendors change. Runtime code should branch on the capability object, never the table.

GPT-6 Astra Responses extensions

GPT-6 Astra introduced Responses protocol features that are not ordinary sampling options. The registry also exposes the verified subset on newer GPT-6 models such as GPT-6.1 Sol, GPT-6 Sol, and GPT-6 Luna. OneRingAI supports them through direct Responses calls and a connector-first WebSocket session. Check agent.getAdvancedCapabilities().responsesExtensions before exposing the features for a selected model.

GPT-6 sampling is effort-dependent. GPT-6 Sol and Luna accept temperature, top_p, and logprob controls only when the effective reasoning effort is none; omitting reasoning defaults to active reasoning and therefore rejects those controls. GPT-6 Astra and GPT-6.1 Sol do not support none at all. OneRingAI validates these combinations before sending the request.

Async function and custom tools

Set top-level async: true on a function or free-form custom tool definition. The model may continue reasoning or answer independent parts while your application runs the tool. When the work finishes, continue from the latest response and return the result with the original call_id:

import { Agent, ContentType, MessageRole, type Tool } from 'oneringai';

const agent = Agent.create({ connector: 'openai-main', model: 'gpt-6-astra' });
const astraInstructions = 'Produce a concise, sourced report.';
const astraAsyncTools = [{
  type: 'function',
  function: {
    name: 'slow_lookup',
    description: 'Run a long database lookup',
    parameters: {
      type: 'object',
      properties: { query: { type: 'string' } },
      required: ['query'],
      additionalProperties: false,
    },
    strict: true,
  },
  async: true,
}] satisfies Tool[];

const first = await agent.runDirect('Start the report and run the slow lookup.', {
  instructions: astraInstructions,
  tools: astraAsyncTools,
});

const call = first.output
  .flatMap((item) => item.type === 'message' ? item.content : [])
  .find((content) => content.type === ContentType.TOOL_USE);

if (call?.type === ContentType.TOOL_USE && call.async) {
  const result = await runSlowLookup(JSON.parse(call.arguments));
  const continued = await agent.runDirect([{
    type: 'function_call_output',
    call_id: call.id,
    output: JSON.stringify(result),
  }], {
    previousResponseId: first.id,
    instructions: astraInstructions,
    tools: astraAsyncTools,
  });
  console.log(continued.output_text);
}

Use custom_tool_call_output for a custom tool. Async calling is supported for application-executed function/custom tools, not provider-hosted tools or programmatic tool calling. Multi-agent requests with async tools must set parallel_tool_calls: false. This protocol flag is separate from OneRingAI's blocking: false: the latter controls the managed Agent.run() tool loop; protocol async calls use runDirect() / streamDirect() and host-managed continuations.

Change reasoning effort without breaking the cached prefix

Add a configuration_update input item before the next user message. Its reasoning effort persists for later continuations until replaced:

const next = await agent.runDirect([
  { type: 'configuration_update', reasoning: { effort: 'high' } },
  {
    type: 'message',
    role: MessageRole.USER,
    content: [{ type: 'input_text', text: 'Now prove the difficult part.' }],
  },
], {
  previousResponseId: first.id,
  instructions: astraInstructions,
  tools: astraAsyncTools,
});

Use this item only when the selected model reports responsesExtensions.configurationUpdates, and only in standard single-agent responses. Consecutive updates are rejected. It is incompatible with automatic context management and truncation: 'auto'; when explicit in-band compaction is needed, place one compaction_trigger as the final input item, then send a fresh configuration update after compaction. Direct responses retain encrypted compaction items in response.output in provider order, and direct streams emit StreamEventType.COMPACTION, so stateless applications can replay the canonical compacted state. The registry declares the exact efforts for each model. Astra accepts low, medium, high, xhigh, and max—not none or minimal. OneRingAI also rejects Astra requests containing temperature, top_p, top_logprobs, or include: ['message.output_text.logprobs'].

Mid-turn steering over WebSocket

Steering requires both a model that reports responsesExtensions.midTurnSteering and the Responses WebSocket transport:

import { OpenAIResponsesWebSocketSession } from 'oneringai';

const session = new OpenAIResponsesWebSocketSession({ connector: 'openai-main' });
session.on('error', console.error);
session.on('event', (event) => {
  if (event.type === 'response.created') console.log(event.response.id);
  if (event.type === 'response.steer.pending') {
    // Supply the required tool outputs once; never rerun already-started tools.
  }
});

await session.connect();
session.createResponse({
  model: 'gpt-6-astra',
  input: 'Draft the migration plan.',
  stream_id: 'planning',
});
session.steer('resp_active', 'Prioritize the zero-downtime path.');

Listen for response.steer.accepted, .pending, and .failed. Steering may end the active response as incomplete with stop reason steered, followed by an automatic successor. If .pending supplies required tool-result stubs, fill them from saved results and send one explicit response.create for that parent. The session deliberately exposes raw typed server events because the application owns pending tools, approvals, lanes, and reconnect ambiguity.

Misalignment monitoring and safety alerts

Pre-stream and streamed misalignment_policy_violation failures map to ProviderMisalignmentError, ahead of generic 403 authentication handling. Stop consequential actions, preserve request/tool records, and do not retry automatically. A safety.alert.created webhook can be correlated and retrieved with a project API key that has api.safety.alerts.read:

import {
  OpenAISafetyAPI,
  ProviderMisalignmentError,
} from 'oneringai';

try {
  await agent.runDirect('Continue the workflow');
} catch (error) {
  if (error instanceof ProviderMisalignmentError) {
    console.error(error.requestId, error.responseId);
  }
}

// After your webhook handler verifies safety.alert.created:
const alert = await new OpenAISafetyAPI('openai-main').retrieveAlert(alertId);

There is no generic resume operation after an alert; decide the next action in trusted host code.

Fast mode and EU data residency

Request Fast mode with vendorOptions: { serviceTier: 'fast' } and inspect response.usage.service_tier for the provider-reported tier. GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol, and GPT-6 Luna Fast mode are unavailable through the EU data residency endpoint. OneRingAI rejects explicit fast/priority requests for those models when the connector uses https://eu.api.openai.com/v1; a project-level default is still ultimately enforced by OpenAI because it is not visible in the request. Ultrafast requests are also rejected through the EU endpoint; Ultrafast is available only with global processing or US data residency.

Official references: latest model, async tool calling, mid-turn steering, reasoning configuration updates, misalignment monitoring, and Fast mode.

Current vendor API additions

The high-level classes below keep credentials connector-first while exposing new vendor APIs that do not fit the ordinary Agent.run() lifecycle.

OpenAI Decisions, GPT-Live, and custom voices

OpenAIDecisions performs ordered predicate, choice, and score evaluations. OpenAILiveSession is the primary GPT-Live WebSocket transport (separate from the existing Realtime voice-agent API), and OpenAIVoices creates prompt- or consent-based custom voices:

import {
  OpenAIDecisions,
  OpenAILiveSession,
  OpenAIVoices,
} from 'oneringai';

const decision = await new OpenAIDecisions('openai-main').create({
  model: 'gpt-6-luna',
  input: 'The deployment passed every required check.',
  questions: [{
    type: 'predicate',
    name: 'ready',
    instructions: 'Is the evidence sufficient to approve deployment?',
  }],
});

const voice = await new OpenAIVoices('openai-main').create({
  type: 'prompt',
  name: 'Release guide',
  prompt: 'Warm, precise, calm, and easy to understand.',
});

const live = new OpenAILiveSession({ connector: 'openai-main' });
live.on('event', (event) => console.log(event.type));
await live.connect({
  model: 'gpt-live-1',
  instructions: 'Help the user while narrating important actions.',
  audio: { output: { voice: { id: voice.id } } },
});

OpenAI custom-voice access is organization-gated. Prompt-designed voices are supported only by GPT-Live. To use a custom voice with TTS or Realtime, create it from an audio sample and a previously created consent-recording ID. The wrapper exposes the SDK's typed create request and returns the saved voice ID; credentials, organization, project, and base URL still come from the connector.

The host owns microphone capture, audio playback, reconnect policy, and clean session shutdown. Call live.close() during teardown.

Claude 5.5 thinking, compaction, and tool search

Claude Opus 5.5 always uses adaptive thinking; Sonnet 5.5 also supports between_tools at high or lower effort. OneRingAI rejects invalid mode/effort combinations before inference. Claude Opus 5 supports disabled only at high or lower; xhigh and max require adaptive thinking. The Anthropic adapter maps current web-search, web-fetch, and code-execution tool versions and supports deferred tools plus regex/BM25 tool search. Provider-owned tool-search blocks are stored as opaque provider state and replayed automatically by managed Agent loops.

const claude = Agent.create({
  connector: 'anthropic-main',
  model: 'claude-sonnet-5-5',
});

const result = await claude.run('Load only the tools needed to investigate this issue.', {
  thinking: { enabled: true, effort: 'high', mode: 'between_tools' },
  nativeTools: [{ capability: 'tool_search', options: { algorithm: 'bm25' } }],
  dataHandling: { allowProviderTools: true },
});

Gemini 3.8 Interactions, computer use, Live, and voices

Gemini 3.8 uses Interactions by default. Responses expose continuation_token; pass it back as continuationToken on a later direct or managed call. File search uses Google file-search store names through the normalized vectorStoreIds field. Computer-use actions arrive as ordinary tool calls: the host must inspect/approve the action, execute it, capture the new screen state, and send the result. Use runDirect() when the application needs to own that confirmation and screenshot loop.

import { GoogleLiveSession, GoogleVoices } from 'oneringai';

const gemini = Agent.create({
  connector: 'google-main',
  model: 'gemini-3.8-flash',
});

const firstGeminiTurn = await gemini.runDirect('Find the relevant contract.', {
  nativeTools: [{
    capability: 'file_search',
    options: { vectorStoreIds: ['fileSearchStores/contracts'] },
  }],
  dataHandling: { allowProviderTools: true },
});

const liveGemini = new GoogleLiveSession({
  connector: 'google-main',
  model: 'gemini-3.8-live',
});
liveGemini.on('interactionStatus', (status) => console.log(status));
await liveGemini.connect();

const googleVoices = new GoogleVoices('google-main');
const availableVoices = await googleVoices.list();

TextToSpeech.listVoices() is intentionally a deterministic built-in catalog; it does not make a remote custom-voice request. Use GoogleVoices.list() for that account-scoped catalog; create(), get(), and delete() expose the rest of the remote lifecycle. A custom voice returned as voice_…, voicekey_…, or voices/… can be passed as the normal voice string to Google TTS; the provider selects the custom-voice field and removes only the resource prefix.

Grok 4.7 hosted tools and reasoning telemetry

Grok text now uses a dedicated xAI adapter instead of inheriting OpenAI model assumptions. It maps xAI web search, X search, and hosted code execution, preserves reasoning through tool turns, and reports provider reasoning/native tool usage and cost telemetry:

const grok = Agent.create({ connector: 'xai-main', model: 'grok-4.7' });
const grokResult = await grok.run('Summarize current discussion and verify the calculation.', {
  thinking: { enabled: true, effort: 'high' },
  nativeTools: [
    { capability: 'x_search' },
    { capability: 'code_execution' },
  ],
  dataHandling: { allowProviderTools: true },
});

console.log(grokResult.usage.native_tool_calls, grokResult.usage.cost_usd_ticks);

Prompt caching

Policy contract

type PromptCachePolicy =
  | { mode: 'off' }
  | {
      mode: 'auto';
      ttl?: 'short' | 'extended';
      key?: string;
      breakpointMode?: 'implicit' | 'explicit';
      strict?: boolean;
    };

mode: 'auto' asks OneRingAI to use the provider's supported caching mechanism. The meaning of the optional fields is deliberately normalized:

Without strict, an unsupported cache request is removed and the call proceeds normally. If caching is supported but the requested TTL is not, OneRingAI removes only the TTL and preserves the remaining cache request/key.

const response = await agent.run(largeInput, {
  promptCache: {
    mode: 'auto',
    ttl: 'extended',
    key: 'contracts-workflow-v7',
    strict: true,
  },
  dataHandling: { allowProviderCaching: true },
});

console.log({
  cacheHitTokens: response.usage.cached_input_tokens,
  cacheWriteTokens: response.usage.cache_creation_input_tokens,
  cacheWriteByTtl: response.usage.cache_creation_details,
});

mode: 'off' has a precise, limited meaning: OneRingAI will not send provider cache controls for that call. OpenAI and Google can apply prompt caching implicitly to eligible ordinary requests; the library cannot turn that provider behavior off. allowProviderCaching: false or omission also blocks only caching explicitly requested/configured by this library. If policy requires a guarantee of no provider-side caching, enforce it through the provider account/product configuration or choose a provider/deployment with the required data posture.

Provider behavior

• Anthropic: auto emits an ephemeral cache control marker. extended selects the provider's extended TTL; usage can report both short- and extended-TTL cache creation.
• OpenAI: eligible prompts benefit from implicit caching. A normalized key and supported extended retention are mapped to Responses API controls. GPT-5.6+ can also mark supported input text/image/file blocks with promptCacheBreakpoint: true; set breakpointMode: 'explicit' to disable the automatic latest-message breakpoint. The default implicit mode keeps that automatic breakpoint and also honors caller-marked content blocks. The converter emits prompt_cache_breakpoint: { mode: 'explicit' } only when caching is enabled and the selected model reports explicitBreakpoints. Non-strict degradation removes both unsupported request controls and content-level markers.
• Google: eligible prompts may report implicit cached-content tokens. Explicit cached-content resource creation/lifecycle is not hidden behind this request policy and is not currently part of the normalized contract.

Prompt caching is prefix-oriented. Place stable instructions, policy text, schemas, and reference material before the changing user-specific suffix. OneRingAI preserves caller order and does not rewrite a prompt to improve hit rate.

Cache state belongs to the selected provider account/connector. OneRingAI does not implement a cross-user semantic response cache and does not reuse provider credentials across connectors.

Provider-hosted tools

Provider-hosted tools are different from ToolFunctions. A ToolFunction runs in your process under ToolManager; a native tool is sent to and executed by the selected LLM provider.

type NativeToolRequest =
  | { capability: 'web_search'; options?: Record<string, unknown> }
  | { capability: 'x_search'; options?: Record<string, unknown> }
  | { capability: 'web_fetch'; options?: Record<string, unknown> }
  | { capability: 'code_execution'; options?: Record<string, unknown> }
  | { capability: 'file_search'; options: { vectorStoreIds: string[]; [key: string]: unknown } }
  | {
      capability: 'remote_mcp';
      server: {
        name: string;
        url: string;
        authorization?: { connector: string; accountId?: string };
        allowedTools?: string[];
        requireApproval?: 'always' | 'never';
      };
      options?: Record<string, unknown>;
    }
  | { capability: 'computer_use'; options?: Record<string, unknown> }
  | { capability: 'hosted_shell'; options?: Record<string, unknown> }
  | { capability: 'apply_patch'; options?: Record<string, unknown> }
  | { capability: 'tool_search'; options?: Record<string, unknown> }
  | { capability: 'image_generation'; options?: Record<string, unknown> };

The options object is a provider-specific escape hatch for settings belonging to that normalized tool. Fixed normalized fields win over conflicting values in options.

Web search, fetch, and code execution

const response = await agent.run('Compare the latest three releases and cite sources', {
  nativeTools: [
    { capability: 'web_search', options: { max_uses: 4 } },
    { capability: 'web_fetch' },
    { capability: 'code_execution' },
  ],
  dataHandling: { allowProviderTools: true },
});

for (const item of response.output) {
  // Provider citations are retained on output-text annotations where available.
}
console.log(response.usage.native_tool_calls);
console.log(response.native_tool_events);

Unsupported tools fail with ProviderCapabilityNotSupportedError; OneRingAI never silently swaps a native tool for a client-side search, fetch, shell, or MCP implementation.

File search

The normalized file-search contract targets OpenAI vector stores and Gemini 3.8 Interactions file-search stores. Pass the provider's identifiers without rewriting them:

await agent.run('Find the termination clause', {
  nativeTools: [{
    capability: 'file_search',
    options: { vectorStoreIds: ['vs_contracts'] },
  }],
  dataHandling: { allowProviderTools: true },
});

At least one vectorStoreIds entry is required. Validation happens before provider execution.

Computer use and hosted coding tools

computer_use, hosted_shell, apply_patch, and tool_search are advertised only when the selected provider adapter has a concrete wire mapping. OpenAI executes a shell in a provider container by default. OpenAI computer use and apply patch are host-executed; tool search can be server- or client-executed. Gemini computer use likewise returns action calls for your application to approve and execute. Use runDirect() for host-executed native tools and send their results from trusted host code. Never treat native_tool_events as permission to perform a consequential local action.

OpenAI host continuations use typed top-level input items. Preserve the original call_id, honor all pending safety checks, and return the appropriate result item with the next direct request:

await agent.runDirect([{
  type: 'apply_patch_call_output',
  call_id: patchEvent.call_id!,
  status: 'completed',
  output: 'Patch applied and tests passed.',
}], {
  previousResponseId: response.id,
  nativeTools: [{ capability: 'apply_patch' }],
});

The public input union also includes computer_call_output, shell_call_output, and client-executed tool_search_output. Computer results carry a computer_screenshot with either an image_url (including a data URL) or an uploaded OpenAI file_id, plus any acknowledged safety checks. Shell results carry stdout/stderr plus an exit or timeout outcome; tool-search results carry the loaded tool definitions.

Remote MCP with connector-first authentication

Remote MCP never accepts a raw token in RunOptions. Reference a named connector instead:

Connector.create({
  name: 'crm-mcp',
  vendor: Vendor.Custom,
  auth: { type: 'api_key', apiKey: process.env.CRM_MCP_TOKEN! },
});

const response = await agent.run('Find Acme and summarize the open opportunities', {
  nativeTools: [{
    capability: 'remote_mcp',
    server: {
      name: 'crm',
      url: 'https://mcp.example.com',
      authorization: { connector: 'crm-mcp' },
      allowedTools: ['find_customer', 'list_opportunities'],
      requireApproval: 'never',
    },
  }],
  dataHandling: { allowThirdPartyTools: true },
});

The security sequence is intentional:

1. Check the concrete model supports remote_mcp.
2. Require allowThirdPartyTools: true.
3. Reject requireApproval: 'always' unless the adapter exposes a host-managed continuation.
4. Validate the server URL and require HTTPS.
5. Resolve the named connector through the agent's scoped IConnectorRegistry.
6. Fetch the token for the current userId/accountId immediately before request conversion.
7. Send the resolved token only in the provider wire request.

Consequently, a rejected request does not fetch or refresh the connector token. A scoped registry cannot fall back to a globally registered connector with the same name.

requireApproval: 'always' requires remoteMcpApproval: true; otherwise it is rejected before credential resolution. never is the normalized no-approval execution path. OpenAI's provider default requires approval, so omission is normalized to never until OneRingAI exposes the mcp_approval_response continuation lifecycle.

Native-tool results and streaming

Non-streaming responses may include:

interface NativeToolEvent {
  capability: string;
  id?: string;
  call_id?: string;
  status?: string;
  phase?: 'call' | 'output';
  details?: unknown;
  error?: { code?: string; message: string; details?: unknown };
}

native_tool_events is not a ToolManager execution queue and never grants authorization. For host-executed OpenAI tools, a phase: 'call' event exposes the provider payload needed by trusted host code; the host must validate and execute it, then submit the corresponding typed output item. Provider-native lifecycle and error details vary by vendor. Streaming reports detailed usage on the terminal response_complete event, but does not synthesize non-streaming native_tool_events for every intermediate provider event.

Asynchronous text batches

Batch inference is intended for large offline workloads where provider latency is acceptable in exchange for asynchronous processing and, where applicable, discounted token pricing.

Lifecycle

interface IAsyncTextBatchProvider<TOptions, TResponse> {
  submitBatch(
    requests: Array<{ customId: string; options: TOptions }>,
    options?: BatchSubmitOptions,
  ): Promise<BatchHandle>;
  getBatch(id: string): Promise<BatchHandle>;
  cancelBatch(id: string): Promise<BatchHandle>;
  getBatchResults(id: string): AsyncIterable<BatchTextResult<TResponse>>;
}

const batch = agent.getBatchProvider();
const caps = agent.getAdvancedCapabilities();
if (!caps.batch.supported || !batch) {
  throw new Error('No executable batch adapter for this provider/model');
}

// This Agent-bound surface injects the Agent's userId and scoped connector
// registry into every batch item before any remote-MCP credential is resolved.

const handle = await batch.submitBatch(
  [
    {
      customId: 'document-001',
      options: {
        model: agent.model,
        input: 'Summarize document 001',
        prompt_cache: { mode: 'auto', ttl: 'extended' },
        data_handling: { allowProviderCaching: true },
      },
    },
    {
      customId: 'document-002',
      options: {
        model: agent.model,
        input: 'Summarize document 002',
        response_format: {
          type: 'json_schema',
          json_schema: {
            name: 'summary',
            schema: {
              type: 'object',
              properties: { summary: { type: 'string' } },
              required: ['summary'],
              additionalProperties: false,
            },
          },
        },
      },
    },
  ],
  {
    completionWindow: caps.batch.completionWindow,
    metadata: { workflow: 'document-summaries' },
    dataHandling: { allowBatchRetention: true },
  },
);

await batchStore.save({
  provider: handle.provider,
  providerBatchId: handle.id,
  submittedAt: new Date().toISOString(),
  state: handle.state,
});

The host must persist the handle before treating submission as complete. OneRingAI does not include a job scheduler, polling daemon, database, webhook receiver, or dead-letter queue.

Normalized states

requestCounts can include total, processing, succeeded, failed, cancelled, and expired counts. Do not infer that state: 'completed' means every item succeeded.

Correlated results

customId is the stable application correlation key. It must be non-empty and unique within one submission.

for await (const item of batch.getBatchResults(handle.id)) {
  if (item.response) {
    await resultStore.saveSuccess(item.customId, {
      providerRequestId: item.providerRequestId,
      text: item.response.output_text,
      usage: item.response.usage,
    });
  } else {
    await resultStore.saveFailure(item.customId, item.error);
  }
}

Each BatchTextResult has either response or error. Item errors preserve normalized code, message, optional HTTP status, and provider-native diagnostic details. Result iteration can therefore produce successes and failures together.

Successful batch responses set usage.processing_mode = 'batch'. This lets callers feed actual usage directly into calculateCost.

Ambiguous submission safety

Batch creation is paid and is not safely retryable without reconciliation. A connection can fail after the provider accepted the job but before the client received its ID.

try {
  const handle = await batch.submitBatch(requests, {
    dataHandling: { allowBatchRetention: true },
  });
  await batchStore.save(handle);
} catch (error) {
  if (error instanceof ProviderAmbiguousOperationError) {
    await recoveryQueue.save({
      provider: error.providerName,
      operation: error.operation,
      recoveryMetadata: error.recoveryMetadata,
    });
    // Reconcile provider-side jobs/files before deciding whether to resubmit.
    throw error;
  }
  throw error;
}

OneRingAI disables SDK retries at known non-idempotent batch-creation boundaries. Do not wrap submitBatch in a generic retry policy. OpenAI recovery metadata includes the uploaded input-file ID when upload succeeded but batch creation became ambiguous; other providers retain submitted correlation IDs/model information where available.

Validation before submission

Before the paid boundary, OneRingAI validates:

• non-empty request list;
• unique, non-empty customId values;
• provider/model batch support;
• supported completion window;
• allowBatchRetention: true;
• provider request options, native tools, data policy, and context limits;
• structured-output schema shape and native-vs-prompt strategy;
• named remote-MCP credentials, after all non-secret checks pass.

Anthropic and OpenAI can accept model choices supported by their provider batch contract. Google's normalized inline batch requires one model for the whole submission. Provider-specific maximums are reported in capabilities.batch.maxRequests when known.

Data-handling policy

Advanced features can change where request data is processed or retained. OneRingAI therefore uses explicit, fail-closed permissions:

interface DataHandlingPolicy {
  allowProviderCaching?: boolean;
  allowBatchRetention?: boolean;
  allowProviderTools?: boolean;
  allowThirdPartyTools?: boolean;
}

These booleans authorize OneRingAI features; they are not a compliance engine. They do not override provider account settings, regional routing, abuse monitoring, ordinary API retention, or implicit prompt caching. Hosts should derive them from authenticated tenant policy rather than accepting them directly from untrusted end-user input.

A conservative host pattern is:

const policy: DataHandlingPolicy = {
  allowProviderCaching: tenant.aiPolicy.promptCaching,
  allowBatchRetention: tenant.aiPolicy.asyncRetention,
  allowProviderTools: tenant.aiPolicy.providerTools,
  allowThirdPartyTools: tenant.aiPolicy.thirdPartyMcp,
};

await agent.run(userPrompt, { nativeTools: approvedNativeTools, dataHandling: policy });

Detailed usage and cost estimation

LLMResponse.usage retains detailed provider telemetry without discarding the common token totals:

interface TokenUsage {
  input_tokens: number;
  output_tokens: number;
  total_tokens: number;
  cached_input_tokens?: number;
  cache_creation_input_tokens?: number;
  cache_creation_details?: {
    short_ttl_input_tokens?: number;
    extended_ttl_input_tokens?: number;
  };
  output_tokens_details?: { reasoning_tokens: number };
  native_tool_calls?: Record<string, number | undefined>;
  processing_mode?: 'interactive' | 'batch';
  service_tier?: string;
}

Fields are absent when a provider does not report them; absence means unknown, not zero. Streaming converters retain the same fields on the terminal response_complete event, and StreamState accumulates them across agent iterations.

Agent execution metrics expose cumulative cachedInputTokens, cacheCreationInputTokens, reasoningTokens, and nativeToolCalls. Structured-output formatting/repair generations contribute to both response usage and execution metrics.

const response = await agent.run('Analyze all records', options);
const usage = response.usage;

const estimatedCost = calculateCost(
  agent.model,
  usage.input_tokens,
  usage.output_tokens,
  {
    cachedInputTokens: usage.cached_input_tokens,
    cacheCreationInputTokens: usage.cache_creation_input_tokens,
    cacheCreationDetails: usage.cache_creation_details && {
      shortTtlInputTokens: usage.cache_creation_details.short_ttl_input_tokens,
      extendedTtlInputTokens: usage.cache_creation_details.extended_ttl_input_tokens,
    },
    processingMode: usage.processing_mode,
  },
);

console.log({ estimatedCost, metrics: agent.getMetrics() });

input_tokens is normalized to total processed input on every adapter; for Anthropic this includes the provider's uncached, cache-read, and cache-creation buckets. calculateCost splits those buckets instead of pricing the whole prompt at one rate and applies Anthropic's short/extended cache-write multipliers when the creation breakdown is supplied. processingMode: 'batch' applies the normalized batch discount only for registered OpenAI, Anthropic, and Google models that advertise batch support. It returns null when the model or the requested pricing mode is not represented safely in the registry.

Native-tool fees and non-token provider charges are reported as counts where possible but are not automatically added to token cost estimates.

Structured output interaction

Advanced capability discovery also describes how structured output composes with tools:

• jsonObject / jsonSchema report native or prompt for the concrete model.
• nativeWithTools reports whether the provider can enforce the native format while tools are offered in that request.
• run() can perform a final constrained, tool-free formatting pass when composition is unavailable.
• stream() cannot replace already-streamed text, so it logs a warning when tools prevent inline enforcement. Use run() when valid JSON is mandatory.
• Low-level and batch response_format requests are normalized before provider execution. An unsupported native schema becomes the prompt fallback; a missing/invalid schema fails before batch submission.

Non-streaming formatted responses expose:

response.structured_output_enforcement;
// 'native' — provider-native format enforcement
// 'prompt' — strict prompt/final formatting pass
// 'repair' — at least one bounded repair generation was required

Prompt fallback guarantees parseable JSON, not local JSON Schema validation. Validate output_parsed in application code when schema conformance is a hard boundary.

Errors and troubleshooting

ProviderCapabilityNotSupportedError

Thrown before provider execution when a strict or retention-sensitive request cannot be honored. The capability property is machine-readable enough to route common cases:

Prefer capability discovery to exception-driven feature detection. Exceptions remain important for race-free enforcement when a caller constructs options dynamically.

ProviderAmbiguousOperationError

This error means a paid/external mutation may have succeeded. It is intentionally distinct from a normal provider failure. Persist recoveryMetadata, stop automatic retries, and reconcile.

Common mistakes

• Caching request rejected: set dataHandling.allowProviderCaching: true, or use promptCache: { mode: 'off' } if no explicit cache request is desired.
• Extended TTL rejected: inspect promptCaching.ttlModes; remove strict to allow TTL downgrade.
• Native tool rejected: inspect nativeTools for the exact agent model, not just the provider.
• Remote MCP connector not found: ensure the named connector exists in the agent's scoped registry and the current userId/accountId can obtain a token.
• Remote MCP approval rejected: use requireApproval: 'never'; always requires an adapter that reports nativeToolOptions.remoteMcpApproval: true and exposes a host continuation.
• File search rejected: pass a non-empty vectorStoreIds array.
• Batch results unavailable: poll getBatch until a terminal state; persist the handle between polls.
• Batch duplicate work: never blindly retry an ambiguous submission.
• Streaming JSON invalid with tools: use non-streaming run() so the final tool-free formatting pass can execute.
• Usage detail missing: the provider did not report that field; do not assume zero for billing.

Production checklist

• Gate UI controls with getAdvancedCapabilities().
• Derive dataHandling from trusted tenant policy.
• Keep connector tokens out of prompts, logs, and public run options.
• Persist batch handles and correlation IDs durably.
• Treat ambiguous batch submissions as reconciliation tasks, not retries.
• Store every batch item result independently; partial failure is normal.
• Validate output_parsed against your schema when correctness depends on it.
• Record detailed usage and provider-native tool counts for billing/audit.
• Re-check provider account retention, residency, and tool terms independently of library policy.