Documentation/Managed model routing
v2.0.0-beta.1Snapshot bebdcd1Markdown ↗

Managed model routing

Managed model routing

Optional modelExecution.routing adds named alternative model profiles to the same Agent. Configure routes: { fast: { model: fastModel } } and a host selector such as select: ({ hints }) => hints.taskType === 'lookup' ? 'fast' : 'default'. Supply { routing: { taskType: 'lookup' } } to managed run()/stream(), force a route with { routing: { route: 'fast' } }, or bypass policy with { routing: null }. The implicit default uses the configured connector/model. Selection makes no hidden classifier call and precedes context preparation/input mutation.

One execution keeps its selected identity through tool iterations, recovery, repair, retries and compaction; public Agent/context defaults stay unchanged. Existing plugins and the one shared ToolManager remain in place. Route capacity, transfer authority and provider-bound state must be compatible. Another connector requires allowContextTransfer: true; opaque native/signed state retains its exact owner regardless of that permission. Authorized host routes can restore a native checkpoint without treating saved policy as authority.

sticky: 'phase' reuses a dispatched route only within the same explicitly shared ModelTask, Agent and latest nonempty phase. Async followups/suspension retain the original route/task. Helpers, direct calls, explicit maintenance and external SDK loops keep their existing configured owners. Failed provider calls do not replay on another model. Automatic delegation-based routing remains deferred.

Inspect response.routing, the existing stream completion event's optional routing, or getRoutingDiagnostics(). Use task snapshots for actual model costs, usage/unknowns and shared limits. Read the routing recipe for complete options, failure behavior, native support and persistence boundaries. Build then run npm run example:model-routing, example:routing-context and example:amos-routing; all default to offline. AMOS /routing configures JSON host rules and shows actual selections. Live examples/tests explicitly bill calls. See release status for package availability.

Ordinary event resumption of a suspended routed run inherits its output format, route and ModelTask. Explicit supported run options override the originating options; data-handling prohibitions cannot be relaxed. Local JSON parsing/repair and the host completion predicate both run on the resumed final candidate. Local coercion guarantees parseability, not full JSON Schema validation. Streaming retains its existing no-repair boundary. See npm run example:routing-suspension. Failed event retries retain the original task and limits; they cannot obtain a fresh allowance. Explicit host agent.cancel(reason) abandons pending execution authority without undoing its action. Structured suspension finalizes before saving or publishing success. A formatting failure saves the registered wait/correlation with optional metadata.suspensionFinalization.status: 'failed'; it never authorizes repeating the wait. Storage failure is explicit and may require preserving the in-memory observation. See the recipe's recovery boundaries and the example's --failed-finalization mode. Task budgets and authorization remain process-local. Saved opaque state keeps exact-owner dispatch checks even when the restoring Agent omits routing. Portable legacy history still permits supported model hooks. This includes signed/provider-owned items first produced by JSON repair: ordered replay state and its admitted identity survive response replacement and save/load. For Google GenerateContent/Vertex, explicit thinking: { enabled: false } now sends a zero thinking budget on supported Gemini 2.5 Flash/Flash-Lite models. Omitted defaults and explicit vendor thinking overrides retain their behavior; this does not add a disable capability to Pro, 3.x or Interactions. See the routing recipe for the exact supported scope and actual pilot cost evidence. A native-capable alternative may be selected with an unsupported default provider; idle compactContext() still uses its configured owner and requires its adapter.

Documentation snapshot · 2026-11-02Suggest a correction ↗

Search documentation

Type to search the v2 documentation.