A complete, authoritative catalog of every AI agent and model the platform uses. Understand how our multi-agent architecture provides resilience, cost-discipline, and a clear path to self-governed, enterprise-scale intelligence.
This is a complete, authoritative catalog of every AI agent and model the platform uses, where each one runs, why it was chosen, its input/output contract and tunable parameters, and how they are orchestrated together. It exists to make the platform's intelligence layer legible — both to operators reasoning about cost/quality/latency trade-offs and to evaluators assessing why this platform is a superior management- and executive-function layer for small-to-medium businesses (SMBs), with a clear path to self-governed, enterprise-scale intelligence.
Source of truth. Model pricing/registry lives in src/ai-pricing.ts (DEFAULT_TEXT_PRICING, DEFAULT_MEDIA_PRICING) and is projected into the runtime AGENT_MODELS array in src/agent-core.ts. Orchestration (routing, fallback, cheap-companion, embedding failover) lives in src/agent-core.ts and src/rag-utils.ts. This document is written to match that code; when the code changes, this file is updated alongside it.
Most products bolt a single LLM onto a UI. This platform instead treats models as a fleet of interchangeable, individually-specialized agents behind a uniform orchestration layer. That yields four properties an SMB cannot easily build alone but an executive layer demands:
1. Resilience
If any one provider's account is exhausted or a model is decommissioned, the platform automatically switches to a capable peer (chat, embeddings, and media all have failover). The business keeps running.
2. Cost Discipline
Cheap models do cheap work (recon, classification, short replies); expensive flagships are reserved for reasoning and long-horizon agentic tasks. Two-tier routing does this per-turn, invisibly.
3. Domain Fit
The right model for the job: search-grounded models for current-events questions, a finance-tuned embedder for money semantics, a long-horizon agentic model for multi-step build tasks.
4. Governance & Auditability
One pricing table, one usage ledger, one place to reason about spend, and per-message settings snapshots so every answer is explainable.
The enterprise scaling path (see §9) replaces or augments the hosted fleet with a self-governed, internally-hosted open-source LLM so an enterprise can expand its private context window across the whole org without sending data to third parties — the same orchestration layer, pointed at models it controls.
| Provider | Role in platform | Chat | Embeddings | Media |
|---|---|---|---|---|
| OpenAI | Primary chat + primary embeddings + transcription/vision | ✅ | ✅ (primary) | image/TTS |
| Anthropic (Claude) | Deep reasoning + long-horizon agentic work | ✅ | ❌ (no API) | — |
| Google (Gemini) | Balanced/cheap chat + embeddings fallback + media | ✅ | ✅ (fallback 1) | image/TTS/video/music |
| Perplexity (Sonar) | Live web-grounded answers | ✅ | ❌ (no API) | — |
| Voyage AI | Embeddings fallback + finance-domain embeddings | ❌ (no API) | ✅ (fallback 2) | — |
| Specialist media | fal.ai, Ideogram, Stability, Runway, Luma, Kling, MusicGPT | — | — | ✅ |
Why Anthropic and Perplexity have no embeddings: neither vendor ships an embeddings API — they are chat/completion only. Voyage AI (Anthropic's recommended embedding partner) fills that role in the failover chain.
All chat models are defined once in DEFAULT_TEXT_PRICING and exposed as AGENT_MODELS. Pricing is USD per 1M tokens (input / output), verified against each vendor's public pricing page.
| Model | In / Out (per 1M) | Cached in | Where used | Why |
|---|---|---|---|---|
| GPT-5.5 | $5.00 / $30.00 | $0.50 | Heavy chat, hardest writing/analysis | Flagship quality when depth matters |
| GPT-5.4 ⭐ default | $2.50 / $15.00 | $0.25 | /ai-chat default, SMS/Slack auto-reply | Best all-round quality-per-dollar |
| GPT-5.4 mini | $0.75 / $4.50 | $0.075 | Cheap-companion for recon rounds | Fast, cheap, good enough for bandwidth-bound work |
| GPT-5.4 nano | $0.20 / $1.25 | $0.02 | Companion's companion | Cheapest tier for trivial turns |
finish_reason: "length" signals truncation (auto-continued). max_completion_tokens = 4096 in /ai-chat.gpt-5.4 (DEFAULT_AGENT_MODEL).| Model | In / Out (per 1M) | Cached in | Where used | Why |
|---|---|---|---|---|
| Claude Fable 5 | $10.00 / $50.00 | $1.00 | Managed/agentic build tasks; advisor | Plans across stages, verifies its own work, works independently longer |
| Claude Opus 4.8 | $5.00 / $25.00 | $0.50 | Deep reasoning; Fable-5 safety fallback | Highest Claude reasoning tier |
| Claude Sonnet 5 | $2.00 / $10.00 | $0.20 | First autonomous fallback | Cheaper mid-tier flagship; great price/perf |
| Claude Sonnet 4.6 | $3.00 / $15.00 | $0.30 | Balanced reasoning | Proven mid-tier |
| Claude Haiku 4.5 | $1.00 / $5.00 | $0.10 | Cheap-companion for all Claude tiers | Fast recon/short-turn model |
system is a top-level field (not a message); stop_reason: "max_tokens" signals truncation.fable5_retention_accepted gate), and ships built-in safety classifiers — cyber/bio-misuse requests are blocked with an error (and not billed). An opt-in Fallback API transparently continues a blocked request on Claude Opus 4.8 at Opus pricing so a run is never lost.runAgent) uses Claude's long-horizon planning + tool-use for multi-round build tasks, with Fable 5 optionally acting as an advisor that reviews a cheaper worker model's plan every N rounds.| Model | In / Out (per 1M) | Cached in | Where used | Why |
|---|---|---|---|---|
| Gemini 3.5 Flash | $1.50 / $9.00 | $0.15 | High-throughput chat | Fast flagship-class Flash |
| Gemini 2.5 Pro | $1.25 / $10.00 | $0.125 | Deeper Gemini reasoning | Long-context strength |
| Gemini 2.5 Flash | $0.30 / $2.50 | $0.03 | Autonomous fallback; cheap-companion | Cheapest capable general model |
| Gemini 2.5 Flash Lite | $0.10 / $0.40 | $0.01 | Ultra-cheap tier | Bulk/low-stakes turns |
:generateContent) — systemInstruction + contents[] (roles user/model); finishReason: "MAX_TOKENS" signals truncation.| Model | In / Out (per 1M) | Where used | Why |
|---|---|---|---|
| Sonar | $1.00 / $1.00 | Current-events / citation answers | Cheap search-grounded |
| Sonar Pro | $3.00 / $15.00 | Higher-quality grounded answers | Better synthesis + citations |
| Sonar Reasoning | $1.00 / $5.00 | Grounded + reasoning | Search + chain-of-thought |
| Sonar Reasoning Pro | $2.00 / $8.00 | Best grounded reasoning | Top Sonar tier |
baseURL: https://api.perplexity.ai. A per-request web-search fee is billed separately by Perplexity (not modeled here).system role, so the platform relocates the user's own RAG / platform / finance context into the last user turn (an [INTERNAL CONTEXT] block) for Perplexity models, so search-grounded answers still see the user's private data. This applies in /ai-chat and in generateAssistantReply (SMS/Slack).The value is not the models; it's the coordination layer around them.
When a model call fails fatally (decommissioned, 404, capability-rejected, or the provider account is exhausted — credit/quota/billing), the platform switches to a capable peer instead of erroring at the user:
AUTONOMOUS_FALLBACK_CHAIN (claude-sonnet-5 → claude-sonnet-4-6 → gpt-5.4 → gemini-2.5-flash → claude-haiku-4-5 → gpt-5.4-mini) as a last resort.In the agentic runtime, when the previous turn only did read-only work (read_file, list_directory, search_files, SELECT/PRAGMA/EXPLAIN SQL), the next round is served by a cheap companion (CHEAP_COMPANION map, e.g. gpt-5.4 → gpt-5.4-mini, claude-*→claude-haiku-4-5, gemini-2.5-pro → gemini-2.5-flash). Recon is bandwidth-bound, not reasoning-bound — mutations and the first round always use the primary model.
gpt-4o → gpt-5.4, claude-3-opus → claude-opus-4-8).WRITING_ASSISTANT_INCOMPATIBLE_MODELS) are filtered from the /ai-chat model dropdown and rejected at the API with a suggested alternative.If a reply stops at the output-token limit, the platform stitches a continuation turn ("resume where you left off") so the user receives one complete answer.
/ai-chat and agent-chat capture the model + context toggles used for each message (in-memory only) so any answer's provenance is explainable in the UI.
A breakdown of where the AI fleet is actually used across the IteraOS platform.
| Surface | Entry point | Models | Purpose |
|---|---|---|---|
| AI Writing Assistant | POST /api/ai-chat | Any chat model (default GPT-5.4) | Draft/edit/brainstorm with RAG, platform self-awareness, finance, web, Bible, attachments, focused-document, and document-creation context |
| Autonomous / coding agent | runAgent (/api/agent/chat) | Primary + cheap-companion + Claude advisor | Multi-round tool-using build/ops tasks (read/write files, run SQL, etc.) with self-healing fallback |
| SMS auto-reply | generateAssistantReply (Textmunication MO webhook) | Default chat model | One-shot grounded reply to inbound SMS, using platform + conversation context |
| Slack assistant | generateAssistantReply (Slack events) | Default chat model | Read-only Q&A over the user's kanban/calendar/CRM data |
| Multi-agent comparison | multi-agent-comparison.ts | Imported ChatGPT/Claude/Perplexity/Gemini exports | Analyzes/compares external agent conversations (offline; no live calls) |
| RAG synthesis | /api/rag/synthesize | RAG_SYNTHESIS_MODEL (gpt-4o-mini) | Summarizes retrieved cross-domain recall into an answer |
| Intent detection | detectIntent (unified-agent-tools.ts) | Heuristic (no LLM) | Routes chat turns (file-gen intent, finance intent, etc.) cheaply before spending tokens |
RAG / recall is powered by vector embeddings, with the same failover philosophy as chat. All embeddings flow through embedTextsResolved() in src/rag-utils.ts.
| Order | Provider · Model | Dims | Notes |
|---|---|---|---|
| Primary | OpenAI text-embedding-3-small | 1536 | Existing corpus lives here |
| Fallback 1 | Gemini text-embedding-004 | 768 | batchEmbedContents |
| Fallback 2 | Voyage AI voyage-3.5 | 1024 | Anthropic's embedding partner |
Switching happens only on account exhaustion (never a transient blip, which would needlessly fragment the index).
Embeddings from different models occupy different vector spaces and are not comparable. The platform therefore: tags every row with the model that produced it; embeds the query once per model space present and scores each row only against the query vector of its own model; returns 0 from cosineSimilarity on any dimension mismatch; and gates the pgvector/HydraDB ANN fast-path to the primary space only. This makes multi-provider embeddings correct, not just redundant.
The ledger_entries RAG source pins embeddingModel: 'voyage-finance-2' so transaction recall understands money/merchant/category semantics — enabling semantic finance queries like "recurring AI subscriptions" or "payments related to cloud infrastructure" that plain SQL aggregation can't answer. If Voyage is unavailable, finance content indexes in the default space and remains searchable (just without the finance tuning).
Documents, group documents, emails, contacts, project cards, voice-memo transcripts, SMS, Slack, orders, AI-chat history, and finance transactions — all embedded and unified so an executive can ask one question across every domain.
Media generation is a pluggable fleet, priced per unit in DEFAULT_MEDIA_PRICING.
| Category | Providers / models | Unit | Notes |
|---|---|---|---|
| Image | OpenAI gpt-image, Flux 1.1 Pro (fal.ai / Replicate), Ideogram 3.0, Google Imagen 4, Gemini 2.5 Flash Image, Stability, self-hosted Stable Diffusion ($0) | per image | Local SD demonstrates the self-hosted cost floor |
| Audio (TTS) | OpenAI TTS / TTS-HD, Gemini TTS | per 1k chars | Long text is chunked |
| Video | Runway Gen-3/4, Luma Dream Machine, Kling, Google Veo 3.1 | per second | Async jobs w/ webhook + poller finalization |
| Music | MusicGPT, Lyria 3 (Vertex) | per track | Webhook cover-callback pipeline |
| Function | Model | Where | Why |
|---|---|---|---|
| Transcription | OpenAI whisper-1 | Sizzle reels (sizzle-reels.ts), voice memos | Audio → text for highlight selection & recall |
| Vision highlight selection | OpenAI GPT-4o vision | Sizzle reels | Frame sampling → clip picks |
| Document classification | Xenova/distilbert-base-uncased-mnli (local, zero-shot, $0) | Email automation | Distinguishes invoices/receipts from quotes/marketing without an API call — a self-hosted intelligence primitive |
| RAG answer synthesis | gpt-4o-mini (RAG_SYNTHESIS_MODEL) | /api/rag/synthesize | Cheap summarization of retrieved context |
The local DistilBERT classifier and self-hosted Stable Diffusion are early proof points of the platform's self-governed intelligence direction (§9): capable, private, zero-marginal-cost models running on infrastructure the operator controls.
The orchestration layer described above is provider-agnostic by design: chat, embeddings, and media are each resolved through a single indirection (AGENT_MODELS / embedTextsResolved() / the media registry). That is the hook for the enterprise tier.
The plan: for enterprise deployments, add a self-hosted, self-governed open-source LLM (and a self-hosted embedder) behind the same interfaces:
Because the platform already tags every embedding with its producing model and isolates vector spaces, an enterprise can stand up its own embedding space alongside the hosted ones without breaking existing recall — the migration is additive, not a rebuild.
| Env var | Powers | Absence behavior |
|---|---|---|
OPENAI_API_KEY | GPT-5.x chat, primary embeddings, Whisper, vision, TTS, RAG synthesis | Chat falls back to other providers; embeddings fall to Gemini/Voyage |
ANTHROPIC_API_KEY | Claude chat + agentic runtime | Claude models filtered out |
GEMINI_API_KEY | Gemini chat + embedding fallback 1 + media | Gemini filtered; embedding fallback 1 skipped |
PERPLEXITY_API_KEY | Sonar web-grounded chat | Sonar models filtered out |
VOYAGE_AI_API_PRIVATE_KEY | Embedding fallback 2 + finance embeddings (voyage-finance-2) | Finance uses default embedding space; chain ends at Gemini |
VOYAGE_EMBEDDING_MODEL (optional) | Override the general Voyage model | Defaults to voyage-3.5 |
RAG_SYNTHESIS_MODEL | RAG answer synthesis | Defaults to gpt-4o-mini |
RAG_AUTO_INDEX / _INTERVAL_MS / _LIMIT | Background auto-indexer | Off when unset |
Pricing for every model is DB-backed and admin-editable (ai_model_pricing), seeded from DEFAULT_TEXT_PRICING / DEFAULT_MEDIA_PRICING, with a weekly auto-refresh worker recording each run — a single governance surface for AI cost.
DEFAULT_TEXT_PRICING in src/ai-pricing.ts; add retired→current aliases in agent-core.ts; update the tables in §3 here.AUTONOMOUS_FALLBACK_CHAIN / CHEAP_COMPANION in agent-core.ts; update §4.embedTextsResolved() / set RagSource.embeddingModel; update §6 and RAG-AND-MODEL-ROUTING-FIXES.md.