Choose the model your serving stack can run

For an agent or copilot, the model layer trade-off is capability versus serving cost: a larger foundation model can answer harder prompts, but it raises token price, GPU demand, and latency. Mid-2025 enterprise LLM spend was $8.4B, with enterprise API spend led by Anthropic at 32%, OpenAI at 25%, Google at 20%, Meta's Llama at 9%, and DeepSeek at 1%. The spread is the arithmetic check before you pick a model.

Scale sets the minimum hardware

Treat a foundation model as a serving decision: pretraining scale sets the minimum hardware, context window, and prompt budget, not the price per token, queue depth, or tail latency. As of late summer 2026, enterprise agent platforms such as AWS Bedrock AgentCore, Google Gemini Enterprise, Microsoft Foundry, and Salesforce Agentforce 360 depend on one or more foundation models. That dependency pushes the model choice into the serving budget.

Pretraining scale also shapes the evaluation workload. Larger models may need longer prompts, more examples, and a bigger regression suite; smaller models may pass the same product test with a shorter prompt and a cheaper endpoint. Put prompt length, context window, and hardware size in the test plan.

Modality changes the serving path

Text-only models can share a simple request path. Image, audio, or video models add preprocessing, storage, and data transfer; a multimodal agent may need a separate worker pool, a larger GPU, and a different cache policy. At Build 2026, Microsoft Foundry was described as hosting more than 11,000 models, including Microsoft's MAI family, GPT-5.5, and Claude. A large catalog is a procurement problem, not a serving answer.

Text models can often reuse the same endpoint for chat, extraction, and classification. Multimodal models often need separate queues for image ingestion, audio transcription, and video frame sampling. Those queues change the cost model before tokens are billed.

Inference behavior decides cost

Models with similar benchmark scores can have different serving costs. One may emit long answers, call tools often, or need a large context window; another may finish in a few tokens and fit a small batch. Measure tokens per request, tool-call rate, tail latency, and GPU seconds per task.

As of mid-2026, Model Context Protocol, a tool-calling standard, had Tier 1 SDKs approaching 500 million monthly downloads, and both the TypeScript and Python SDKs had each surpassed one billion total downloads. Tooling adoption changes integration cost, not raw model quality.

Fine-tuned models change the serving cost. A fine-tuned small model can cut prompt length, reduce tool calls, and lower tail latency, but it can also lock you to a narrower task. If the fine-tune is for classification, extraction, or routing, it may belong on a cheap endpoint. If it is for open-ended generation, keep it behind the same serving controls as the base model.

The serving path comes before the model

Use the three-axis serving matrix: pretraining scale, modality, and inference behavior. Compare API, self-hosted, and hybrid paths against latency, data path, and monthly request volume.

• API serving: handles large pretraining scale and multimodal work, but token spend scales with usage and vendor pricing.
• Self-hosted: fits known inference behavior and stable modality, but you own GPUs, upgrades, and idle capacity.
• Hybrid: splits modality and inference behavior across paths, but routing adds policy and logging work.

A hybrid router needs a clear policy: route by task class, token budget, data sensitivity, and latency target.

AWS Bedrock AgentCore reached general availability in 2026 and had more than one million SDK downloads from customers such as Cox Automotive, Druva, Cohere Health, Ericsson, Sony, and Thomson Reuters. Adoption proves demand, but it does not prove your product can serve the same model at your latency target. Broadcom's VMware AI Factory can compress the bare-metal-to-first-model deployment window from weeks into hours. VMware Cloud Foundation customers can run more than 150 open-source and commercial models, including Nemotron 3, Gemma 4, cotomi, Qwen, and GLM 5.2. A shorter self-hosted path does not remove GPU cost, maintenance, or capacity planning.

When building an agent or copilot, start with API serving for the harder work and a small self-hosted model for routine calls. If your data path or latency target rules out external APIs, move the whole stack to self-hosted. If you need both, build a hybrid router and measure the split. Log the chosen path for every request, then adjust the split when cost or latency data changes.