Find the Costing Layer: A Six-Step AI Infrastructure Map

Your AI feature has several cost centers: tokens, GPU hours, memory bandwidth, agent steps, eval runs, and security checks. Say a request costs $0.001 at 200 ms and 40% of that time is queueing: the model is not the problem. The layer that owns the queue is the layer you price first.

The bottleneck is a budget line

Teams usually budget the model and are surprised by the serving bill. A cheap per-token model can still cost more with long context, high concurrency, or retries. A cheap GPU hour can still waste money when it idles between requests. Split the workload into layers and give each layer a cost unit and a latency unit.

The six-layer test finds the costly layer

• Compute. Raw silicon and memory. Cost question: what do you pay per GPU hour or per memory-optimized server? Latency question: is bandwidth, not FLOPs, the constraint? Vendor category: managed GPU clusters or memory-focused inference silicon.
• Inference. Prompt in, tokens out. Cost question: what is the price per token for each open model? Latency question: what is tail latency under your concurrency? Vendor category: token-priced inference APIs or fine-tuning platforms.
• Data. Retrieval and context. Cost question: what do you pay for retrieval, storage, and context assembly? Latency question: how much time does retrieval add before the model starts? Vendor category: vector search, context pipelines, or hybrid local-cloud routing.
• Agent runtime. The loop that calls tools, models, and memory. Cost question: what do you pay per tool call, step, or session? Latency question: how much time does orchestration add across models? Vendor category: agent orchestration software or heterogeneous workload coordinators.
• Eval. Measurement of improvement or regression. Cost question: what do you pay per scored trace or regression run? Latency question: how long until a bad change is detected? Vendor category: eval harnesses, regression suites, or quality telemetry.
• Security. Controls for what leaves the device, what is logged, and what is retried. Cost question: what do you pay for posture checks, agent controls, and governance? Latency question: does policy enforcement add measurable time to the request path? Vendor category: AI security posture management, agent security, or AI governance tools.

Vendors sell different units: capacity, tokens, orchestration, confidence, control. The same chat feature can start as an inference problem, become a data problem when context grows, and become an agent problem when it calls tools.

Match the vendor to the metric

Compute is where throughput per dollar matters at your target latency. If memory bandwidth limits token generation, a cheaper GPU may still underperform. XCENA raised $135 million at a $570 million valuation on May 29, 2026, and says its MX1 chip processes data near DRAM and that a single server could potentially handle workloads that previously required 10 servers, describing inference as a problem of scaling memory rather than only compute. Test that claim with your own request mix before you buy.

Price inference by completed requests, not list price per token. Fireworks AI raised $1.5 billion and reached a $17.5 billion valuation. It processes more than 40 trillion tokens per day. It provides a cloud platform for fine-tuning open-source AI models and gives developers access to managed GPU clusters under usage-based pricing.

River AI's API is priced per 1 million tokens, and the rate changes according to the open model selected. Its seed/Series A round totaled $1.1 billion, with General Catalyst and AMP PBC leading and Nvidia, AMD Ventures, Y Combinator, and Temasek participating. River AI said an enterprise can run a complex reinforcement learning job in 15 to 20 minutes without an infrastructure team, while saving two to four times compared with closed-source options.

Data shows up in context cost and retrieval latency. Perplexity AI, valued at $20 billion, demonstrated at Computex 2026 that local models on Intel Core Ultra Series 3 devices decide which information stays on the device and which goes to cloud-based models. Perplexity said the hybrid inference feature had not reached users yet and was expected to arrive within weeks. If your feature handles private data, that routing decision is a data-layer cost, not a model-layer cost.

Agent runtime decides the bill through cost per completed task and added latency per step. Callosum's software, announced with Cerebras on August 20, 2026, breaks down and coordinates multi-agent workloads across different models and silicon to improve performance. If your feature calls tools, the runtime can become the expensive layer even when each model call is cheap.

Eval is measured by detection time. A regression that ships and is found later costs more than a slower pipeline that catches it. If you cannot measure quality per release, you are paying for inference without knowing whether it improved the product.

Security is measured by control overhead. CRN named 10 AI security startups in midyear 2026, covering AI security posture management, agent security, and AI governance. If policy checks add latency, price that latency. If they prevent a data leak, include the avoided incident in the budget.