180W Inference Card $/Token Test Before GPU Racks

You are deciding whether to pilot an inference card before moving off GPU racks. The trade-off is power per token versus SLO risk. FuriosaAI says the 180W air-cooled RNGD accelerator matches a GPU's throughput while using half the power or less, so the power ratio is at least double at equal throughput before cooling, capex, and software risk. If it holds your serving SLO, the $/token math improves; if it breaks latency, it does not.

Use this five-point test before you commit to the card. Each item is a quick check, not a procurement memo.

• Baseline $/token: record current GPU $/token at your target request rate and token budget.
• Power-per-token: compare card power at equal throughput against the GPU baseline.
• vLLM compatibility: verify model loading, framework definitions, and a vLLM-compatible replacement.
• SLO/latency risk: run tail end-to-end latency against your ceiling.
• Supply-chain/TCO readiness: confirm production volume and commercial service before rollout.

The order follows failure mode. A weak baseline hides savings, a weak power check hides cooling, a weak runtime check breaks serving, a weak latency check breaks users, and a weak supply check breaks rollout. The test is cheap because it uses the workload you already run, not a vendor demo.

Power per token is the first gate

TSMC's 2026 run for the Seoul-based chipmaker is 20,000 RNGD accelerators. In LG's EXAONE model tests, RNGD hit 2.25x the reference GPU system's performance per watt, so the same token output needs less than half the reference power before cooling and software overhead. Power efficiency is the only metric that survives a rack power budget, because cooling, PDU headroom, and thermal limits all scale with watts, not with marketing claims. A different cooling path can shrink the performance-per-watt number.

Serving compatibility is the second gate

vLLM is the leading serving option for open-source LLMs at scale, but it does not solve every serving problem. vLLM deployments typically load Hugging Face-style checkpoints first, and some cases still call for NVIDIA-oriented quantization formats. vLLM has added compatibility layers so torch.compile, third-party accelerators, and older GPUs keep working. The checkpoint path is the first compatibility test, because a missing runtime path stops the benchmark before it starts.

FuriosaAI's deployment path handles Hugging Face model loading and PyTorch model definitions, plus a vLLM-compatible replacement. You can tune vLLM workloads for cost-effectiveness by choosing the best accelerator and configuration. Running gemma-3-27b-it on vLLM can require about 57 GB of VRAM at the minimum for a specific workload. Memory is the quiet latency risk: a model that fits at low concurrency can fail at production load.

Latency and TCO decide the switch

A vLLM serving workload can set a 10-second P99 end-to-end latency ceiling as its primary SLO. It can also target 100 requests per second and a 2,000-token budget. If the card misses either number, the $/token gain is not real. Tail latency is the user-facing constraint, and request rate is the revenue-facing constraint. A card that wins on power but loses on tail latency is not a production option.

FuriosaAI raised $620 million in a Series D at about $2.9 billion post-money. The Korean National Growth Fund put $280 million into the Series D and was the largest investor. LG Group runs EXAONE on RNGD, Upstage serves live search summaries on RNGD, and Samsung SDS started a commercial NPU service on RNGD in July. Capital matters because accelerator supply is a schedule problem. A chip that is in production and a chip that is in a pilot are different cost curves.

GPU economics are usually priced on compute, but the rack is priced on power. A lower-power inference accelerator changes the denominator in the $/token formula even when the chip price is higher, so the power-per-token check comes before the unit price check. Add labor and downtime to the cost line when the card needs a different rack layout.

Keep the GPU rack until the card clears all five, then reprice the rack.