Benchmark Luminal's compiler before it becomes a serving line item

Decode throughput and idle spend set the line item

The trade-off is decode throughput against idle spend. Fewer tokens per GPU-hour means more billable GPU-hours for the same traffic. A compiler earns its place in the serving budget only when it improves tok/s per GPU-hour on your own workload.

Software changes alone produced a 5x increase in tokens per watt for DeepSeek V4 within one month on the same hardware.

Luminal completed a $29.2M Series A. The round is dated Oct 7, 2026. Its Nov 2025 seed financing was $5.3M. Y Combinator, Felicis, Liquid 2 Ventures, Saga Ventures, and Palm Drive Capital are among Luminal's recorded investors. Its product combines an ML compiler with a serverless cloud for PyTorch model optimization and deployment. The compiler converts PyTorch models into optimized GPU code and is described as raising GPU utilization above 80%. Use the funding to size the pilot, not to approve the stack.

Vendor claims need a workload gate

Vendor benchmarks are useful, but they are not your p95. For an AI factory, tokens per watt—not GPU count, teraflops, or rack density—decides whether it can justify its grid commitment.

GB300 NVL72 delivered up to 25 times more tokens per watt than Hopper on DeepSeek V4 Pro, up to 20 times more on GLM5.1, and up to 10 times more on Kimi K2.6 in SemiAnalysis InferenceX benchmarks.

With Gemma 4 31B and a 100,000-token context, the Groq 3 LPX system reached 3,400 tokens per second for one user, marking a record for that model. Your p95 TPOT is the number that matters.

Nvidia's 35x higher inference throughput per megawatt and up to 10x revenue opportunity projections for the combined Vera Rubin NVL72 and Groq 3 LPX configuration were forward-looking projections rather than independently validated production results.

AutoPass, an LLM-based LLVM tuning system, produced average speedups, measured as geometric means, of 1.043x on x86-64 and 1.117x on ARM64 relative to LLVM -O3.

The benchmark gate is a fixed-workload checklist

Run the candidate against your current serving stack under a fixed workload. Record the numbers below.

• Same model, same quantization, same batch size.
• Record tok/s per GPU-hour.
• Record GPU utilization.
• Record decode tok/s.
• Record p95 TTFT and TPOT.
• Compute cost per 1M tokens, including idle or scale-to-zero cost.
• Compare against your current LLM serving stack.
• Reject if any metric regresses.

The first item removes confounders; the next three separate throughput from utilization; the latency pair protects users; the cost item turns performance into a line item. Keep the run artifacts: model hash, quantization, batch size, traffic shape, and the raw tok/s trace. Save the date, operator, and container.

Do not approve a compiler because it improves a single metric. A production serving stack has a mix of prefill and decode, a context-length distribution, and a concurrency curve.

If the compiler wins on short outputs but loses on long contexts, your p95 TPOT will show it. A larger batch can buy decode throughput while leaving cost per token flat. A different quantization invalidates the comparison. A new driver, container, or GPU pool adds migration cost.

Judge the compiler by the line item, the latency SLO, and the operating burden. Keep the gate in the change request.

Cost is the final number

A compiler can raise decode throughput and still lose on idle GPU cost if it keeps GPUs warm for traffic that arrives in bursts. Serverless inference can remove that idle cost, but only if scale-to-zero is real and cold starts do not break p95 TTFT.

Convert every run to inference cost per token. Include the GPU-hours, the idle time, the cold starts, and the power penalty. A typical AI factory converts only about 60% of grid electricity into useful AI compute, with the remaining 40% lost to cooling, power distribution, and rack-level inefficiencies. That loss changes the break-even point for the line item.