Local Mac Inference: 12 tok/s, 8k Prefill, Cost Break-Even

Costs

The trade-off is arithmetic, not taste: local inference costs amortized hardware plus power, API serving costs tokens, and local wins only when (machine cost ÷ expected tokens) is below API price per token. The target model is a 125B-parameter Qwen3.8-Flash-Next mixture-of-experts, 104 GB on disk at 4-bit, and it is streamed from SSD rather than fully loaded into memory. The stated hardware requirements are Apple Silicon, macOS 14 or later, about 110 GB of free disk, and a 512 GB Mac as the realistic minimum. Those requirements make disk the first constraint. Long prompts create a prefill bottleneck, with 8,000 tokens taking about a minute on a 48 GB Mac and over three minutes on a 16 GB Mac. If the machine cannot hold the weights, cannot prefill inside the user's patience, or cannot decode at a speed the product can tolerate, the token price is irrelevant. If it can, the token price becomes the only number that matters.

Estimate the machine's useful life in months, the power draw while serving, the expected token volume, and the API rate in dollars per million tokens. Compare local cost per month divided by expected tokens against API cost per token. The local line includes depreciation, storage, cooling, and engineering time. The API line includes token spend, rate limits, and the margin you are willing to pay for elasticity. If the local line is lower and the latency line is acceptable, local is a unit-economics choice, not a hobby.

The trap is treating the machine as free after purchase. A local server is a fixed cost that must be amortized across enough requests. Low traffic makes the per-token local cost explode. High traffic can make it look cheap, but only if the workload stays inside the machine's thermal, storage, and latency envelope. The break-even is a range where hardware cost, power, and token volume intersect.

The rule

• Disk: confirm the machine has room for the weights, logs, and operating system before you promise a deployment.
• Warm decode: check whether the target UX can tolerate the token rate on the actual machine.
• Prefill: test the longest prompt you expect, not the average prompt, because first-token latency is the visible wait.
• API compatibility: verify the endpoint works with your existing client, prompt templates, and error handling.
• Cost: compare amortized machine cost plus power against API tokens at the expected volume, then re-run the math when volume changes.

Models

Even if the memory tier looks adequate, the weight footprint can make a small drive unusable. For a product, the deployment target is not just a chip family; it is a storage class. A machine that can boot the model but has no room for weights, logs, and the operating system is not a viable server.

Quantization is the reason this is possible at all: a compressed model is smaller and faster to move, but it is also a lossy approximation of the original weights. The product question is not whether the model runs; it is whether the quality after quantization is acceptable for the feature. Make that a testable quality check: test the model on the feature's own prompts, not on a generic benchmark, and require the feature's own prompts to pass. If the feature needs long-context reasoning, tool use, or precise extraction, run those prompts as the acceptance set. If the feature is a lightweight assistant, a draft generator, or a private summarizer, the trade may be worth it.

Serving

Prefill is the part users wait for before the first token appears. It is the hardest latency to hide. A feature that sends large documents, long chat histories, or retrieved context will feel slow even if decode is acceptable. The product decision is to cap prompt length, summarize before sending, or route long-context requests to a faster path. If the SLA cannot absorb the prefill wait, local serving is not a drop-in replacement for a hosted endpoint.