LLM inference cost · cost per token · API vs self-hosted GPU · edge AI chip cost · on-device inference
LLM Inference Cost: API vs Cloud GPU vs Edge Chip Over 3 Years
Fanshi Semiconductor engineering team · Published

LLM inference cost comes in three forms. API usage is priced per million tokens and is an ongoing expense; a self-hosted cloud GPU amortises rental or depreciation to about $0.70 per million tokens; a dedicated edge chip is bought once, after which the marginal cost of each inference approaches zero. At 10 million tokens a day, three years of a frontier-model API costs about $192,000, nearly a thousand times an illustrative $210 chip.
Three ways to pay for inference
API: input and output tokens are priced separately. On 2026 list prices, blended at roughly 1:1 input to output, frontier models cost $6.9 to $17.5 per million tokens and low-cost models $0.07 to $2.2. There is no upfront investment and the elasticity is unlimited, but spend grows linearly with usage and never ends.
Self-hosted cloud GPU: priced by GPU-hour rental or by purchase depreciation. Spreading a 2026 median H100 rental rate across its throughput gives a blended inference cost of roughly $0.70 per million tokens. That is an order of magnitude cheaper than a frontier API, but it needs an operations team, carries utilisation risk and does not suit edge devices: the data still has to go to the cloud.
Dedicated edge chip: a one-time purchase. After that, each inference consumes only electricity and its marginal cost approaches zero. The constraints are that the chip determines the model size, and that device integration happens up front.
A worked example you can check
Assume a device processes 10 million tokens a day. At $17.5 per million tokens (GPT-5.5, blended), that is 10 × $17.5 = $175 a day, or $175 × 365 × 3 ≈ $191,600 over three years. A self-hosted GPU at $0.70 per million tokens comes to 10 × $0.70 × 1,095 days ≈ $7,700. An edge chip at an illustrative one-time cost of $210 stays at $210 over the same three years, plus electricity.
The same volume against other models over three years: Claude Opus 4.8 at $15 comes to 10 × $15 × 1,095 ≈ $164,000; Gemini 3 Pro at $6.9 to about $75,600; Qwen Max at $2.2 to about $24,100; DeepSeek V3 at $0.07 to about $770. Even the cheapest API costs more over three years than the chip.
Payback is a single division: chip cost ÷ monthly API spend. In the GPT-5.5 case, monthly spend is $175 × 30.4 ≈ $5,320, so the chip pays for itself in about a day ($210 ÷ $175 per day ≈ 1.2 days). Drop usage to 100,000 tokens a day and monthly API spend falls to about $53; payback still arrives within four months ($210 ÷ $53 ≈ 3.95).
The cost calculator on the Fanshi homepage uses exactly this method. It lets you vary the chip cost between $70 and $420 and drag the daily volume to see the result for your own workload.
Three variables beyond the bill
Latency: an API adds a network round trip; an edge chip does not. For voice interaction and real-time control, that difference is about experience rather than cost.
Data: an API needs the data in the cloud; edge inference does not. For air-gapped and restricted deployments, that decides whether the solution is allowed at all.
Model: an API can switch to the newest model at any time, while a MaskROM chip's model is fixed. If the business depends on steadily improving model capability, choose the near-memory or SRAM compute-in-memory route, or adopt a hybrid design in which the device runs a stable model and escalates complex tasks to the cloud.
When to move inference to the edge
Watch for three signals: the monthly API bill already exceeds the price of a chip; the device fleet is large enough that running a cloud inference cluster no longer pays; or the business simply cannot let data leave the device. Any one of them is reason enough to put an edge option into your evaluation.



