LLM inference ASIC · AI inference chip comparison · Taalas HC1 · Etched Sohu · Cerebras · Groq LPU
LLM Inference ASICs Compared: Five Chip Architectures in 2026
Fanshi Semiconductor engineering team · Published

LLM inference hardware now falls into five architectures: general-purpose GPUs, wafer-scale engines, language processing units, transformer-only ASICs and model-on-chip designs that fix one model into silicon. They differ in how much flexibility is given up for speed. Reported speeds for the same 8B model range from about 230 tokens per second per user on an NVIDIA H200 to about 17,000 on a hardwired Taalas HC1, but the figures mix vendor claims with independent measurements and are not directly comparable. All five are data-centre products; devices need a different class of chip.
Why inference is getting its own silicon
A GPU can train a model, run any model and render graphics. Inference needs far less: the same fixed sequence of matrix operations, repeated for every token. Each generated token requires reading all of the model's weights, so the limit is usually how fast weights reach the compute units, not how fast the arithmetic runs.
As inference bills overtake training bills, vendors are trading generality for speed and efficiency. The five architectures below sit along that line: the further down the table, the less the chip can do and the faster it does it.
The five architectures at a glance
| Architecture | Public example | Core idea | Publicly reported speed | Flexibility |
|---|---|---|---|---|
| General-purpose GPU | NVIDIA H200 | Programmable parallel compute with weights in high-bandwidth memory beside it | About 230 tokens/s per user on Llama 3.1 8B (Nvidia baseline, cited by heise) | Any model; training and inference |
| Wafer-scale engine | Cerebras WSE-3 | One chip the size of a wafer, with memory on the same silicon | About 1,900 tokens/s on Llama 3.1 8B (Artificial Analysis, cited by heise) | Any supported model |
| Language processing unit | Groq LPU | Inference-only processor with compiler-scheduled execution | About 600 tokens/s on Llama 3.1 8B (Artificial Analysis, cited by heise) | Any supported model |
| Transformer-only ASIC | Etched Sohu | The transformer datapath is fixed in silicon; weights stay in memory, so the model can change | Vendor claim: over 500,000 tokens/s in aggregate on Llama 70B for an 8-chip server; no independent benchmark published as of mid-2026 | Transformer models only |
| Model-on-chip | Taalas HC1 | The weights of one model are hardwired into the chip, so nothing is fetched at runtime | About 17,000 tokens/s per user on Llama 3.1 8B (vendor-reported) | One model per chip; adapters and context size configurable |
Figures are as reported by vendors or cited in press coverage during 2026, on different models, batch sizes and test conditions. Per-user speed and aggregate server throughput are different metrics. Sources are listed at the end.
What the numbers leave out
Per-user speed and aggregate throughput are not the same thing. A figure such as 17,000 tokens per second per user describes how fast one conversation runs. A figure such as 500,000 tokens per second for a server adds up many conversations running at once. Neither converts into the other without knowing the batch size.
Model size and precision change everything. An 8B model and a 70B model are different workloads, and a chip that stores weights at 3 or 4 bits is not doing the same job as one running 8-bit or 16-bit weights. Taalas has said its first silicon combines 3-bit and 6-bit parameters, with some loss of quality against GPU baselines.
Finally, check who measured. Some figures in the table come from an independent tracker, some from the vendor's own announcement, and one has no third-party confirmation at all. Power, price and availability are often missing from the announcement entirely.
All five are data-centre answers
Every chip in the table lives in a rack. A wafer-scale system dissipates kilowatts; the Taalas HC1 is designed for servers of around 2.5 kW; the others sit in standard accelerator servers. None of them fits in a translator, a car, a robot or an industrial camera.
Devices need the same idea, weights that do not move, at one watt to a few tens of watts and on a supply chain that can ship in volume. That is a sixth class: edge inference chips built on compute-in-memory and near-memory compute.
Fanshi Semiconductor works in this class on a mature SMIC 28 nm process. Forge uses MaskROM compute-in-memory for watt-class devices running Qwen 3.5 0.8B. Trident uses near-memory compute for models up to Qwen 3.5 35B on a 70 W system budget, with weights that can still be updated. Argus uses SRAM compute-in-memory for real-time vision. All Fanshi figures are design targets.
How to choose
- The model changes every few weeks, or you also train: stay on GPUs.
- You serve a few popular models to many users from a data centre: wafer-scale, LPU and transformer ASIC services are worth benchmarking on your own prompts.
- One settled model carries most of your traffic: a model-on-chip accelerator trades flexibility for the highest per-user speed.
- The model has to run inside a device, offline, on a fixed power budget: look at edge chips built on compute-in-memory or near-memory compute.
Frequently asked questions
What is an LLM inference ASIC?
An LLM inference ASIC is a chip designed only to run large language models, not to train them. By giving up the generality of a GPU it removes overhead, mainly the cost of moving weights from memory to the compute units, and delivers more tokens per second per watt.
Which LLM inference chip is fastest?
On per-user speed for a small model, the highest public figure is the Taalas HC1 at about 17,000 tokens per second on Llama 3.1 8B, reported by the company. Etched claims the highest aggregate server throughput. These figures use different models and metrics, and not all are independently verified.
Do inference ASICs replace GPUs?
No. Training, research and long-tail models still need general-purpose hardware. Inference ASICs fit where the model is settled and either the volume is very large or the power budget is very tight.
Can an LLM inference ASIC run inside a device?
The data-centre designs cannot: they draw hundreds of watts to kilowatts. Edge inference chips built on compute-in-memory or near-memory compute target one watt to a few tens of watts and are made for devices.



