The vocabulary of LLM inference chips.
Short, plain definitions of the terms that come up when you put a language model into silicon. Each one links to a longer explanation where we have written it.
- Token
- The unit of text a language model reads and writes: a short word, part of a longer word, or a punctuation mark. In English one token is about three quarters of a word. Read more →
- Tokens per second token/s, TPS
- The rate at which a model generates output. Quoted either per user, for one request running alone, or in aggregate across every request a server handles at once. Read more →
- Time to first token TTFT
- The wait between sending a prompt and seeing the first token of the answer. It matters more than decode speed for voice and other interactive uses. Read more →
- LLM inference
- Running a trained language model to produce output, as opposed to training it. Inference repeats the same fixed sequence of matrix operations for every token.
- LLM inference ASIC
- A chip designed only to run large language models. It gives up the generality of a GPU to deliver more tokens per second per watt. Read more →
- Model-on-chip hardwired LLM, model-specific ASIC
- A chip with the weights of one specific model fixed into the silicon at manufacture, so no weights are fetched at runtime. Very fast and efficient, at the cost of flexibility. Read more →
- Memory wall
- The gap between how fast processors compute and how fast memory can supply data. For LLMs it is the main limit, because every token requires reading all the weights. Read more →
- Memory bandwidth
- How many gigabytes per second can be read from memory. Decode speed cannot exceed memory bandwidth divided by model size when weights sit in ordinary memory. Read more →
- Compute-in-memory CIM, in-memory computing
- A circuit technique in which multiply-accumulate operations happen inside the memory array, so weights never leave the storage cells. Read more →
- MaskROM compute-in-memory
- Compute-in-memory in which the weights are cast into a read-only via array at the mask stage of manufacture. Lowest power and marginal cost; the model is fixed. Read more →
- SRAM compute-in-memory SRAM CIM
- Compute-in-memory built on SRAM cells. Weights are loaded at power-up, so the model can be replaced in the field. Read more →
- Near-memory compute NMC
- Compute units placed right next to high-bandwidth memory, with on-chip streaming to keep data movement short. Suits large models that are still changing. Read more →
- Edge inference on-device inference
- Running a model on the device where the data is produced instead of in a data centre. It works offline and keeps data on the device. Read more →
- NPU neural processing unit
- A neural-network accelerator inside a phone or PC chip. It shares system memory, so for LLMs it meets the same bandwidth limit as the CPU. Read more →
- TOPS
- Trillions of operations per second, a measure of raw arithmetic throughput. It says nothing about memory, so it predicts LLM decode speed poorly. Read more →
- Quantization
- Storing weights at lower precision, such as 8-bit or 4-bit integers instead of 16-bit floats. It shrinks the model and speeds up inference, with some loss of quality at very low precision.
- Tape-out
- The point at which a finished chip design is sent to the foundry for manufacture. It is a fixed cost, which is why dedicated chips need volume. Read more →
- Mask set
- The set of photomasks that define each layer of a chip. In a MaskROM design the model weights are part of the mask set. Read more →
- FPGA prototyping
- Running a chip design on reprogrammable logic before tape-out, to check function and performance with real models and data. Read more →
- Inference IP core
- A licensable block of chip design that runs a class of model architectures, integrated by a customer into their own ASIC, SoC or FPGA. Read more →
- Hybrid attention
- A model architecture that mixes linear-attention layers with full-attention layers, as in Qwen3.5 and later. It keeps long-context cost low while preserving accuracy. Read more →
- Linear attention
- An attention mechanism whose cost grows linearly with sequence length and which keeps a fixed-size state, suited to streaming and long contexts. Read more →
- Mixture of experts MoE
- A model in which only a subset of the weights, the selected experts, is active for each token. It lets a large model run within a limited bandwidth. Read more →
- Process node
- The manufacturing generation of a chip, such as 28 nm or 6 nm. Smaller nodes pack more transistors; mature nodes offer lower cost and more predictable supply.
- LPU language processing unit
- An inference-only processor whose execution is scheduled by the compiler. It runs any supported model and is used in data centres. Read more →
- Wafer-scale engine
- A single chip the size of a whole wafer, with compute and memory on the same silicon. It shortens the path between weights and compute at data-centre power levels. Read more →
- Decode speed
- How fast a model produces output tokens once it has started answering. It is what tokens per second usually refers to. Read more →
- Context window
- The amount of text, measured in tokens, that a model can take into account at once. Longer contexts need more working memory during inference.
- KV cache key-value cache
- Working memory that a transformer keeps for every token already processed, so it does not recompute them. It grows with context length. Read more →
- LoRA adapter low-rank adaptation
- A small set of extra weights that adjusts a model's behaviour without changing its base weights. Some fixed-weight chips support adapters for this reason. Read more →
- Embodied AI
- AI that controls a physical machine such as a robot. Planning models run on board because network delay varies and connectivity is not guaranteed. Read more →
- OTA model update over-the-air update
- Delivering new model weights to devices already in the field. It requires hardware that keeps weights in rewritable memory. Read more →
Further reading
LLM Inference ASICs Compared: Five Chip Architectures in 2026GPU, wafer-scale engine, LPU, transformer-only ASIC and model-on-chip: how the five LLM inference architectures differ, the tokens per second each has reported, and what those numbers leave out.
Tokens per Second Explained: How Fast Does an LLM Need to Be?What tokens per second measures, how it compares with reading and speaking speed, why per-user and aggregate figures differ, and how memory bandwidth caps it on a device.
What Is a Compute-in-Memory AI Chip? Three Routes ExplainedA compute-in-memory AI chip runs multiply-accumulate inside the memory array, so weights stop moving. How MaskROM, SRAM CIM and near-memory compute differ.
Have a specific project to discuss?
Send the target model, performance and power goals. We reply with a route and an evaluation plan.
