Skip to content

tokens per second · LLM inference speed · time to first token · memory bandwidth · on-device LLM

Tokens per Second Explained: How Fast Does an LLM Need to Be?

Fanshi Semiconductor engineering team · Published

Tokens per second is the rate at which a language model generates output. A person reads at roughly 5 to 7 tokens per second, so a chat answer feels fluid at 10 to 30. Reasoning models, agents and control loops need hundreds to thousands, because most of the tokens they generate are never read by a person. On a device, the ceiling is set by memory bandwidth divided by model size, which is why edge chips try to stop moving weights at all.

What a token is, and what the number measures

A token is a piece of text the model handles as one unit: a short word, part of a longer word, or a punctuation mark. A common rule of thumb for English is that one token is about three quarters of a word.

Tokens per second usually means decode speed: how quickly the model produces output once it has started. Two other numbers sit next to it. Time to first token is the wait before anything appears, and prompt processing speed is how fast the model reads its input. A system can be quick on one and slow on another.

How fast is fast enough?

UseWhat sets the barUseful speed
Reading a chat answerPeople read roughly 200 to 300 words a minute, about 5 to 7 tokens per second10 to 30 tokens/s feels fluid
Voice assistantSpeech runs near 150 words a minute, 3 to 4 tokens per second; the wait before the first word matters moreTens of tokens/s, with a short time to first token
Reasoning models and agentsHidden reasoning steps and tool calls multiply the tokens behind each visible answerHundreds to thousands of tokens/s
Robots and real-time controlThe model sits inside a control loop with a fixed time budget per decisionWhatever the loop requires, delivered consistently
Batch document processingTotal work done, not the experience of one userAggregate tokens/s per watt and per dollar

The pattern is simple. When a person reads the output, anything much above reading speed is wasted. When software consumes the output, speed turns directly into shorter task times, and thousands of tokens per second become useful.

Per-user speed and aggregate throughput

A vendor can quote tokens per second per user, meaning one request running alone, or the total across every request a server handles at once. The second number is always larger and says little about how responsive one conversation feels.

When you compare chips, make sure both figures use the same definition, the same model and the same precision. A small model at low precision will post a very different number from a large model at high precision on the same hardware.

On a device, memory bandwidth sets the ceiling

Generating one token requires reading every weight in the model. If the weights sit in ordinary memory, decode speed cannot exceed memory bandwidth divided by model size, however fast the processor is.

Take a device with 50 GB/s of memory bandwidth. An 8B-parameter model at 8-bit precision is about 8 GB, so the ceiling is around 6 tokens per second. A 0.8B model, about 0.8 GB, tops out near 60. Real systems land below the ceiling because the same memory serves everything else too.

This is the memory wall, and it is why edge inference chips use compute-in-memory or near-memory compute. If the weights never travel across a bus, the bandwidth limit stops applying.

Reading a spec sheet

  • Which model, and at what precision?
  • Per user or in aggregate, and at what batch size?
  • Measured on silicon, or a design target?
  • With what prompt and context length?
  • At what power, for the whole system and not only the chip?

Fanshi publishes its own figures on these terms: Forge targets 17,000 token/s on Qwen 3.5 0.8B in a watt-class device, and Trident targets 15,000 token/s on models up to Qwen 3.5 35B within a 70 W system budget. Both are design targets, and final specifications are confirmed per project.

Frequently asked questions

How many tokens per second can a person read?

Roughly 5 to 7. Adults read English at about 200 to 300 words a minute, and one token is about three quarters of a word.

What is a good tokens-per-second rate for a local LLM?

For chat, 10 to 30 tokens per second feels fluid. For agents, reasoning models and automated pipelines, where software consumes the output, more is better and hundreds to thousands are useful.

Is tokens per second the same as TOPS?

No. TOPS counts arithmetic operations per second and says nothing about memory. LLM decode speed is usually limited by how fast weights can be read, so a chip with high TOPS can still generate tokens slowly.

Why do small models run so much faster?

Each token requires reading all the weights, so a model one tenth the size can be read roughly ten times as often on the same memory bandwidth.

Bring the question to your project

Send the target model, performance and power goals. We reply with a route and an evaluation plan.

Request an evaluation kit