How many tokens per second can your device reach?
When weights sit in ordinary memory, every generated token requires reading all of them. That gives decode speed a ceiling you can calculate: memory bandwidth divided by model size.
- Model size
- 8.0 GB
- Decode ceiling
- 6.3 token/s
Fluid for chat
This is a theoretical ceiling. Real systems land lower, because the same memory serves everything else and the bandwidth figure is a specification peak.
Common devices compared (8-bit weights)
| Device (theoretical peak bandwidth) | 0.8B · 0.8 GB | 4B · 4 GB | 8B · 8 GB | 27B · 27 GB | 70B · 70 GB |
|---|---|---|---|---|---|
| Raspberry Pi 5 · LPDDR4X-426717 GB/s | 21 | 4.3 | 2.1 | 0.6 | 0.2 |
| PC · DDR5-5600 ×289.6 GB/s | 112 | 22 | 11 | 3.3 | 1.3 |
| Apple M2100 GB/s | 125 | 25 | 13 | 3.7 | 1.4 |
| NVIDIA Jetson AGX Orin204.8 GB/s | 256 | 51 | 26 | 7.6 | 2.9 |
| Apple M2 Max400 GB/s | 500 | 100 | 50 | 15 | 5.7 |
Bandwidth is the theoretical peak from published memory specifications. Values are tokens per second and are for order-of-magnitude comparison only.
Why the ceiling exists
A faster processor does not help: the compute is waiting for weights. This is the memory wall. The only way past it is to stop moving the weights, which is what compute-in-memory and near-memory compute do by placing the arithmetic where the weights are.
Frequently asked questions
How is the ceiling calculated?
Model size in GB is parameters in billions times bits per weight divided by 8. The decode ceiling in tokens per second is memory bandwidth in GB/s divided by model size in GB. Each token requires reading all the weights once, so speed cannot exceed this value.
Why are measured speeds lower than the ceiling?
Memory bandwidth is shared with the system and other tasks, the specification value is a theoretical peak, and attention caches also have to be read and written. Measured speed is usually a fraction of the ceiling.
How can a dedicated chip exceed this ceiling?
A compute-in-memory chip does the arithmetic inside the memory array. The weights never cross a bus, so the bandwidth ceiling no longer applies.
Have a specific project to discuss?
Send the target model, performance and power goals. We reply with a route and an evaluation plan.
