run LLM without GPU · on-device LLM hardware · local LLM · NPU vs GPU · LLM ASIC
How to Run an LLM Without a GPU: Five Hardware Options for Devices
Fanshi Semiconductor engineering team · Published

A language model can run without a data-centre GPU on five kinds of hardware: a CPU with a quantized model, the NPU inside a phone or PC chip, an embedded GPU module, an FPGA, or a dedicated LLM chip. The first three share one bottleneck, memory bandwidth, because every token requires reading all the weights. A dedicated chip removes that bottleneck but only pays off when the model is settled and the product ships in volume.
Why look beyond the GPU
A data-centre GPU is the easy way to run a model and the wrong part for most devices. It draws hundreds of watts, costs more than the product it would go into, and needs cooling that a handheld, a vehicle module or an industrial camera does not have.
There are also reasons that have nothing to do with hardware. Many products must work without a network, and many deployments are not allowed to send data off the device. Both push inference onto the device itself.
Five options compared
| Option | What limits speed | Changing the model | Typical fit |
|---|---|---|---|
| CPU with a quantized model | Memory bandwidth and core count | Load a different file | Prototypes, low volume, small models |
| NPU inside a phone or PC chip | Shared system memory, and which operations the vendor toolchain supports | Any model the toolchain can compile | Products already built on that chip |
| Embedded GPU module | Memory bandwidth and the module power budget | Any model that fits in memory | Robots and gateways with tens of watts to spare |
| FPGA | Logic capacity and memory interfaces; high unit cost | Reprogrammable | Proving an accelerator design; low volume |
| Dedicated LLM chip (compute-in-memory or model-on-chip) | Fixed at design time; the weights do not move | Depends on the route: fixed at tape-out, reloaded at power-up, or updated in external memory | Volume devices with a settled model and a tight power budget |
The first three share one bottleneck
On a CPU, an NPU or an embedded GPU the weights sit in ordinary memory and are read once for every token. Decode speed is capped by memory bandwidth divided by model size. Adding compute does not help once that cap is reached, which is why a chip with an impressive TOPS rating can still generate text slowly.
Smaller models and lower precision raise the cap, and that is where most on-device work goes today: quantize harder, pick a smaller model, accept the quality that results.
When a dedicated chip is worth it
A dedicated chip keeps the weights where the arithmetic happens, so the bandwidth cap no longer applies. The cost is a tape-out, which is a fixed expense, and a model that is partly or fully decided in advance.
It pays off when three things are true: the model is settled for the life of the product or changes only within one family, the product ships in enough volume to spread the fixed cost, and the power budget is too tight for a general-purpose part. If any of the three is missing, a module or an NPU is the better choice.
A practical path
- Prototype on a CPU or a GPU module and measure the tokens per second your product actually needs.
- Fix the model family and the precision you can accept.
- Run the target model on an FPGA evaluation board to confirm speed and behaviour before committing.
- Move to a dedicated chip, standard or custom, when volume and power budget justify it.
Fanshi supports the last two steps: FPGA evaluation boards with a model mapping toolchain, three standard edge chips, custom chips built around a customer model, and licensable inference IP for teams designing their own SoC.
Frequently asked questions
Can you run an LLM on a CPU?
Yes. Small quantized models run on ordinary CPUs. Speed is limited by memory bandwidth, so it falls quickly as the model grows.
Do I need an NPU to run an LLM on a device?
No. An NPU helps when its toolchain supports your model, but it shares system memory with everything else and meets the same bandwidth limit as a CPU.
What is the lowest-power way to run an LLM?
A chip that does not move weights at all. Compute-in-memory designs run the arithmetic inside the memory array, which removes the largest share of inference energy.
When does a custom LLM chip make sense?
When the model is settled, the product ships in volume and the power budget rules out general-purpose parts. Otherwise a module or an NPU is the better choice.



