Skip to content

run LLM without GPU · on-device LLM hardware · local LLM · NPU vs GPU · LLM ASIC

How to Run an LLM Without a GPU: Five Hardware Options for Devices

Fanshi Semiconductor engineering team · Published

A language model can run without a data-centre GPU on five kinds of hardware: a CPU with a quantized model, the NPU inside a phone or PC chip, an embedded GPU module, an FPGA, or a dedicated LLM chip. The first three share one bottleneck, memory bandwidth, because every token requires reading all the weights. A dedicated chip removes that bottleneck but only pays off when the model is settled and the product ships in volume.

Why look beyond the GPU

A data-centre GPU is the easy way to run a model and the wrong part for most devices. It draws hundreds of watts, costs more than the product it would go into, and needs cooling that a handheld, a vehicle module or an industrial camera does not have.

There are also reasons that have nothing to do with hardware. Many products must work without a network, and many deployments are not allowed to send data off the device. Both push inference onto the device itself.

Five options compared

OptionWhat limits speedChanging the modelTypical fit
CPU with a quantized modelMemory bandwidth and core countLoad a different filePrototypes, low volume, small models
NPU inside a phone or PC chipShared system memory, and which operations the vendor toolchain supportsAny model the toolchain can compileProducts already built on that chip
Embedded GPU moduleMemory bandwidth and the module power budgetAny model that fits in memoryRobots and gateways with tens of watts to spare
FPGALogic capacity and memory interfaces; high unit costReprogrammableProving an accelerator design; low volume
Dedicated LLM chip (compute-in-memory or model-on-chip)Fixed at design time; the weights do not moveDepends on the route: fixed at tape-out, reloaded at power-up, or updated in external memoryVolume devices with a settled model and a tight power budget

The first three share one bottleneck

On a CPU, an NPU or an embedded GPU the weights sit in ordinary memory and are read once for every token. Decode speed is capped by memory bandwidth divided by model size. Adding compute does not help once that cap is reached, which is why a chip with an impressive TOPS rating can still generate text slowly.

Smaller models and lower precision raise the cap, and that is where most on-device work goes today: quantize harder, pick a smaller model, accept the quality that results.

When a dedicated chip is worth it

A dedicated chip keeps the weights where the arithmetic happens, so the bandwidth cap no longer applies. The cost is a tape-out, which is a fixed expense, and a model that is partly or fully decided in advance.

It pays off when three things are true: the model is settled for the life of the product or changes only within one family, the product ships in enough volume to spread the fixed cost, and the power budget is too tight for a general-purpose part. If any of the three is missing, a module or an NPU is the better choice.

A practical path

  • Prototype on a CPU or a GPU module and measure the tokens per second your product actually needs.
  • Fix the model family and the precision you can accept.
  • Run the target model on an FPGA evaluation board to confirm speed and behaviour before committing.
  • Move to a dedicated chip, standard or custom, when volume and power budget justify it.

Fanshi supports the last two steps: FPGA evaluation boards with a model mapping toolchain, three standard edge chips, custom chips built around a customer model, and licensable inference IP for teams designing their own SoC.

Frequently asked questions

Can you run an LLM on a CPU?

Yes. Small quantized models run on ordinary CPUs. Speed is limited by memory bandwidth, so it falls quickly as the model grows.

Do I need an NPU to run an LLM on a device?

No. An NPU helps when its toolchain supports your model, but it shares system memory with everything else and meets the same bandwidth limit as a CPU.

What is the lowest-power way to run an LLM?

A chip that does not move weights at all. Compute-in-memory designs run the arithmetic inside the memory array, which removes the largest share of inference energy.

When does a custom LLM chip make sense?

When the model is settled, the product ships in volume and the power budget rules out general-purpose parts. Otherwise a module or an NPU is the better choice.

Bring the question to your project

Send the target model, performance and power goals. We reply with a route and an evaluation plan.

Request an evaluation kit