NPU vs GPU · NPU for LLM · on-device LLM chip · edge AI accelerator · LLM ASIC
NPU vs GPU Module vs Dedicated Chip for On-Device LLMs
Fanshi Semiconductor engineering team · Published

An NPU, an embedded GPU module and a dedicated LLM chip can all run a language model inside a device, but they fail in different ways. An NPU is already in the product and costs nothing extra, yet it shares system memory and depends on the vendor toolchain. A GPU module runs any model that fits in its memory and draws tens of watts. A dedicated chip removes the memory bottleneck and uses the least power, but the model is decided in advance. The choice comes down to three questions: how settled the model is, how many units you ship, and how many watts you have.
Three parts, three bottlenecks
All three run the same arithmetic. What separates them is where the weights live while that arithmetic happens, because generating each token requires reading every weight in the model.
An NPU reads weights from the system memory it shares with the processor, the display and everything else. An embedded GPU module has its own faster memory, and pays for it in power. A dedicated LLM chip keeps the weights inside or right beside the compute, so they are not fetched at all.
Side by side
| NPU in a phone or PC chip | Embedded GPU module | Dedicated LLM chip | |
|---|---|---|---|
| Where the weights live | Shared system memory | The module's own memory | In or next to the compute array |
| What limits tokens/s | System memory bandwidth, and operator support in the toolchain | Memory bandwidth of the module | Set at design time; not bandwidth-bound |
| Typical power | Part of the host chip budget | Tens of watts | One watt to a few tens of watts, by model size |
| Changing the model | Any model the vendor toolchain compiles | Any model that fits in memory | Depends on the route: fixed at manufacture, reloaded at power-up, or updated in external memory |
| Up-front cost | None beyond the host chip | Per-unit module price | A tape-out, shared across volume |
| Fits best when | The product already uses that chip and a small model is enough | Volume is low or the model changes often | The model is settled, volume is high and power is tight |
Qualitative comparison. Actual figures depend on the specific part, the model and its precision.
Where an NPU runs out
An NPU is rated in TOPS, a count of arithmetic operations per second. That number says nothing about memory. For a language model the ceiling is memory bandwidth divided by model size, and an NPU draws on the same memory as the rest of the system.
The second limit is software. A model runs on an NPU only if the vendor toolchain supports every operation in it. New architectures often arrive before the toolchain does, and the unsupported parts fall back to the CPU.
For a small model on a product that already contains the chip, an NPU is the sensible first choice. It stops being enough when the model grows or the response has to be fast.
Where a GPU module runs out
A GPU module is the most flexible option: any model that fits in its memory will run, and the software is mature. The costs are power, heat and unit price. Tens of watts and a heatsink rule it out for handhelds and for most battery-powered products.
It is also still bound by memory bandwidth. A larger model on the same module generates proportionally fewer tokens per second.
When a dedicated chip is the right answer
A dedicated chip makes sense when three things hold together: the model is settled for the life of the product or changes only within one family, the product ships in enough volume to spread a tape-out, and the power budget is too tight for a general-purpose part.
How fixed the model is depends on the route. A MaskROM compute-in-memory chip fixes the weights at manufacture. An SRAM compute-in-memory chip loads them at power-up. A near-memory chip keeps them in external memory and updates them like firmware.
Fanshi builds chips on all three routes and licenses the inference cores as IP for teams designing their own SoC. Published figures are design targets.
A short decision path
- Is the model under about 1B parameters and is a few tokens per second acceptable? Start with the NPU you already have.
- Is the model still changing every few weeks, or is volume low? Use a GPU module.
- Is the model settled, the volume high and the budget a few watts? Evaluate a dedicated chip, starting on an FPGA board.
- Are you designing your own SoC? License an inference core instead of adding a second chip.
Frequently asked questions
Is an NPU faster than a GPU for LLMs?
Not generally. Both are limited by how fast weights can be read from memory. An NPU shares system memory, so for language models it usually reaches fewer tokens per second than a GPU module with dedicated memory.
Why does a high TOPS rating not mean fast text generation?
TOPS counts arithmetic operations and ignores memory. Language-model decode speed is capped by memory bandwidth divided by model size, so compute beyond that cap goes unused.
Can a dedicated LLM chip run a different model later?
It depends on the route. SRAM compute-in-memory and near-memory designs can load new weights. A MaskROM design fixes the weights at manufacture, so a new model needs new masks.
What is the difference between an NPU and an LLM inference IP core?
An NPU is a general neural-network accelerator already built into a chip. An inference IP core is a design block built for specific model architectures that a chip team integrates into its own SoC.



