edge LLM inference chip · edge AI chip · on-device LLM · AI accelerator selection · NPU vs ASIC
How to Choose an Edge LLM Inference Chip: 7 Questions
Fanshi Semiconductor engineering team · Published

To choose an edge inference chip, answer seven questions first: which model, how many parameters, how often it changes, what power budget, how fast decoding must be, whether data may leave the device, and how many units you will ship. Those answers largely decide between a general-purpose GPU or NPU and a dedicated compute-in-memory or near-memory ASIC.
1. Which model, and how many parameters
This is the first and hardest constraint. For language models, look at parameter count and context length; for vision models, input resolution and frame rate. A 0.5B to 1B language model can run at sub-watt power on a MaskROM compute-in-memory chip. A 7B to 35B model needs near-memory compute and a system budget of tens of watts. Real-time segmentation is measured in pixels processed per second, not in TOPS.
Do not be steered by TOPS. A general-purpose NPU's TOPS rating is its peak multiply-accumulate throughput, but LLM decoding is bandwidth-bound and real utilisation is often in the single-digit percent. The better question is: how many times per second can this chip read the entire set of weights?
2. How often the model changes
This decides whether MaskROM is an option. If the product is well defined, the model is thoroughly validated and it will not change within a year, MaskROM casts the weights into silicon for the lowest power and cost. If the model still changes every quarter, choose SRAM compute-in-memory or near-memory compute, so the weights can be loaded at power-up or held in external memory.
A practical approach is to stage it: bring the model up on an FPGA evaluation board with inference IP first, then decide on a MaskROM tape-out once the model is frozen.
3. Power budget
Handheld devices usually have 1 to 3 W for the whole system. In-vehicle systems and robots have 10 to 70 W, and industrial edge nodes can reach around 100 W. The power budget alone rules out most options: GPU modules typically start at 15 W before you account for cooling.
Compute-in-memory gets its power advantage from not moving weights. The effect is strongest during decoding, which is purely bandwidth-bound.
4. How fast it has to be
Voice interaction needs a time to first token within a few hundred milliseconds, and decoding faster than people read (roughly 10 to 20 token/s) is enough. Batch processing and multi-stream workloads want as much throughput as possible: thousands to tens of thousands of token/s are what bring the cost per token down. For vision, frame rate rules: defect inspection has to keep pace with the production line, and drone inspection with flight speed.
5. Whether data may leave the device
Air-gapped networks, restricted terminals, healthcare and industrial sites often forbid sending data to the cloud as a matter of policy. There, on-device inference is not a cost question but a question of whether the product can exist at all. Fully local inference means the device can run completely offline, and that guarantee comes from the architecture rather than from a policy.
6. Production volume
For a few thousand units, a general-purpose module plus software optimisation is the easiest path. At hundreds of thousands of units, a dedicated ASIC's unit cost and power advantage can move the system BOM by an order of magnitude. That holds especially for MaskROM: the masks are a one-time cost, so the higher the volume, the better the economics.
7. Who does the software
On a general-purpose platform you handle model conversion, quantization and operator support yourself; with a dedicated ASIC the chip vendor usually provides the model mapping. Ask directly: is the target model natively supported, what is the quantization scheme, can you get an evaluation board, and how soon can you have an engineering sample?
A one-page checklist
Fill in the seven answers and the route follows. Model of 1B or less, unchanged for a year, power of 3 W or less, high volume: MaskROM compute-in-memory. A fast-moving model that must be swapped in the field: SRAM compute-in-memory. A model of 7B or more with a budget of tens of watts: near-memory compute. A model that is not yet frozen: bring it up first on an FPGA evaluation board under an IP licence.



