offline LLM device · offline AI translator chip · on-device LLM handheld · low-power LLM chip · AI toy chip
Running an LLM Offline in a Handheld: Translators, Recorders, Earbuds and Toys
Fanshi Semiconductor engineering team · Published

A translator, a voice recorder, a pair of earbuds or a toy has one to three watts for the whole device and a battery that has to last the day. Running a language model there means a small model, typically under one billion parameters, on hardware that does not spend its power moving weights around. Because such products ship in volume with a model that stays the same for the life of the product, they are the clearest fit for a chip with the model fixed into it.
Why offline
These products are used on planes, abroad without data, in classrooms, in meetings and by children. A feature that stops working without a network is a feature that fails when it is needed. Offline operation also means speech is not sent to a server, which matters for recordings, meetings and anything used by a child.
There is a cost argument too. A cloud model bills for every request for as long as the device is in use. A model in the device costs nothing per request after the hardware is paid for.
The budget
| Constraint | Typical range | What it rules out |
|---|---|---|
| Device power | One to three watts for the whole product | GPU modules, which start in the tens of watts |
| Battery life | A full day of use | Any design that idles at high power |
| Bill of materials | A few dollars for the compute | Leading-edge processors with large memory |
| Size and heat | No fan, little thermal mass | Anything that needs active cooling |
| Response | Speech must start quickly | Hardware with a long time to first token |
Ranges are typical for the category and vary by product.
Why small models and fixed weights fit
Within one to three watts the practical model size is under about one billion parameters. Models of this size now handle translation, summarizing, dialogue and command understanding well enough to ship.
These products also have a property most others lack: the model is chosen once and stays for the life of the product. That makes it possible to cast the weights into the chip at manufacture. With MaskROM compute-in-memory the weights are part of the silicon and the arithmetic happens in the same array, so no energy goes into moving them and no separate memory chip is needed for the weights.
Fanshi's Forge is built on this route for Qwen 3.5 0.8B in a watt-class device, with a design target of 17,000 token/s. At that speed a full reply is generated in a fraction of a second and the chip spends most of its time idle, which is what a battery needs.
What speed buys you in a small device
A person hears and reads slowly, so it can seem that a handheld needs only a few tokens per second. Three things change that. A fast chip finishes and goes back to sleep, saving energy. Translation and summarizing work on whole passages, where the wait is the total length divided by the speed. And newer small models reason before they answer, which multiplies the tokens behind each reply.
What to check before choosing a chip
- Which model is fixed in the chip, and can a newer model in the same family reuse the design?
- Is the tokens-per-second figure measured or a design target, and at what precision?
- What is the power when idle, not only when generating?
- Is there an evaluation board to test your own prompts and languages?
- What is the minimum order quantity and the supply schedule?
Frequently asked questions
Can an LLM run fully offline on a translator?
Yes. Models under about one billion parameters run within a one-to-three-watt budget on hardware designed for it, with no network connection.
How large a model fits in a battery-powered device?
Within one to three watts, models under roughly one billion parameters are practical. Larger models need near-memory designs and tens of watts.
What happens if the model needs to change?
With a MaskROM chip a new model means a new mask set, although a newer model in the same family can reuse the architecture. If the model is expected to change in the field, an SRAM or near-memory design is the better fit.
Why not use the processor already in the device?
A general-purpose processor reads weights from shared memory for every token, which costs time and energy. A compute-in-memory chip does the arithmetic where the weights are stored.


