Skip to content

offline LLM device · offline AI translator chip · on-device LLM handheld · low-power LLM chip · AI toy chip

Running an LLM Offline in a Handheld: Translators, Recorders, Earbuds and Toys

Fanshi Semiconductor engineering team · Published

A translator, a voice recorder, a pair of earbuds or a toy has one to three watts for the whole device and a battery that has to last the day. Running a language model there means a small model, typically under one billion parameters, on hardware that does not spend its power moving weights around. Because such products ship in volume with a model that stays the same for the life of the product, they are the clearest fit for a chip with the model fixed into it.

Why offline

These products are used on planes, abroad without data, in classrooms, in meetings and by children. A feature that stops working without a network is a feature that fails when it is needed. Offline operation also means speech is not sent to a server, which matters for recordings, meetings and anything used by a child.

There is a cost argument too. A cloud model bills for every request for as long as the device is in use. A model in the device costs nothing per request after the hardware is paid for.

The budget

ConstraintTypical rangeWhat it rules out
Device powerOne to three watts for the whole productGPU modules, which start in the tens of watts
Battery lifeA full day of useAny design that idles at high power
Bill of materialsA few dollars for the computeLeading-edge processors with large memory
Size and heatNo fan, little thermal massAnything that needs active cooling
ResponseSpeech must start quicklyHardware with a long time to first token

Ranges are typical for the category and vary by product.

Why small models and fixed weights fit

Within one to three watts the practical model size is under about one billion parameters. Models of this size now handle translation, summarizing, dialogue and command understanding well enough to ship.

These products also have a property most others lack: the model is chosen once and stays for the life of the product. That makes it possible to cast the weights into the chip at manufacture. With MaskROM compute-in-memory the weights are part of the silicon and the arithmetic happens in the same array, so no energy goes into moving them and no separate memory chip is needed for the weights.

Fanshi's Forge is built on this route for Qwen 3.5 0.8B in a watt-class device, with a design target of 17,000 token/s. At that speed a full reply is generated in a fraction of a second and the chip spends most of its time idle, which is what a battery needs.

What speed buys you in a small device

A person hears and reads slowly, so it can seem that a handheld needs only a few tokens per second. Three things change that. A fast chip finishes and goes back to sleep, saving energy. Translation and summarizing work on whole passages, where the wait is the total length divided by the speed. And newer small models reason before they answer, which multiplies the tokens behind each reply.

What to check before choosing a chip

  • Which model is fixed in the chip, and can a newer model in the same family reuse the design?
  • Is the tokens-per-second figure measured or a design target, and at what precision?
  • What is the power when idle, not only when generating?
  • Is there an evaluation board to test your own prompts and languages?
  • What is the minimum order quantity and the supply schedule?

Frequently asked questions

Can an LLM run fully offline on a translator?

Yes. Models under about one billion parameters run within a one-to-three-watt budget on hardware designed for it, with no network connection.

How large a model fits in a battery-powered device?

Within one to three watts, models under roughly one billion parameters are practical. Larger models need near-memory designs and tens of watts.

What happens if the model needs to change?

With a MaskROM chip a new model means a new mask set, although a newer model in the same family can reuse the architecture. If the model is expected to change in the field, an SRAM or near-memory design is the better fit.

Why not use the processor already in the device?

A general-purpose processor reads weights from shared memory for every token, which costs time and energy. A compute-in-memory chip does the arithmetic where the weights are stored.

Bring the question to your project

Send the target model, performance and power goals. We reply with a route and an evaluation plan.

Request an evaluation kit