on-device LLM for robots · embodied AI hardware · robot inference chip · edge AI robotics · local LLM robot
On-Device LLMs for Robots: Why Embodied AI Needs Local Inference Hardware
Fanshi Semiconductor engineering team · Published

A robot that plans with a language or vision-language model cannot depend on a data centre: the network adds delay that varies from moment to moment, and connectivity is not guaranteed where robots work. Running the model on board turns inference speed into a fixed part of the control loop. The hardware then has to meet three constraints at once: tokens per second high enough for the planning rate, a power budget of roughly ten to seventy watts on the robot body, and behaviour that stays the same on every unit shipped.
Why the model has to be on the robot
A cloud model answers in a time that depends on the network. For a chat window that is an inconvenience. For a machine that is moving, a response that arrives late is a different response. Planning and perception that run on board take a known amount of time, which is what a control system needs.
There are two further reasons. Robots work in warehouses, factories, fields and homes where connectivity is uneven or not permitted. And camera and microphone data that never leaves the machine is far easier to deploy in private and regulated spaces.
What the workload looks like
| Layer | What it does | What it needs from hardware |
|---|---|---|
| Low-level control | Motor and balance loops at high rates | A real-time controller; not a language-model task |
| Perception | Vision and sensor fusion on every frame | Consistent frames per second at fixed latency |
| Task planning | A language or vision-language model turns a goal into steps | High tokens per second, because plans are consumed by software, not read by a person |
| Dialogue | Speaking with people nearby | A short time to first token; modest tokens per second |
A robot usually runs these layers at different rates on different hardware.
Why tokens per second matters more here than in chat
A person reads at about five to seven tokens per second, so a chat answer gains little above thirty. A planner is different. Its output is parsed by software, and every reasoning step and tool call adds tokens that nobody reads. The time to produce them is time the robot spends waiting.
If a plan takes 300 tokens, a chip generating 20 tokens per second needs fifteen seconds and a chip generating 3,000 needs a tenth of a second. The first robot pauses before each action; the second replans while it moves.
The power and bandwidth budget
A robot body typically has ten to seventy watts to spare for compute, with limited cooling. Within that budget a general-purpose module is capped by memory bandwidth: decode speed cannot exceed bandwidth divided by model size, however much compute it has.
This is where near-memory compute fits. Placing the compute beside high-bandwidth memory and streaming weights on chip raises the ceiling for larger models while the weights remain updatable. For real-time vision, SRAM compute-in-memory performs the arithmetic inside the memory array.
Fanshi targets this class with Trident, a near-memory design for models up to Qwen 3.5 35B within a 70 W system budget, and Argus, an SRAM compute-in-memory design for real-time vision. Both are design targets, confirmed per project.
Sizing the hardware
- Write down the planning rate: how many tokens per decision, and how often a decision is needed.
- Fix the model family and the precision you can accept.
- Set the compute power budget on the robot body, including cooling.
- Check that the weights can be updated in the field if the model is still improving.
- Run the target model on an evaluation board before committing to a chip.
Frequently asked questions
Can a robot use a cloud LLM?
It can for tasks that are not time-critical. For planning inside a control loop, network delay varies too much, and many sites have limited or restricted connectivity.
How many tokens per second does a robot need?
It depends on the tokens per decision and the decision rate. Because plans are consumed by software, hundreds to thousands of tokens per second translate directly into shorter pauses.
How much power can a robot spend on inference?
Robot bodies typically have about ten to seventy watts available for compute, with limited cooling. That rules out data-centre accelerators.
Can the model on a robot be updated?
Yes, if the chip holds the weights in SRAM or external memory. Near-memory and SRAM compute-in-memory designs both allow new weights to be loaded.


