in-car LLM · on-device LLM automotive · cockpit AI chip · offline voice assistant car · edge LLM vehicle
In-Car LLMs: Why the Cockpit Assistant Is Moving On Board
Fanshi Semiconductor engineering team · Published

A cockpit assistant built on a language model has to answer in a tunnel, in an underground car park and on a road with no coverage. It also handles speech from everyone in the cabin, and a vehicle stays in service for ten years or more, during which a cloud model bills for every request. These pressures move inference into the car. The hardware then needs enough tokens per second for a multi-step assistant, a power budget of tens of watts, weights that can be updated over the air, and a supply that lasts as long as the vehicle programme.
Why not the cloud
A car loses its connection routinely: tunnels, car parks, rural roads, border crossings. An assistant that falls silent in those places teaches the driver not to rely on it. Running the model on board makes the response independent of coverage.
Two more reasons follow. Cabin audio is personal data from the driver and every passenger, and keeping it in the vehicle simplifies compliance in many markets. And a vehicle lives for a decade or more; a per-request cloud cost over that life is a liability the maker carries for every car sold.
What the assistant actually computes
| Task | Who consumes the output | What matters |
|---|---|---|
| Wake word and speech recognition | The system | Always on at very low power |
| Understanding a request and choosing actions | Software in the vehicle | Tokens per second; the driver waits for the action, not the text |
| Multi-step tasks such as navigation with stops and conditions | Software, over several model calls | Tokens per second across the whole chain |
| Spoken reply | The driver | A short time to first token |
| Reading the manual or summarizing messages | The driver | Context length and steady speed |
Most of the tokens in a multi-step request are consumed by software before the driver hears anything.
The hardware budget in a vehicle
A cockpit domain typically allows ten to seventy watts for this workload, with passive or shared cooling and wide temperature swings. That is enough for a mid-sized model on hardware built for it, and not enough for a data-centre accelerator.
A general-purpose processor in the cockpit is bound by memory bandwidth, which it also shares with displays and other functions. Decode speed cannot exceed bandwidth divided by model size. Near-memory compute raises that ceiling by placing the compute beside dedicated high-bandwidth memory.
Why updatable weights matter here
A vehicle receives software updates for years, and the model that ships at launch will not be the model in year five. Hardware that fixes the weights at manufacture is a poor fit for the main assistant. Near-memory compute keeps the weights in external memory, so a new model arrives over the air like any other update.
Fixed-weight chips still have a place in a vehicle for small, stable functions that must respond instantly at very low power.
Fanshi's Trident is a near-memory design for this class: models up to Qwen 3.5 35B within a 70 W system budget, with updatable weights. These are design targets, confirmed per project.
Questions for a supplier
- Which models are supported today, and how is a new model delivered to vehicles in the field?
- What is the tokens-per-second figure at the precision you will ship, measured or design target?
- What are the operating temperature range and the qualification plan?
- How long is supply committed, and on which process node?
- Is there an evaluation platform to run your own assistant before the hardware decision?
Frequently asked questions
Can a car run an LLM without an internet connection?
Yes. With an on-board inference chip the model runs entirely in the vehicle, so the assistant works in tunnels, car parks and areas without coverage.
How large a model can run in a car?
Within a budget of tens of watts, near-memory designs target models in the tens of billions of parameters. The practical size depends on the tokens per second the assistant needs.
How is the model in a car updated?
If the chip keeps weights in external memory, a new model is delivered over the air like other vehicle software.
Why do in-car assistants need high tokens per second?
A multi-step request involves several model calls whose output is consumed by software. The driver waits for all of them, so speed turns directly into a shorter wait.


