Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Running a language model on a phone offers two obvious benefits: sensitive information can stay local, and an app can continue working when connectivity is poor. The less obvious challenge is producing a model that is accurate enough for a particular task while also fitting the performance, memory, energy, and latency limits of a particular phone.
In Mobile AI Factory, Evan Li, Krishna Deshpande, Wen Cheng, and Shyamnath Gollakota present an agentic system for automating that process. Its goal is not simply to find the most accurate small language model. It searches for a fine-tuned and quantized model that can actually run within a developer’s requirements on a named device.
Why mobile deployment needs more than model specifications
The authors’ measurements show why desktop assumptions do not transfer cleanly to phones. They evaluated model variants on a OnePlus 15, Galaxy S23, OnePlus 8 Pro, and Redmi 9A, using the llama.cpp and MNN inference engines and CPU or GPU backends where available.
Performance varied dramatically. In one example, a Qwen3-0.6B model at four-bit precision generated 72 tokens per second on the OnePlus 15 but only 1.7 tokens per second on the Redmi 9A. Even among more capable phones, the fastest backend could change. GPU prefill was much faster than CPU prefill on the OnePlus 15, but slower on the OnePlus 8 Pro. The best engine, precision, and backend therefore cannot be reliably inferred from parameter count or a phone’s specification sheet.
Quantization also affected accuracy, not just speed and memory. On a 151-class intent task, the authors report that MNN’s default four-bit export substantially reduced the score for some models, while a different four-bit recipe stayed close to the llama.cpp result. The deployment artifact—not merely the original weights—must be evaluated.
How the factory searches
Mobile AI Factory takes a task specification and a target phone as inputs. The specification can include data sources, an evaluation metric, an output verifier, and resource budgets such as latency, memory, and energy.
A phone profiler first measures candidate combinations and records throughput, peak memory, energy, engine, and backend. Configurations that fail or exceed the stated budgets are removed before expensive task-specific training begins. The profile is reusable across tasks for that phone.
The remaining pipeline combines deterministic rules with an orchestrator language model. The orchestrator chooses one intervention at a time:
- Hyperparameter tuning, including changes to LoRA rank, learning rate, weight decay, or training epochs.
- Mining additional real data from available sources.
- Generating targeted synthetic examples aimed at the model’s observed failure categories.
After each intervention, the system fine-tunes, quantizes, loads the artifact in the relevant inference engine, and scores it. Regressions are rolled back to the best model so far. The search begins with the smallest feasible model tier and moves upward only when progress stalls. Its accuracy target is based on a teacher model’s few-shot performance, with rules that allow the target to rise when it proves too easy.
What the evaluation found
The authors evaluated ten tasks, including biomedical and fine-grained entity extraction, calendar JSON extraction, function calling, intent classification, spam detection, emotion detection, grammatical error correction, and dialogue summarization. The candidate pool contained 27 combinations of nine base models and three precision levels.
On the authors’ frozen held-out sets, the factory’s best models beat the stronger of two five-shot frontier-model baselines on seven tasks under the default setup. The paper’s more important generalization check used unseen test sets: there, the specialized models won on five tasks, by 0.024 to 0.342, and tied on three more. The largest reported example is biomedical entity extraction, where a 517 MB model scored 0.828 while Claude scored 0.652 with five examples.
Most of the improvement came from the initial fine-tuning, but iterative search still mattered. Across tier-level comparisons, search improved on the first fine-tune by more than the authors’ 0.015 noise threshold in 34 of 42 pairs, with a median gain of 0.051. The benefit was larger for smaller models. In a biomedical extraction comparison, tiered search came within 0.03 of exhaustively training all 27 variants while using 19 times less training GPU time.
Phone-aware deployment produced a second result: the best configuration changed across devices. In 15 of 18 multi-phone deployments, different phones required different configurations. Without real measurements, Claude Opus 5.5 given detailed hardware and model specifications selected a configuration that missed the requirements in 25% of easy and moderate cases and 60% of hard cases. This does not show that an automated factory is universally better than expert engineering, but it does support the authors’ narrower claim that measurements reveal behavior specifications miss.
Limitations and practical lessons
The system still depends on developer work. Someone must provide a useful task definition, metric, and verifier. The verifier is especially important because it can reject malformed synthetic examples; the paper’s results suggest that synthetic data is less useful when few generated rows survive quality control.
The study also uses a limited candidate pool, LoRA fine-tuning, four Android phones, and CPU/GPU profiling under controlled, mostly idle conditions. It does not cover mobile NPUs, sustained concurrent workloads, changing thermal conditions, or every prompt length. A profile can be slow to create, and the final model may behave differently when the phone is sharing resources with other applications.
For developers, the practical message is straightforward: define deployment budgets before choosing a model, benchmark the exact quantized artifact on representative phones, and treat the inference engine and backend as part of the model configuration. Automated search can reduce repetitive experimentation, particularly when many tasks or devices must be supported. But it is best viewed as a measured engineering workflow—not as a replacement for a sound evaluation harness or realistic production testing.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.