FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

FreeToken is an edge-native serving system for Mixture-of-Experts (MoE) models that treats a personal machine not as a small GPU but as a unified, elastic inference platform. Rather than commit to a fixed offloading strategy, it continuously maps computation and model state onto whatever resources a given machine actually has — the paper's two framing realities being that agent workloads keep changing their execution pattern, and that edge hardware exposes a heterogeneous balance of CPU, GPU, and memory that differs machine to machine.

The system co-designs the entire serving stack: model layout and loading, expert residency, CPU–GPU execution, agentic state reuse, and runtime memory management. It supports more than 20 MoE models and real coding and tool-using agents, spanning hardware from an 8GB laptop GPU to a single workstation GPU. The headline is the jump in what each machine can practically serve: a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on a single workstation GPU.

The stated thesis is that this turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. The system is released at an external project URL (not captured in this abstract). Note this source is the arXiv abstract only — no author list, figures, or evaluation details were included in the fetch.