Graphcore Develops Compact Model for Mobile AI Using 2.7-bit Weights

Mobile phones are well-suited for running multimodal AI since they integrate cameras, microphones, and sensors, preserving personal data on-device and enhancing responsiveness by reducing reliance on cloud services.

However, fitting a model like Llama 3.2 Vision 11B, which needs 21.3 GB for its bfloat16 weights alone, into mobile device constraints poses difficulties. Even after 4-bit quantization, the model still requires over 5 GB. To address this, developers must maintain model accuracy while ensuring that the hardware can efficiently decode any compressed weights.

Graphcore Research, in collaboration with Arm, has introduced Llama-Mobile, which utilizes two innovative strategies: S3D8, a new 2.7-bit weight format optimized for Arm CPUs, and quantization-aware training that does not require the original training data.

This method reduces the weight size of Llama 3.2 Vision 11B by over 80%, from 21.3 GB to 3.7 GB, while applying INT8 quantization to activations. The model achieves an average visual question-answering score of 66.1%, compared to 74.4% for the original bfloat16 configuration.

The S3D8 format encodes three weights in a single byte, with a shared 5-bit centroid index and an additional sign bit per weight. This compact format allows Arm’s Advanced SIMD (Single Instruction Multiple Data), known as Arm Neon, to decode multiple values simultaneously, leveraging existing INT8 matrix multiplication capabilities to manage the model's computational needs.

For the token-generation workload, S3D8 achieves a throughput of 33.8 GMAC/s, outperforming INT8 by approximately 28%. However, for more compute-intensive tasks like text prefill and vision encoding, S3D8’s performance aligns closely with INT8.

The model’s training incorporated quantization-aware distillation (QAD), allowing a quantized student model to learn from the original bfloat16 teacher model during fine-tuning. This employs a synthetic dataset generated from responses to images sourced from ImageNet, eliminating dependency on proprietary or inaccessible original training datasets.

Compared to methods like direct casting and GPTQ, which face challenges retaining accuracy at lower bit widths, QAT allows S3D8 to maintain a task score of 66.1% at about 2.7 bits per parameter, in contrast with significantly lower scores from other approaches.

Llama-Mobile indicates how an 11B parameter vision-language model can be effectively compressed for use on Arm mobile CPUs. The project showcases the advantages of adaptable weight formats that decouple storage from computation, enhancing performance without needing specialized low-bit arithmetic hardware. The successful design of S3D8 highlights the importance of aligning quantization formats with target hardware to maximize inference efficiency.

This research suggests a broader direction for practical edge AI deployment, emphasizing collaborative design across model architecture, numerical representation, and hardware capabilities. Future efforts may broaden these principles to accommodate various model structures and mobile devices, expanding the scope of private and responsive AI experiences.