Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp
Take Qwen and Yitian 710 as examples

Abstract

This article optimizes the inference performance of the Qwen-1.8B[6] model by performing Int8 quantization, vectorizing some operators in llama.cpp[9], and modifying the compilation script to improve the compiler optimization level. On the Yitian 710 experimental platform, the prefill performance is increased by 1.6 times, the decoding performance is increased by 24 times, the memory usage is reduced to 1/5 of the original, and the accuracy loss is almost negligible.

I Introduction

Large Language Models (LLMs) have achieved remarkable performance in most natural language processing downstream tasks, such as text understanding, text generation, machine translation, and interactive Q&A. However, the billions or even trillions of model parameters pose significant challenges for efficient deployment of LLMs at the edge. With the growth rate of model parameters far outpacing the improvement in hardware performance, the academic and industrial communities are exploring software and hardware collaborative methods like model compression, dataflow optimization and operator invocation to deploy and run large models under the limited hardware conditions.

This article uses the default quantizer of llama.cpp to perform Int8 quantization on the Qwen-1.8B model, uses ARM’s NEON instructions to vectorize some operators in llama.cpp, and modifies the compilation script to improve the GCC compiler optimization level. In the test, the prefill performance increased from 86 token/s to 145 token/s, the decode performance increased from 2 token/s to 48 token/s, the memory usage was reduced from 10GiB to 2.3GiB, and tested using the piqa[7] data set in lm-evaluation-harne[8], the accuracy is only reduced by 0.0076. All codes in this article are open source to https://github.com/Longhao-Chen/Aicas2024

II Experiment platform

II-A Yitian 710 processor

Yitian 710 is the first Arm server chip released by T-Head. It is independently designed and developed by T-Head. It adopts an advanced architecture and has the characteristics of high energy efficiency, high bandwidth and is compatible with the Armv9 architecture.

In the Armv9 architecture, VDOT, MMLA and other instructions for Int8 calculations are provided, which can greatly accelerate Int8 type model inference.

II-B Qwen

Qwen is a LLM officially released by Alibaba Cloud Computing Co. Ltd., with parameter scales ranging from billions to trillions. The comprehensive performance of this model is well-rounded in mainstream benchmark evaluations.

In this article, we use Qwen 1.8B with 24 decode layers as an experimental model.

III Optimization

III-A Use the latest compiler

Old compilers do not support Integer Matrix Multiply intrinsics, specifically __ARM_FEATURE_MATMUL_INT8 is not defined[2]. To be able to use Integer Matrix Multiply intrinsics we need to use a newer compiler. We are using GCC 13.2.0. In this version, Integer Matrix Multiply intrinsics are fully supported.

III-B Use more compiler optimizations

Compiler optimization is an effective and convenient optimization method, and the GCC compiler supports multiple optimization levels. By default, llama.cpp uses -O3 optimization. To use higher-level optimization methods, you can use the command LLAMA_FAST=1 make -j8 to compile. This will enable a higher level of -Ofast optimization. Compared with -O3, -Ofast will enable -fallow-store-data-races, -fassociative-math, -fcx-limited-range, -fexcess-precision=fast, -ffinite-math-only, -freciprocal-math, -funsafe-math-optimizations, and disable -fsemantic-interposition, -fsigned-zeros, -fmath-errno, -ftrapping-math

Link Time Optimization (LTO) gives GCC the capability of dumping its internal representation (GIMPLE) to disk, so that all the different compilation units that make up a single executable can be optimized as a single module. This expands the scope of inter-procedural optimizations to encompass the whole program (or, rather, everything that is visible at link time)[4]. Therefore, for programs composed of multiple source files, enabling LTO can achieve better optimization. In GCC, we can enable LTO by passing -flto.

III-C Select the architecture of your host system

Modern cloud computing facilities usually use virtual machines or containers to isolate processes of different users. In these virtual machines or containers, programs often cannot obtain information about the underlying hardware. Therefore, at compile time, the host’s architecture needs to be explicitly specified to the compiler. For Yitian 710 processor, you can use -mcpu=neoverse-n2 -mtune=neoverse-n2 [1].

III-D Rewrite some operators using NEON

Arm Neon technology is an advanced Single Instruction Multiple Data (SIMD) architecture extension for the A-profile and R-profile processors. Neon technology is a packed SIMD architecture. Neon registers are considered as vectors of elements of the same data type, with Neon instructions operating on multiple elements simultaneously. Multiple data types are supported by the technology, including floating-point and integer operations [5]. For simple functions, the compiler’s Auto-vectorization can already generate high-performance NEON instructions, but complex functions require manual writing of NEON instructions. We rewrote functions ggml_fp16_to_fp32_row, ggml_fp32_to_fp16_row, ggml_compute_forward_norm_f32, ggml_compute_forward_rms_norm_f32, ggml_compute_fp16_to_fp32.

III-E Reduce unnecessary type conversions

The 8-bit quantization of llama.cpp treats 32 data as a group by default, and each group uses the same scaling factor. This scaling factor is saved as a float16 type[3]. Limited by processor instruction set, in each calculation, the corresponding scaling coefficient needs to be converted into data types, which will take up part of the time. If you directly use the float32 type to save the scaling factor, no conversion is required, which will save more time. But the memory usage will increase 5.88% .

IV Evaluate

We use the speed of the inference interface in the Python package transformers 4.38.2 as a baseline.

The llama.cpp item in the table is the unmodified original program.

TABLE I: Precision

Test items	Accuracy(piqa)
Baseline	0.7312
llama.cpp + fp16a	0.7252
llama.cpp + Int8a	0.7236
llama.cpp + fp16b	0.7252
llama.cpp + Int8b	0.7236
Ours	0.7236
aCompiler: gcc 9.4.0
bCompiler: gcc 13.2.0

TABLE II: Memory usage

Test items	Physical memory (MiB)	Virtual memory (MiB)
Baseline	10627	12228
llama.cpp + fp16a	3807	4204
llama.cpp + Int8a	2165	2562
llama.cpp + fp16b	3807	4205
llama.cpp + Int8b	2166	2563
Ours	2250	2649
aCompiler: gcc 9.4.0
bCompiler: gcc 13.2.0

TABLE III: Inference rate

Test items	Prefill rate (tokens/s)	Decode rate (tokens/s)
Baseline	86.79	2.07
llama.cpp + fp16a	113.63	24.05
llama.cpp + Int8a	38.53	24.36
llama.cpp + fp16b	116.85	23.51
llama.cpp + Int8b	98.55	37.88
Ours	145.86	48.36
aCompiler: gcc 9.4.0
bCompiler: gcc 13.2.0

V Discussion

Through experimental data, it can be seen that our solution greatly improves the inference performance of large language models with a low accuracy drop. Next, we can consider using the float type to save the scaling factor during quantization, which may result in smaller accuracy loss.