Retrieval-Augmented Generation with Quantized Large Language Models: A Comparative Analysis

Large language models have demonstrated emergent intelligence and ability to handle a wide array of tasks. However, the reliability of these models in terms of factual accuracy and timely knowledge acquisition remains a challenge. Researchers explore the implementation of retrieval-augmented generation methods, aiming to enhance the authenticity and specificity in knowledge-intensive tasks. This paper discusses the practical application in industrial settings, particularly in assisting design personnel with navigating complex standards and quality manuals. Utilizing an open-source model with 6 billion parameters, the study employs quantization technology for local deployment, addressing computational challenges. The retrieval-augmented generation framework is analyzed, emphasizing the integration of document parsing, vector databases, and text embedding models. Experimental results compare models at different quantization levels, revealing trade-offs between response time, model size, and performance metrics. The findings suggest that 4-bit integer quantization is optimal for standard document retrieval and question-answering tasks, highlighting practical considerations for CPU inference. The paper concludes with insights into hyper-parameter tuning, model comparisons, and future optimizations for enhanced performance in edge device deployments of large language models.