Sign up to save your podcasts
Or
Share Microscaling Quantization for Large Language Models
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Microscaling Quantization for Large Language Models
Double paper review for modern quantization techniques.
These two academic papers address the crucial challenge of quantizing Large Language Models (LLMs) to reduce their computational and storage demands while preserving accuracy. The first source, "Post Training Quantization of Large Language Models with Microscaling Formats," investigates combining SmoothQuant, AWQ, and GPTQ post-training quantization techniques and extends their applicability to microscaling (MX) formats, demonstrating improved perplexity and accuracy, especially at lower bit-widths. The second source, "CROSSQUANT: A POST-TRAINING QUANTIZATION METHOD WITH SMALLER QUANTIZATION KERNEL FOR PRECISE LARGE LANGUAGE MODEL COMPRESSION," introduces the novel concept of the "quantization kernel"—elements quantized to zero—and proposes CrossQuant, a method that significantly reduces this kernel to maintain precision during activation quantization, outperforming other baselines across various LLMs and tasks. Both sources aim to enhance the efficiency and deployability of LLMs through advanced quantization strategies.
View all episodes
By mcgrofDouble paper review for modern quantization techniques.
These two academic papers address the crucial challenge of quantizing Large Language Models (LLMs) to reduce their computational and storage demands while preserving accuracy. The first source, "Post Training Quantization of Large Language Models with Microscaling Formats," investigates combining SmoothQuant, AWQ, and GPTQ post-training quantization techniques and extends their applicability to microscaling (MX) formats, demonstrating improved perplexity and accuracy, especially at lower bit-widths. The second source, "CROSSQUANT: A POST-TRAINING QUANTIZATION METHOD WITH SMALLER QUANTIZATION KERNEL FOR PRECISE LARGE LANGUAGE MODEL COMPRESSION," introduces the novel concept of the "quantization kernel"—elements quantized to zero—and proposes CrossQuant, a method that significantly reduces this kernel to maintain precision during activation quantization, outperforming other baselines across various LLMs and tasks. Both sources aim to enhance the efficiency and deployability of LLMs through advanced quantization strategies.