Sign up to save your podcasts
Or
Share SFMP Search-Free Mixed-Precision LLM Quantization
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
SFMP Search-Free Mixed-Precision LLM Quantization
This episode explores SFMP, a post-training, weight-only quantization method for large language models that targets awkward memory budgets like 3.25 or 3.75 bits per weight, where standard 3-bit or 4-bit schemes are a poor fit. It explains the core ideas behind mixed-precision quantization, including salience scoring with diagonal Fisher information, block-wise allocation, and why hardware constraints make many fine-grained methods difficult to deploy in practice. The discussion centers on SFMP’s main claim: instead of running an expensive search over bit allocations, it ranks blocks by importance and assigns only the two neighboring integer precisions, such as giving the most salient 25 percent of blocks 4-bit weights and the rest 3-bit for a 3.25 BPW target. Listeners would find it interesting because it frames quantization not as a narrow accuracy trick, but as a concrete engineering problem about trading off model quality, memory limits, and real serving-system compatibility.
Sources:
1. SFMP Search-Free Mixed-Precision LLM Quantization
https://arxiv.org/pdf/2602.01027
2. HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019
https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+AWare+Quantization+of+Neural+Networks+with+Mixed-Precision
3. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2023
https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers
4. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 2024
https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration
5. AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models — Sangjun Lee, Seung-taek Woo, Jungyu Jin, Changhun Lee, Eunhyeok Park, 2025
https://scholar.google.com/scholar?q=AMQ%3A+Enabling+AutoML+for+Mixed-precision+Weight-Only+Quantization+of+Large+Language+Models
6. BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments — Xinghao Wang, Pengyu Wang, Bo Wang, Dong Zhang, Yunhua Zhou, Xipeng Qiu, 2024
https://scholar.google.com/scholar?q=BitStack%3A+Fine-Grained+Size+Control+for+Compressed+Large+Language+Models+in+Variable+Memory+Environments
7. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models — Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, Xiaojuan Qi, 2024
https://scholar.google.com/scholar?q=SliM-LLM%3A+Salience-Driven+Mixed-Precision+Quantization+for+Large+Language+Models
8. FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs — Xilong Xie, Liang Wang, Limin Xiao, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu, 2025
https://scholar.google.com/scholar?q=FineQ%3A+Software-Hardware+Co-Design+for+Low-Bit+Fine-Grained+Mixed-Precision+Quantization+of+LLMs
9. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2024
https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge
10. MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design — Zhen Zheng et al., 2024
https://arxiv.org/abs/2412.14590
11. SpinQuant: LLM quantization with learned rotations — Zechun Liu et al., 2024
https://arxiv.org/abs/2405.16406
12. DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization — Yuantian Shao et al., 2025
https://arxiv.org/abs/2511.04063
13. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Coleman Hooper et al., 2024
https://arxiv.org/abs/2401.18079
14. QAQ: Quality Adaptive Quantization for LLM KV Cache — Shichen Dong et al., 2024
https://arxiv.org/abs/2403.04643
15. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3
16. AI Post Transformers: PackKV Lossy Compression for KV Caches — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-04-packkv-lossy-compression-for-kv-caches-b37bce.mp3
17. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3
18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
View all episodes
By mcgrofThis episode explores SFMP, a post-training, weight-only quantization method for large language models that targets awkward memory budgets like 3.25 or 3.75 bits per weight, where standard 3-bit or 4-bit schemes are a poor fit. It explains the core ideas behind mixed-precision quantization, including salience scoring with diagonal Fisher information, block-wise allocation, and why hardware constraints make many fine-grained methods difficult to deploy in practice. The discussion centers on SFMP’s main claim: instead of running an expensive search over bit allocations, it ranks blocks by importance and assigns only the two neighboring integer precisions, such as giving the most salient 25 percent of blocks 4-bit weights and the rest 3-bit for a 3.25 BPW target. Listeners would find it interesting because it frames quantization not as a narrow accuracy trick, but as a concrete engineering problem about trading off model quality, memory limits, and real serving-system compatibility.
Sources:
1. SFMP Search-Free Mixed-Precision LLM Quantization
https://arxiv.org/pdf/2602.01027
2. HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019
https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+AWare+Quantization+of+Neural+Networks+with+Mixed-Precision
3. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2023
https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers
4. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 2024
https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration
5. AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models — Sangjun Lee, Seung-taek Woo, Jungyu Jin, Changhun Lee, Eunhyeok Park, 2025
https://scholar.google.com/scholar?q=AMQ%3A+Enabling+AutoML+for+Mixed-precision+Weight-Only+Quantization+of+Large+Language+Models
6. BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments — Xinghao Wang, Pengyu Wang, Bo Wang, Dong Zhang, Yunhua Zhou, Xipeng Qiu, 2024
https://scholar.google.com/scholar?q=BitStack%3A+Fine-Grained+Size+Control+for+Compressed+Large+Language+Models+in+Variable+Memory+Environments
7. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models — Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, Xiaojuan Qi, 2024
https://scholar.google.com/scholar?q=SliM-LLM%3A+Salience-Driven+Mixed-Precision+Quantization+for+Large+Language+Models
8. FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs — Xilong Xie, Liang Wang, Limin Xiao, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu, 2025
https://scholar.google.com/scholar?q=FineQ%3A+Software-Hardware+Co-Design+for+Low-Bit+Fine-Grained+Mixed-Precision+Quantization+of+LLMs
9. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2024
https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge
10. MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design — Zhen Zheng et al., 2024
https://arxiv.org/abs/2412.14590
11. SpinQuant: LLM quantization with learned rotations — Zechun Liu et al., 2024
https://arxiv.org/abs/2405.16406
12. DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization — Yuantian Shao et al., 2025
https://arxiv.org/abs/2511.04063
13. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Coleman Hooper et al., 2024
https://arxiv.org/abs/2401.18079
14. QAQ: Quality Adaptive Quantization for LLM KV Cache — Shichen Dong et al., 2024
https://arxiv.org/abs/2403.04643
15. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3
16. AI Post Transformers: PackKV Lossy Compression for KV Caches — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-04-packkv-lossy-compression-for-kv-caches-b37bce.mp3
17. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3
18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3