Sign up to save your podcasts
Or
Share A 30B Model Crawled on My Mini PC, But the Weights Were Innocent
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
A 30B Model Crawled on My Mini PC, But the Weights Were Innocent
This story was originally published on HackerNoon at: https://hackernoon.com/a-30b-model-crawled-on-my-mini-pc-but-the-weights-were-innocent.
GLM-4.7-Flash is a 30B model with 3B active params, so why did it crawl on my mini PC? Not the weights, not the quant. The attention design, and the runtime.
Check more stories related to tech-stories at: https://hackernoon.com/c/tech-stories.
You can also check exclusive content about #local-llms, #mixture-of-experts, #multi-head-latent-attention, #kv-cache, #llamacpp, #glm-4.7-flash-benchmark, #gguf-quantization-benchmark, #hackernoon-top-story, and more.
This story was written by: @joshgreen-dev. Learn more about this writer by checking @joshgreen-dev's about page,
and for more stories, please visit hackernoon.com.
GLM-4.7-Flash should fly on a small box: 30B total, ~3B active. On a Radeon 780M mini PC it crawled and fell apart past short context. The cause was not the weights or the quant. It was Multi-head Latent Attention with no runtime that implemented it, so the model fell back to plain attention and the KV cache ballooned ~17x. Swapping to ik_llama.cpp with a real MLA path fixed it.
View all episodes
By HackerNoonThis story was originally published on HackerNoon at: https://hackernoon.com/a-30b-model-crawled-on-my-mini-pc-but-the-weights-were-innocent.
GLM-4.7-Flash is a 30B model with 3B active params, so why did it crawl on my mini PC? Not the weights, not the quant. The attention design, and the runtime.
Check more stories related to tech-stories at: https://hackernoon.com/c/tech-stories.
You can also check exclusive content about #local-llms, #mixture-of-experts, #multi-head-latent-attention, #kv-cache, #llamacpp, #glm-4.7-flash-benchmark, #gguf-quantization-benchmark, #hackernoon-top-story, and more.
This story was written by: @joshgreen-dev. Learn more about this writer by checking @joshgreen-dev's about page,
and for more stories, please visit hackernoon.com.
GLM-4.7-Flash should fly on a small box: 30B total, ~3B active. On a Radeon 780M mini PC it crawled and fell apart past short context. The cause was not the weights or the quant. It was Multi-head Latent Attention with no runtime that implemented it, so the model fell back to plain attention and the KV cache ballooned ~17x. Swapping to ik_llama.cpp with a real MLA path fixed it.