Prompt Cache is a method for accelerating inference in large language models by reusing attention states of frequently occurring text segments. It significantly reduces latency without compromising output accuracy or requiring modifications to the model parameters.

https://arxiv.org/abs//2311.04934

YouTube: https://www.youtube.com/@ArxivPapers

TikTok: https://www.tiktok.com/@arxiv_papers

Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016

Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers