Every time you use an app or browse sites online, your Android phone builds up a collection of small files in the background called a cache. That’s how apps and websites load images, scripts and page elements faster the next time you open them, instead of downloading the same data over and over again.
Most large language models (LLMs) use transformer architectures with softmax attention. During inference, their key-value (KV) cache grows with the conversation or document context. This can reduce concurrency, increase serving cost, and force agentic applications to discard useful history.
