KV Cache Management

DuoAttention: Enhancing Long-Context Inference Efficiency in Large Language Models

DuoAttention reimagines efficiency for Large Language Models (LLMs) by categorizing attention heads into Retrieval and Streaming types, allowing for effective memory optimization in long-context scenarios. This mechanism enables LLMs to…