DuoAttention: Enhancing Long-Context Inference Efficiency in Large Language Models
DuoAttention reimagines efficiency for Large Language Models (LLMs) by categorizing attention heads into Retrieval and Streaming types, allowing for effective memory optimization in long-context scenarios. This mechanism enables LLMs to…
You must be logged in to post a comment.