Sparse attention is a family of modifications to the transformer's attention mechanism in which each token attends to only a chosen subset of other tokens rather than to every one. Standard full attention compares all token pairs, a cost that grows quadratically: double the sequence length and the attention computation quadruples. Sparse variants cut this by restricting the comparisons, using sliding windows where tokens see only their neighbors, global tokens that everything can see, strided or block patterns, or learned routing that picks the most relevant tokens dynamically.
This is one of the core techniques behind the long context windows modern models advertise. Full attention over hundreds of thousands of tokens would be prohibitive in compute and memory, so architectures mix sparse and dense layers to approximate full attention at a fraction of the cost; Mistral's early models used sliding-window attention, research designs like Longformer and BigBird established the patterns, and recent work such as DeepSeek's native sparse attention pushes trainable sparsity further. The premise is that attention weights are naturally concentrated, most tokens matter little to most others, so pruning the negligible pairs sacrifices little quality. The practical caveat is that long-context models can still show uneven recall across their window, so a large advertised context is a capability to verify against your workload, not a guarantee.
For arosplatforms this is model-selection due diligence. When client use cases hinge on long documents, contracts, transcripts, technical manuals, we test candidate models for retrieval fidelity across the full context length rather than trusting the number on the label, and we still prefer focused retrieval over stuffing the window, because attention efficiency never makes irrelevant context free.