Attention Sinks in Diffusion Language Models

Publication

Findings of the Association for Computational Linguistics: ACL 2026

Introduction

A small subset of tokens in a transformer often receives a disproportionate amount of attention from other tokens in the sequence. These tokens are known as attention sinks. In autoregressive language models, sinks are typically static and appear near the beginning of the sequence. Masking attention towards them can cause severe performance degradation.

Masked Diffusion Language Models (DLMs) offer a different generation process: they iteratively refine a masked sequence through successive denoising steps, using bidirectional attention. This raises a natural question: how does attention sinking behave when generation is no longer strictly left to right?

In our paper, we analyse three open-source masked DLMs: LLaDA-8B, Dream-7B, and MMaDA-8B. Our findings reveal that DLMs also exhibit attention sinks, but with distinct characteristics. Sink positions can shift throughout generation, and the tested models are substantially more robust to sink masking than the autoregressive baseline.

Incoming attention scores for LLaDA tokens across denoising steps, showing attention sinks shifting position during generation. — Figure 1. Incoming attention scores for each token in LLaDA across denoising steps. Attention sinks shift across the sequence as tokens are progressively unmasked.

From left-to-right generation to iterative denoising

Autoregressive models generate text sequentially, one token at a time. Each prediction depends on the preceding tokens, and a causal attention mask prevents tokens from attending to future positions.

Masked DLMs start with a sequence of [MASK] tokens and progressively denoise it into a fully unmasked output. Crucially, their attention is bidirectional: every token can attend to every other token in the sequence. The model can therefore use both left and right context during generation.

The models we study implement this process differently. LLaDA is trained from scratch as a diffusion model, while MMaDA is a multimodal DLM developed from LLaDA. At inference time, both use semi-autoregressive block decoding: the sequence is divided into blocks, and tokens within the current block are gradually unmasked before proceeding to the next block from left to right.

Dream is initialized from a pretrained autoregressive model. It retains a shifted prediction objective, where each position predicts the following token. In the configuration we study, Dream can unmask tokens at any position in the sequence.

Inference diagrams comparing autoregressive generation, block-based denoising in LLaDA and MMaDA, and denoising in Dream. — Figure 2. A snapshot of an inference step under the three generation schemes. Autoregressive models and Dream predict the next token; LLaDA and MMaDA predict the current one. LLaDA and MMaDA unmask tokens within the current block, while Dream may unmask a token at any position.

Attention sinks also emerge in DLMs

We first analyse the distribution of incoming attention. Across all three models, only a few tokens receive very high attention scores, while the majority receive much less. These outliers are the attention sinks.

To locate them, we track the attention each token receives from other tokens at each denoising step, separately for each head and layer. This allows us to follow both the intensity and the location of sinks throughout generation.

In an attention map, a sink appears as a bright vertical line: many query tokens attend strongly to the same key position. In DLMs, these lines need not remain in the same place. Sinks can appear in the middle or towards the end of the sequence, move between positions, or disappear altogether.

LLaDA attention maps at denoising steps 38 and 39, with the main sink moving from token position 62 to position 88. — Figure 3. A moving sink in LLaDA. Between steps 38 and 39, the sink shifts from position 62 to position 88. These heatmaps show attention averaged across the whole model.

Moving sinks take different forms

Alongside the static sinks familiar from autoregressive models, we identify moving sinks: sinks that do not remain at the same position across denoising steps. Their behaviour varies considerably with layer depth and backbone model.

LLaDA exhibits diverse moving sink patterns. A sink often remains at one position for several consecutive steps before vanishing. In more unstable cases, it appears for only a single step. As we progress to deeper layers, the number of sinks decreases, often converging to one or two per layer.

LLaDA's sinks also frequently form on structural tokens, including punctuation, whitespace, and end-of-sequence markers. This suggests that sink selection depends on token content as well as position.

Dream shows a primarily positional pattern. Its sinks often originate at the rightmost masked token and shift leftward as tokens are progressively unmasked. This migration is especially prominent in early layers and follows the unmasking frontier. We hypothesize that Dream's initialization from an autoregressive model contributes to this positional preference.

MMaDA has the most stable sink behaviour of the three models. Its sinks are generally less frequent and often remain fixed throughout generation. Many layers show no clear sink pattern. Moving sinks still occur, including cases where a substantial group of tokens shifts its attention towards an already unmasked token.

Trajectories of the largest attention sink in selected heads of LLaDA, Dream, and MMaDA across denoising steps. — Figure 4. Examples of how sinks move over time. At each iteration, we track the largest sink in a selected head of each model. These trajectories illustrate head-specific behaviour, rather than attention averaged across the whole model.

Some of the deepest LLaDA layers show another distinctive pattern: masked and unmasked tokens maintain separate attention sinks. At the end of a block, the attention map has a clear boundary between these two groups.

LLaDA attention map showing separate sinks for masked and unmasked tokens, with a clear boundary between the two groups. — Figure 5. Separate attention sinks for masked and unmasked tokens in LLaDA (layer 30, head 18, step 32). The map is taken at the end of a block, producing the clear boundary between the two groups.

What happens when we mask the sinks?

The dynamic nature of these sinks raises a second question: do DLMs depend on them as strongly as autoregressive models do?

We evaluate mathematical reasoning on GSM8K and code generation on HumanEval. At each denoising step, we identify the top-ranked sinks from the previous step's attention maps, separately across heads and layers, and set attention towards them to zero. We test masking one, five, or ten sinks.

For the autoregressive comparison, we apply a corresponding intervention to Llama-3.1-8B, selecting sinks from the last token's attention distribution. The table below shows the effect of masking a single sink.

Model	GSM8K: original	GSM8K: 1 sink masked	HumanEval: original	HumanEval: 1 sink masked
Dream-7B	0.82 ± 0.01	0.79 ± 0.01	0.60 ± 0.03	0.64 ± 0.03
LLaDA-8B	0.76 ± 0.01	0.75 ± 0.01	0.37 ± 0.03	0.37 ± 0.03
MMaDA-8B	0.54 ± 0.01	0.53 ± 0.01	0.16 ± 0.02	0.16 ± 0.03
Llama-3.1-8B	0.85 ± 0.01	0.02 ± 0.00	0.66 ± 0.04	0.00 ± 0.00

*cores and uncertainty terms reproduced from the paper. Higher is better.

The difference is substantial. Masking one sink reduces Llama's GSM8K score from 0.85 to 0.02 and its HumanEval score from 0.66 to zero. The DLMs retain performance much closer to their original scores under the same masking budget.

Masking five sinks also has a comparatively modest effect on the DLMs. With ten sinks masked, larger losses emerge: GSM8K scores fall to 0.75 for Dream, 0.55 for LLaDA, and 0.37 for MMaDA. DLMs are therefore robust to this intervention, but the strength of that robustness depends on the model and the number of sinks removed.

For MMaDA, we report our own evaluation results: we were unable to reproduce its originally published benchmark scores despite following the published implementation details.

Why might DLMs be more robust?

We hypothesize that bidirectional attention and iterative denoising work together to provide this robustness.

Bidirectional attention lets tokens access the full context at each step, potentially providing alternative attention pathways when a sink is unavailable. The decoding strategy adds another source of flexibility: among the positions eligible for unmasking, the model selects those where it is most confident.

When a sink is masked, the model may become less confident about the tokens most affected by its removal. Those tokens can remain masked while other, more confident tokens are unmasked. This may help explain why disrupting a sink need not immediately cause generation to fail.

This remains a hypothesis. Our study is empirical: it documents attention patterns and the effects of intervention, without establishing a detailed mechanistic explanation. The experiments cover three pretrained instruct models, rather than their corresponding base versions or alternative training procedures.

What remains open

Attention sinks consistently emerge in the DLMs we study, but their behaviour differs from the familiar static pattern in autoregressive models. Their positions can change throughout denoising, their selection can reflect structural tokens, and their removal has a much smaller effect under the tested intervention.

Several questions remain. What information is stored in sinks that occupy future positions? Why do some heads maintain separate sinks for masked and unmasked tokens? And how do training and initialization shape these patterns?

Understanding these mechanisms could clarify the role of attention sinks in long-context generation and help guide more efficient diffusion inference. Our findings provide an empirical starting point: diffusion models organize attention through flexible mechanisms that evolve with the denoising process.

Code and experiments