Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Venue

Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing

Introduction

Masked Diffusion Language Models (DLMs) progressively generate text by unmasking tokens. Unlike standard diffusion-based approaches, the models we study are not explicitly conditioned on a timestep. This raises a natural question: do these models internally represent denoising progress, and how is such information used downstream?

In our work, we show that they encode a latent representation related to denoising progress within their residual streams. This signal can be recovered from individual token representations, manipulated to produce predictable changes in model confidence and entropy, and characterised through its low-dimensional geometry.

We investigate this in LLaDA-1.5 and Dream-7B, addressing three questions: can we recover the signal, does it affect the model's computation, and how is it represented across layers?

A three-dimensional PCA projection of LLaDA's mean activation vectors forms a continuous, ordered curve. — Figure 1. A 3D projection of latent denoising-progress modelling in LLaDA. The mean activation vectors follow a low-dimensional, manifold-like curve, progressing from all [MASK] tokens (red) to no remaining [MASK] tokens (purple).

Measuring denoising progress

DLMs are trained to recover clean text from a corrupted sequence in which some positions have been replaced by a special `[MASK]` token. At inference time, generation begins with a window of masked tokens appended to the prompt. At each denoising step, the model predicts token distributions and a subset of masked positions is revealed. Generation ends when all positions have been unmasked.

We measure progress through the fraction of response tokens already unmasked:

\tau_t = 1 - \frac{\text{number of masked response tokens at step }t}{\text{response length}}.

Thus, τ starts at 0 and reaches 1 when the response is fully unmasked. We use it as an empirical measure of denoising progress and a proxy for denoising time.

The mask ratio is directly available in the input. What remains unclear is whether, and how, the model represents this sequence-level information internally.

Recovering the signal

To test whether τ can be recovered from the residual stream, we train a separate multilayer perceptron (MLP) probe at each layer. Each probe predicts the current denoising progress from a single contextualised token representation, without direct access to the full sequence or neighbouring activations.

We fit probes using masked tokens, unmasked tokens, or all tokens. This allows us to investigate whether progress information is preferentially encoded in a particular token type.

A heatmap shows high probe R-squared scores across LLaDA's 32 layers for masked tokens, unmasked tokens, and all tokens. — Figure 2. MLP probes recover the τ signal throughout LLaDA. Columns correspond to layers; rows indicate the token subset used for training. Higher R² means better recovery. Both masked and unmasked tokens carry denoising-progress information. Click the figure to inspect the values at full resolution.

In LLaDA, the probes maintain R² above 0.5 across all layers, with the strongest performance through much of the middle of the network. We observe similar trends in Dream. Masked tokens yield marginally better predictions, although both token types support accurate recovery.

The key observation is that each probe only sees one hidden state. Individual contextualised token representations therefore carry a latent representation of a sequence-level statistic. Denoising-progress information is distributed across the residual stream and can be read from both masked and unmasked positions.

Assessing the signal's importance

Having established that the signal can be recovered, we next investigate whether modifying it changes the model's behaviour.

To approximate its representation, we compute mean activation vectors: we average hidden states across tokens and examples, grouping them by denoising progress. Averaging suppresses instance-specific information while preserving systematic signals shared across examples.

We discretise progress into 100 bins. These are progress bins, not a requirement to generate with 100 inference steps. Applying the trained probes to the mean vectors, without further tuning, yields Pearson correlations above 0.96 in both models. The mean vectors track the same progress-related signal recovered by the probes.

We then steer the model by subtracting the mean vector for its current progress and adding the mean vector for a target progress:

\widetilde{h} = h - \mu_{\mathrm{current}} + \mu_{\mathrm{target}}.

Here, h is a hidden state at the chosen layer. We apply the same shift across all tokens. The part of the mean representation that is independent of progress cancels in the subtraction.

Steering LLaDA at layer 29 changes entropy, confidence, and KL divergence systematically; matched random perturbations have weaker effects. — Figure 3. Downstream effects of steering LLaDA at layer 29. Orange curves show mean-vector steering towards progress bins 25, 50, and 75; red curves show norm-matched random controls. The panels measure changes in entropy (top left), confidence (top right), and KL divergence from the original distribution (bottom). The horizontal axis follows denoising progress across the 100 bins.

The strongest downstream effects emerge in the final layers of both models. Steering towards later denoising progress increases confidence and decreases entropy. Steering towards earlier progress reverses this behaviour. Moving the target further from the current progress also produces larger changes in the output distribution, measured by KL divergence.

These effects closely match what would be expected from modifying the model's internal notion of denoising progress. As a control, we apply random perturbations sampled using the activations' empirical covariance and matched to the steering vectors' norm. They do not produce the same coherent trends in confidence and entropy, and their downstream impact is substantially weaker.

What happens to the signal at early layers?

Steering shallow layers produces much weaker effects. To understand why, we inject a perturbation at layer 6 and track the probe-predicted progress through the remaining layers.

The difference between clean and steered probe predictions shrinks through later LLaDA layers after a perturbation at layer 6. — Figure 4. Average probe drift after steering at layer 6 in LLaDA. Later computation progressively reduces the discrepancy between clean and steered representations. Most perturbations shrink substantially, while extreme late-progress targets leave a larger residual.

The model corrects most of the perturbation as computation proceeds. For many targets, probe predictions become nearly indistinguishable from their clean values in the final blocks. Extreme late-progress targets remain harder to compensate for.

This suggests that the signal is repeatedly updated across depth, allowing later layers to compensate for early perturbations. It also helps explain why interventions closer to the output have stronger downstream effects.

Characterising the signal

Having established that the signal affects model predictions, we examine its geometry.

Principal component analysis (PCA) reveals that a large fraction of the variance of the mean activation vectors can be captured by a single principal component, with the distribution concentrated above 90% explained variance. Two- and three-dimensional projections reveal a continuous, ordered structure, with many layers exhibiting a similar parabola-like shape.

To examine whether this shape is shared across layers, we standardise each layer's two-dimensional projection and average the points corresponding to the same progress bin.

The standardised two-dimensional mean trajectory across LLaDA layers forms a parabola-like curve, coloured by denoising progress. — Figure 5. A shared two-dimensional geometry in LLaDA. After standardising each layer's principal-component projections, the mean vectors closely follow a common parabola-like trajectory. Colour runs from early progress (purple) to late progress (yellow).

The geometry also has functional relevance. We repeat the steering experiment using only the component of the steering vector lying within the first two principal directions, and compare it with the component in the orthogonal complement. Both perturbations are rescaled to match the original vector's norm.

Steering within the top two principal components closely reproduces full-vector steering in LLaDA, while orthogonal steering produces different, incoherent effects. — Figure 6. Two-dimensional steering at LLaDA layer 29. Steering within the top-two-PC subspace (blue) closely reproduces full-vector steering (orange). Perturbations in the orthogonal complement (green) do not follow the same coherent pattern, despite having the same norm.

In this experiment, the first two principal components recover the behaviour of the full intervention. The low-dimensional structure therefore captures much of the geometry relevant to steering the progress signal.

Across depth, the models differ in how they preserve the directions of these representations. Most LLaDA layers are strongly aligned, with the final layer an exception. Dream exhibits a more heterogeneous organisation, with alignment concentrated within particular blocks of layers.

Within layers, we also observe strongly anti-correlated mean representations in the self-attention and MLP outputs. We hypothesise that this interaction may help explain the correction of early perturbations, but the underlying mechanism remains unresolved.

How is this signal computed?

A key implication of our findings is that probes or auxiliary models operating on DLM hidden states need not receive an explicit timestep embedding to access denoising-progress information. This information is already encoded in the residual stream.

Our results also leave several questions open. We characterise a signal closely related to the fraction of unmasked tokens, but do not identify the circuit that constructs or updates it. Future work could investigate explicit mask-ratio estimation, distributed sequence-level statistics, or interactions between attention and MLP components.

Predictable changes in confidence do not establish a general improvement in generation quality. In our downstream experiments, Dream's GSM8K score falls from 68.8 to 23.1 when steered towards the final progress bin. Qualitative inspection links this degradation to premature end-of-sequence tokens, suggesting that the progress signal is entangled with the token distribution itself.

Our analysis is restricted to LLaDA and Dream. Whether this representation extends to other diffusion language models, and whether it can support more efficient decoding or remasking strategies, remain open directions. Understanding how the signal is computed would further clarify how these models organise denoising and could inform better architecture designs.