Information estimation for discrete data
INFO-SEDD (ICLR 2026) brings diffusion-based information estimation to discrete data such as DNA sequences and text. It replaces the diffusion with a continuous-time Markov chain, and a single masked network provides every term it needs.
Which positions of a DNA sequence tell us whether it is a promoter, the region that switches a gene on? This is a mutual information question, between a window of bases and a label, but the data are sequences over four symbols. The estimators in the rest of this series do not apply directly: they rely on the score \nabla \log p, which has no meaning on a finite set.
The usual workaround is to embed the symbols in a vector space, for example with a pretrained language model, and run a continuous estimator on the embeddings. That requires a good embedding for each application, may lose the discrete structure of the data, and remains affected by the curse of dimensionality. Counting does not scale either: a sequence of M symbols from an alphabet of size N has N^M possible values.
The identity for Markov chains
The discrete counterpart of a diffusion is a continuous-time Markov chain on the finite state space, with generator Q_t: nonnegative jump rates Q_t(x, y) off the diagonal, and rows that sum to zero. Its time reversal is again a Markov chain, with rates that depend on the ratios p_t(y)/p_t(x). These ratios play the role of the score (they are called the discrete or concrete score), and a network can learn them with the score-entropy loss of discrete diffusion models.
Run two chains with the same generator, from initial laws p_0 and q_0. Using Dynkin’s formula one obtains the discrete version of the score-difference identity,
\operatorname{KL}(p_0 \,\|\, q_0) = \mathbb E\left[\int_0^T \sum_{y \neq X_t} Q_t(X_t, y)\left( K\!\left(\frac{p_t(y)}{p_t(X_t)}\right) + \frac{q_t(y)}{q_t(X_t)} - \frac{p_t(y)}{p_t(X_t)} \log \frac{q_t(y)}{q_t(X_t)} \right) \mathrm dt \right],
where K(a) = a(\log a - 1) and X_t is the chain started from p_0. The terminal term drops out because both chains converge to the same reference distribution. The estimator plugs in learned ratios and averages over sampled times and trajectories.
Two choices make this practical for long sequences. A sparse generator lets only one position change per jump, so the sum runs over sequences at Hamming distance one and grows linearly with the length. And every position jumps, at rate \sigma(t), to an extra absorbing “mask” symbol.
The mask has a useful property. If X and Y are diffused together, the ratio p_t(x, \text{mask}) / p_t(\tilde x, \text{mask}), with Y fully masked, equals the marginal ratio p^X_t(x) / p^X_t(\tilde x). One network trained on the joint distribution therefore gives the joint ratios and both marginal ones, and since I(X;Y) = \operatorname{KL}(p_{XY} \,\|\, p_X \otimes p_Y), one training run gives the mutual information. Masking subsets of positions gives the mutual information between any subsets of X and Y, from the same model. Entropy follows from \mathcal H(p_0) = \log |\mathcal X| - \operatorname{KL}(p_0 \,\|\, u_0), where u is uniform and its ratios are known in closed form under the absorbing chain. The method also works on top of pretrained discrete diffusion models.
Benchmarks
Discrete problems become large in two ways, through the size of the alphabet and through the length of the sequence. The first table varies the alphabet for a pair of variables with mutual information 0.5 nats.
| Alphabet size | 2 | 16 | 64 | 256 | 1024 |
|---|---|---|---|---|---|
| MINDE | 0.51 | 0.02 | 0.00 | 0.00 | 0.00 |
| MINE | 0.50 | 0.00 | diverges | 0.00 | n/a |
| KSG | 0.49 | 0.49 | 0.48 | 0.44 | 0.24 |
| INFO-SEDD | 0.50 | 0.51 | 0.54 | 0.58 | 0.61 |
MINDE and MINE collapse as soon as the alphabet grows, and KSG drifts down at the largest size. INFO-SEDD stays close to the true value, with a slight upward bias. For binary sequences of growing length it is accurate up to length 128 (0.48 nats). At lengths 512 and 2,048 every method is far off, INFO-SEDD included. With a length of 10 and true values from 1 to 5 nats, INFO-SEDD gives 1.01, 2.03, 3.02, 4.01, and 5.07.
The entropy of a 20 \times 20 Ising model is known in closed form (Onsager), which makes it a good test for the entropy estimator. A single model of about 90k parameters, trained on Metropolis-Hastings samples, covers the whole range of temperatures.
The estimates are accurate at low temperature, where the divergence from the uniform distribution is largest, and slightly above the closed form at high temperature.
Finding a promoter motif
Back to the opening question. We take promoter and non-promoter sequences of the plant Arabidopsis thaliana (251 bases around the transcription start site), fine-tune a pretrained DNA diffusion model with the promoter label prepended to each sequence, and slide a window of 6 bases along it. With every other position masked, the same network gives the mutual information between the window and the label.
The profile is flat over most of the sequence and peaks inside the annotated TATA box, 26 to 39 bases upstream of the start site, without any motif labels or sequence alignment. ALICE later reproduced this analysis without training on the sequences at all.
The ICLR version of the paper adds consistency tests on real genomic and text data, where labels are corrupted with a known probability so that the true mutual information changes in a controlled way, and an application to choosing among text summarization models.
References
- A. Foresti, G. Franzese, P. Michiardi. Information Estimation with Discrete Diffusion. ICLR 2026. OpenReview; preprint INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimators, arXiv:2502.19183; code.
- A. Lou, C. Meng, S. Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. ICML 2024.
- R. Umarov, V. Solovyev. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PLoS One, 2017.