Estimating transfer entropy
TENDE (AISTATS 2026) estimates transfer entropy as the difference between two conditional scores of a diffusion model: the score of the next value of a series given both pasts, and given its own past only. It stays accurate in settings where nearest-neighbor, copula, and variational estimators fail.
Breathing and heart rate are coupled, and a natural question is which one drives the other. Mutual information cannot answer it, since it is symmetric. Transfer entropy asks a more specific question: once we know the past of the heart rate, how much does the past of respiration still tell us about the next heartbeat?
Definition
Take two time series \{X_t\} and \{Y_t\}, and write \mathbf Y_{t-\ell} = [Y_{t-1}, \dots, Y_{t-\ell}] and \mathbf X_{t-k} = [X_{t-1}, \dots, X_{t-k}] for their recent histories. Schreiber (2000) defined the transfer entropy from X to Y as the conditional mutual information
\mathrm{TE}_{X\to Y}(k,\ell) = I\big(Y_t;\ \mathbf X_{t-k} \,\big|\, \mathbf Y_{t-\ell}\big).
It is zero when the next value of Y is conditionally independent of the past of X given its own past, and positive when the past of X adds something. In general \mathrm{TE}_{X\to Y} \neq \mathrm{TE}_{Y\to X}, and comparing the two gives a direction. For linear Gaussian models it reduces to Granger causality.
The difficulty is the conditioning set. With N_y-dimensional series and \ell lags it has \ell N_y dimensions, and it grows with every lag we add. Nearest-neighbor estimators suffer from this directly. Vine-copula methods rely on a simplifying assumption that does not always hold. Donsker-Varadhan bounds need exponentially many samples to certify large values (McAllester and Stratos, 2020), and cross-entropy methods assume a categorical or parametric form for the conditionals.
The estimator
A conditional mutual information is an average of KL divergences, I(X;Y\mid Z) = \mathbb E_{y,z}\big[\operatorname{KL}(p_{X\mid y,z} \,\|\, p_{X\mid z})\big], and the identity behind this series turns each of them into an integrated squared difference of scores. In the transfer entropy setting the variable we diffuse is the next value of the target, Y_t, while both histories are kept clean as conditioning. With the noising process \mathrm dY_s = f_s Y_s\,\mathrm ds + g_s\,\mathrm dW_s in diffusion time s,
\mathrm{TE}_{X\to Y}(k,\ell) \simeq \int_0^T \frac{g_s^2}{2}\; \mathbb E\Big[\big\lVert \nabla \log p_s(y \mid \mathbf X_{t-k}, \mathbf Y_{t-\ell}) - \nabla \log p_s(y \mid \mathbf Y_{t-\ell}) \big\rVert^2\Big]\, \mathrm ds .
In words: the score of the next value given both pasts, minus its score given its own past only. (With the Ornstein-Uhlenbeck process of the hub note, g_s^2/2 = 1.)
Both scores come from one network. It receives the three blocks [Y_t, \mathbf X_{t-k}, \mathbf Y_{t-\ell}] together with a code in \{-1, 0, 1\} for each block: 1 for the block being diffused, 0 for a clean conditioning block, and -1 for a block that is marginalized out (set to zero). The code [1,0,0] gives the score conditioned on both histories, and [1,-1,0] the score conditioned on the target history only. Training is denoising score matching with the code drawn at random, and the reverse direction uses the same procedure with the roles of the two series swapped.
The form above estimates a KL divergence directly, so it cannot go negative. The paper also derives three alternatives that write the conditional mutual information as a difference of entropies or of mutual informations; they are useful for comparison but can propagate errors. The estimation error is neither an upper nor a lower bound, and errors shared by the two scores cancel.
Benchmarks
The synthetic systems have known transfer entropy: a linear Gaussian process, and a nonlinear system in which the coupling switches on above a threshold \lambda. We compare against AGM, NPEET (nearest neighbors), MINE adapted to conditional mutual information, TREET (a transformer with a Donsker-Varadhan bound), and Tigramite.
Two further tests use properties that transfer entropy must satisfy. Appending independent noise dimensions to each series leaves it unchanged: TENDE stays stable, while TREET becomes noisy and even negative. Stacking d independent copies of a system multiplies it by d: TENDE follows, TREET and Tigramite do not.
Higher dimensions are still hard. With 70 dimensions (35 per series), three TENDE variants recover the linear Gaussian value of 4.47 nats to within 0.15 nats. On the nonlinear system, whose true value is 14.53 nats, every method underestimates substantially: TENDE gives 6.9 with 10k samples and 10.9 with 50k, AGM 6.8, MINE 0.9, and NPEET 0.8.
Breathing and heart rate
On Santa Fe dataset B (a sleep-lab recording at 2 Hz, 1,200 samples, about ten minutes), we estimate \mathrm{TE}(k, \ell = 2) between respiration force and heart rate for source lags from 1 to 5 seconds.
The transfer entropy from respiration to heart rate is two to three times larger than in the other direction for most lags, with small variability across seeds, and it decreases once more than about three seconds of respiratory history are included. The direction agrees with earlier studies of the same data. The baselines are noisy (MINE, TREET), negative (NPEET), or orders of magnitude smaller (Tigramite). As always with physiological systems, this measures directed predictability; reading it as causation needs care.
References
- S. P. Galeano Muñoz, M. Bounoua, G. Franzese, P. Michiardi, M. Filippone. TENDE: Transfer Entropy Neural Diffusion Estimation. AISTATS 2026. arXiv:2510.14096. A worked example is in the course notebook; the implementation builds on the MINDE code.
- T. Schreiber. Measuring information transfer. Physical Review Letters, 2000.
- D. Kornai, R. Silva, N. Nikolaou. AGM-TE: Approximate generative model estimator of transfer entropy for causal discovery. 2025.
- O. Luxembourg, D. Tsur, H. Permuter. TREET: Transfer entropy estimation via transformers. IEEE Access, 2025.