Dr. Manuel Álvarez Chaves, formerly a PhD student with Dr. Anneli Guthke’s research group “Statistical Model-Data Integration” at the Stuttgart Center for Simulation Science (SimTech/ University of Stuttgart), has received the Entropy Best Paper Award. The award is presented annually to recognize outstanding publications of high scientific quality, significance and impact published in the journal Entropy.
The award-winning article, “On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data”, was authored by Manuel Álvarez Chaves together with Hoshin V. Gupta, Uwe Ehret, and Anneli Guthke. The paper, published in Entropy in 2024, addresses a fundamental challenge: how can data-driven methods reliably measure the amount of information contained in data and models?
Information-theoretic quantities such as entropy, Kullback–Leibler divergence, and mutual information are important tools for answering this question. In simple terms, entropy describes how much uncertainty or information is contained in a dataset. Mutual information can be used to measure how strongly two variables are related to each other, including complex and non-linear relationships. Kullback–Leibler divergence helps compare two probability distributions, for example, to assess how well a model represents observed data.
These concepts are highly relevant for many scientific fields, including artificial intelligence, environmental sciences, experimental design, neuroscience, and economics. However, applying them in practice is often difficult. Real-world data are typically continuous, noisy, and high-dimensional. This means that researchers first need to estimate probability distributions from sample data before they can calculate the desired information-theoretic quantities. In higher dimensions, this step can quickly become unreliable or computationally unfeasible.
In their study, the authors systematically compare three widely used non-parametric estimation approaches: histogram-based methods, kernel density estimation, and k-nearest-neighbour (kNN) methods. They test these approaches on a broad range of synthetic datasets with different distribution shapes, sample sizes, and dimensions. This allows them to evaluate how accurate, robust, and computationally efficient each method is under different conditions.
The results show that the performance of the methods strongly depends on the data and the quantity being estimated. Histogram-based methods can work well in simple, low-dimensional settings, but become less reliable in higher dimensions. Kernel density estimation can provide accurate results in some cases, but often requires considerable computational effort. The k-nearest-neighbour approach performs particularly well across many of the tested scenarios, especially when data are available in sufficient quantities. It offers a good balance between accuracy, computational efficiency, and practical usability.
Dr. Guthke is very happy to see this foundational work recognized and especially highlights Dr. Álvarez Chaves’ efforts towards providing concrete guidance for researchers and practitioners who want to use information-theoretic tools in their own work. By systematically revealing the strengths and limitations of different estimation methods, the study supports more informed choices in data analysis and modelling workflows in a wide range of scientific communities
To make the results directly usable, Dr. Álvarez Chaves compiled the investigated estimation methods in the freely accessible Python toolbox UNITE. This toolbox is intended to lower the barrier for applying information-theoretic concepts in different research fields and to promote their use in the analysis of data and models.
The study also places strong emphasis on transparency and reproducibility. The scripts used to generate the data, run the estimator comparisons, and produce the results are openly available via the University of Stuttgart’s data repository DaRUS. This allows other researchers to reproduce the analyses, build on the benchmark cases, and further develop the methods.
These concepts are currently extended by Dr. Guthke’s group within the DFG-funded follow-up project UNI-BENCH; additions will e.g. include an efficient kNN-based estimator for transfer entropy.
The Entropy Best Paper Award includes CHF 500, a certificate, and a voucher to waive the Article Processing Charges for one future submission to the journal. Congratulations!