Concepts and Recent Advances in Generalized Information Measures and Statistics

We provide a brief description of the historical development of the concept of Entropy, from its origins related to heat to its modern interpretation as a special measure of information.

This introductory chapter provides a basic review of the Shannon entropy and of some important related quantities like the joint entropy, the conditional entropy, the mutual information and the relative entropy. We also discuss the Fisher information, the fundamental property of concavity, the basic elements of the maximum entropy approach and the definition of entropy in the Quantum case. We close this chapter with the axioms which determine the Shannon entropy and a brief description of other information measures.

Clausius introduced, in the 1860s, a thermodynamical quantity which he named entropy S. This thermodynamically crucial quantity was proposed to be extensive, i.e., in contemporary terms, S(N) ∝ N in the thermodynamic limit N →∞. A decade later, Boltzmann proposed a functional form for this quantity which connects S with the occurrence probabilities of the microscopic configurations (referred to as complexions at that time) of the system. This functional is, if written in modern words referring to a system with W possible discrete states, S_BG = −k_B ∑^w_i=1 pi ln pi, with ∑^W_i=1 pi=1, k_B being nowadays called the Boltzmann constant (BG stands for Boltzmann-Gibbs, to also acknowledge the fact that Gibbs provided a wider sense for W). The BG entropy is additive, meaning that, if A and B are two probabilistically independent systems, then S_BG(A+B) = S_BG(A)+S_BG(B). These two words, extensive and additive, were practically treated by physicists, for over more than one century, as almost synonyms, and S_BG was considered to be the unique form that S could take. In other words, the functional SBG was considered to be universal. It has become increasingly clear today that it is not so, and that those two words are not synonyms, but happen to coincide whenever we are dealing with paradigmatic Hamiltonians involving short-range interactions between their elements, presenting no strong frustration and other “pathologies”. Consistently, it is today allowed to think that the entropic functional connecting S with the microscopic world transparently appears to be nonuniversal, but is rather dictated by the nature of possible strong correlations between the elements of the system. These facts constitute the basis of a generalization of the BG entropy and statistical mechanics, introduced in 1988, and frequently referred to as nonadditive entropy Sq and nonextensive statistical mechanics, respectively. We briefly review herein these points, and exhibit recent as well as typical applications of these concepts in natural, artificial, and social systems, as shown through theoretical, experimental, observational and computational predictions and verifications.

Shannon and Khinchin showed that assuming four information theoretic axioms the entropy must be of Boltzmann-Gibbs type, S = ∑_i pi log pi. Here we note that in physical systems one of these axioms may be violated. For non-ergodic systems the so called separation axiom (Shannon-Khinchin axiom 4) will in general not be valid. We show that when this axiom is violated the entropy takes a more general form, S_c,d ∝ ∑^w_i Γ(d + 1, 1 − c log pi), where c and d are scaling exponents and Γ a, b) is the incomplete gamma function. The exponents (c, d) define equivalence classes for all interacting and non-interacting systems and unambiguously characterize any statistical system in its thermodynamic limit. The proof is possible because of two newly discovered scaling laws which any entropic form has to fulfill, if the first three Shannon- Khinchin axioms hold. (c, d) can be used to define equivalence classes of statistical systems. A series of known entropies can be classified in terms of these equivalence classes. We show that the corresponding distribution functions are special forms of Lambert-W exponentials containing – as special cases – Boltzmann, stretched exponential and Tsallis distributions (power-laws). In the derivation we assume trace form entropies, S = ∑_i g(pi), with g some function, however more general entropic forms can be classified along the same scaling analysis.

We review the concept of majorization and its relation with generalized information measures. Majorization theory provides an elegant framework for comparing two probability distributions, leading to a rigorous concept of disorder which is more stringent than that based on the Shannon entropy. Nevertheless, it is shown that it can be fully captured through general entropic inequalities based on generalized entropic forms. A brief review of generalized entropies is also provided. As illustration, we discuss the majorization properties of generalized thermal distributions derived from generalized entropies, and identify rigorous mixing parameters. We also describe majorization in quantum systems. We discuss in particular its capability for providing a disorder based criterion for the detection of quantum entanglement, which is stronger than that based on the von Neumann entropy and leads to a generalized entropic separability criterion.

Many problems of statistical and quantum mechanics can be established in terms of a distance between probability distributions. The present work is devoted to review some of the most relevant applications of the notion of distance in the context of statistical mechanics, quantum mechanics and information theory. Although we make a general overview of the most frequently used distances between probability distributions, we will center our presentation in a distance known as the Jensen-Shannon divergence both in their classical and quantum versions. For the classical one we present its main properties and we discuss its relevance as a segmentation tool for symbolic sequences. In the quantum case we show that the quantum Jensen-Shannon divergence is an adequate measure of entanglement.

In this chapter, a statistical measure of complexity is introduced and some of its properties are discussed. Also, some straightforward applications are shown.

A generalized Statistical Complexity Measure (SCM) is a functional of the probability distribution P associated with the time series generated by a given dynamical system. A SCM is the composition of two ingredients: i) an entropy and ii) a distance in the probability-space. We address in this review important topics underlying the SCM structure, viz., a) the selection of the information measure I; b) the choice of the probability metric space and associated distance D, which in this context is called a “disequilibrium” (denoted with the letter Q). Q, indeed the crucial SCM ingredient, is cast in terms of an associated distance D. c) The adequate way of picking up the probability distribution P associated with a dynamical system or time series under study, which is indeed a fundamental problem. A good analysis of this topics is essential to get a SCM that quantifies not only randomness but also the presence of correlational structures. In this chapter we specially stress how sensible improvements in the final results can be obtained if the underlying probability distribution is “extracted” via appropriate considerations regarding causal effects in the system’s dynamics. As an illustration, we show just how these issues affect the description of the celebrated logistic map.

The Fisher information functional occupies a very special position among the uncertainty measures of quantum systems of any dimensionality. In contrast with (global) entropic measures of Shannon, Renyi or Tsallis types, it contitutes a local measure in the sense that it is sensitive to the oscillatory character of the probability density distribution. Moreover, the Fisher information is closely related to various macroscopic properties of the system described by different density functionals. Here, we review its main analytical properties such as, inequality-based relationships with some radial expectation values and its uncertainty relation. Tighter versions of these inequalities are shown for those systems subject to an arbitrary central potential. Moreover, the Fisher information is computed for D-dimensional hydrogenic systems explicitly in terms of the radial and angular hyperquantum numbers characterizing the quantum states of the system in both position and momentum spaces. Finally, the utility of this quantity is discussed for various physico-chemical processes of recent interest: Abstraction and nucleophilic substitution reactions.

We review links between the entanglement concept and the entropic-information one, as represented by different measures like the Shannon and Tsallis ones. This remarkable linkage illuminates several aspects of the physics of information-theoretic activity and it is thus of importance to understand the structure of the concomitant connections

We review here the difference between quantum statistical treatments and semiclassical ones, using as the main concomitant tool a semiclassical, shift-invariant Fisher information measure built up with Husimi distributions. Its semiclassical character notwithstanding, this measure also contains abundant information of a purely quantal nature. Such a tool allows us to refine the celebrated Lieb bound for Wehrl entropies and to discover thermodynamic-like relations that involve the degree of delocalization. Fisher-related thermal uncertainty relations are developed and the degree of purity of canonical distributions, regarded as mixed states, is connected to this Fisher measure as well.

This chapter deals with the use of the Statistical Complexity Measure, as defined by Lopez Ruiz, Mancini and Calbet [Phys. Lett. A 209 (1995) 321–326] and modified by Rosso and coworkers [P. W. Lamberti, M. T. Martin, A. Plastino, O. A. Rosso; Physica A 334 (2004) 119–131] to characterize pseudo random number generators (PRNG’s) obtained from chaotic dynamical systems. It is shown that two probability distribution functions are required for a proper characterization: the first one is based on the histogram and is used to characterize the uniformity of the values in the time series; the second one is based on the permutation procedure proposed by Bandt and Pompe [Phys. Rev. Lett. 88 (2002) 174102] and characterize the uniformity of patterns of several consecutive values of the time series.

Quantifiers based on Information Theory, the normalized Shannon entropy, the Fisher information measure and the Shannon-Jensen statistical complexity are used to characterize changes in the dynamical behavior of EL Nino Southern-Oscillation (ENSO) during the Holocene. For this purpose we analyze the ENSO proxy record corresponding to Pallcacocha Lake sedimentary data for the Holoceno period (11,000 yr BP to present). By recourse of these quantifiers we found evidence of a shift in dynamics and cyclic behavior of the ENSO proxy, which is consistent with the results of Moy et al. [Nature 420 (2002) 162-165]. In addition, we have been also able to localize these cycles in time and to analyze connections to epochs of rapid climate change (RCC) during the Holocene.

We present an application of wavelet-based Information Theory quantifiers (Relative Wavelet Energy, Normalized Total Wavelet Entropy, Wavelet MPR-Statistical Complexity and Entropy-Complexity plane) on red blood cells membrane viscoelasticity characterization. These quantifiers exhibit important localization advantages provided by the Wavelet Theory. The present approach produces a clear characterization of this dynamical system. We have found out an evident manifestation of a random process on the red cell samples of healthy individuals, which is weakly less when studying glucose incubated erythrocytes, while there is a sharp reduction of randomness on analyzing a human haematological disease, such as β-thalassemia minor.

We have applied an information theoretic approach to gain insights of the role of spike correlations in the neuronal code. First, we illustrate and compare the different methods used in the literature to remove sample size dependent bias from information estimations. Then, we use a modified version of the information components breakdown to quantify the contribution of individual members of the population, the interaction between them, and the overall information encoded by the ensemble of neurons making an especial emphasis of the separation between contributions due to the noise and signal spike correlations. This formalism is applied to a set of multi-neuronal spike data with different stimuli configurations.