article

Some properties of Rényi entropy and Rényi entropy rate

Authors:

Leila Golshani,

Einollah Pasha,

Gholamhossein YariAuthors Info & Claims

Information Sciences—Informatics and Computer Science, Intelligent Systems, Applications: An International Journal, Volume 179, Issue 14

Pages 2426 - 2433

https://doi.org/10.1016/j.ins.2009.03.002

Published: 15 June 2009 Publication History

Abstract

In this paper, we define the conditional Renyi entropy and show that the so-called chain rule holds for the Renyi entropy. Then, we introduce a relation for the rate of Renyi entropy and use it to derive the rate of the Renyi entropy for an irreducible-aperiodic Markov chain. We also show that the bound for the Renyi entropy rate is simply the Shannon entropy rate.

References

[1]

Andai, A., On the geometry of generalized Gaussian distributions. Journal of Multivariate Analysis. v100. 777-793.

Digital Library

[2]

Bentes, S.R., Menezes, R. and Mendes, D.A., Long memory and volatility clustering: is the empirical evidence consistent across stock markets?. Physica A. v387. 3826-3830.

[3]

Bercher, J.F., On some entropy functionals derived from Rényi information divergence. Information Sciences. v178. 2489-2506.

Digital Library

[4]

C. Cachin, Entropy measures and unconditional security in cryptography, PhD Thesis, Swiss Federal Institute of Technology Zurich, 1997.

[5]

Csiszár, I., Generalized cutoff rates and Rényi's information measures. IEEE Transactions on Information Theory. v41. 26-34.

Digital Library

[6]

Cover, T.M. and Thomas, J., The Elements of Information Theory. 1991. John Wiley and Sons.

Digital Library

[7]

De Gregorio, A. and lacus, S.M., On Rényi information for ergodic diffusion processes. Information Sciences. 279-291.

Digital Library

[8]

Dukkipati, A., Bhatnagar, S. and Murty, M.N., Gelfand-Yaglom-Perez theorem for generalized relative entropy functionals. Information Sciences. v177. 5707-5714.

Digital Library

[9]

Farhadi, A. and Charalambous, C.D., Robust codind for a class of sources: applications in control and reliable communication over limited capacity channels. Systems and Control Letters. v57. 1005-1012.

[10]

Guiasu, S., Information Theory with Applications. 1977. McGraw-Hill, Inc.

[11]

N.J.A. Harvey, K. Onak, J. Nelson, Streaming algorithms for estimating entropy, in: IEEE Information Theory Workshop, 2008, pp. 227-231.

[12]

Jacquet, P., Seroussi, G. and Szpankowski, W., On the entropy of hidden Markov process. Theoretical Computer Science. v395. 203-219.

Digital Library

[13]

Jenssen, R. and Eltoft, T., A new information theoretic analysis of sum-of-squared-error kernel clustering. Neurocomputing. v72. 23-31.

Digital Library

[14]

Jizba, P. and Arimitsu, T., The world according to Rényi: thermodynamics of multifractal systems. Annals of Physics. v312. 17-59.

[15]

Kanaya, F. and Han, T.S., The asymptotics of posterior entropy and error probability for Bayesian estimation. IEEE Transactions on Information Theory. v41. 1988-1992.

[16]

Khinchin, A.I., Mathematical Foundations of Information Theory. 1957. Dover Publications, New York.

[17]

Kirchanov, V.S., Using the Rényi entropy to describe quantum dissipative systems in statistical mechanics. Theoretical and Mathematical Physics. v156. 1347-1355.

[18]

Klimko, E.M. and Sucheston, L., On convergence of information in spaces with infinite invariant measure. Zeitschrift fur Wahrscheinlichkeitstheorie und Verwandte Gebiete. v10. 226-235.

[19]

Kolmogorov, A.N., Sur la notion de la moyenne. Atti della Accademia Nazionale dei Lincei Rend. v12 i6. 388-391.

[20]

Lake, D.E., Rényi entropy measures of heart rate gaussianity. IEEE Transactions on Biomedical Engineering. v53. 21-27.

[21]

Nagumo, M., Uber eine klasse der mittelwerte. Japanese Journal of Mathematics. v7. 71-79.

[22]

Nilsson, M. and Kleijn, W.B., On the estimation of differential entropy from data located on embedded manifolds. IEEE Transactions on Information Theory. v53. 2330-2341.

Digital Library

[23]

Paris, J.B. and Rad, S.R., Inference processes for quantified predicate knowledge. Logic, Language, Information and Computation. v5110. 249-259.

Digital Library

[24]

E. Pasha, L. Golshani, The Rényi entropy rate for Markov chains with countable state space, Bulletin of the Iranian Mathematical Society (submitted for publication).

[25]

Z. Rached, A. Fady, L.L. Campbell, Rényi's entropy rate for discrete Markov sources, in: Proceedings of the CISS'99, Baltimore, MD, March 1999, pp. 17-19.

[26]

Rached, Z., Fady, A. and Campbell, L.L., Rényi's divergence and entropy rate for finite alphabet Markov sources, sources. IEEE Transactions on Information Theory. v47. 1553-1560.

Digital Library

[27]

Rényi, A., On measures of entropy and information. In: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics Probability, vol. 1. Univ. Calif. Press, Berkeley, CA. pp. 547-561.

[28]

Rényi, A., Probability Theory. 1970. North-Holland, Amsterdam.

[29]

Seneta, E., Nonnegative Matrix and Markov Chains. 1981. Springer-Verlag, New York.

[30]

Shannon, C.E., A mathematical theory of communication. Bell System Technical Journal. v27. 379-423.

[31]

H. Shimokawa, Rényi's entropy and error exponent of source coding with countably infinite alphabet, in: IEEE International Symposium on Information Theory, vol. 1- 6, 2006, pp. 1831-1835.

[32]

Vollbrecht, K.G.H. and Wolf, M.M., Conditional entropies and their relation to entanglement criteria. Journal of Mathematical Physics. v43. 4299-4306.

[33]

Wimmer, H.K., Spectral radius and radius convergence. American Mathematical Monthly. v81. 625-627.

[34]

K. Zografos, Entropy and Divergence Measures for Mixed Variables, Statistical Models and Methods for Biomedical and Technical Systems, Statistics for Industry and Technology, Birkhauser, 2008, pp. 519-534.

Cited By

Wu CLi YXu LHan G(2024)Rényi Entropy Rate of Stationary Ergodic ProcessesIEEE Transactions on Information Theory10.1109/TIT.2023.331826570:1(1-15)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIT.2023.3318265
Megahed MMohammed A(2024)A comprehensive review of generative adversarial networksWIREs Computational Statistics10.1002/wics.162916:1Online publication date: 21-Jan-2024
https://dl.acm.org/doi/10.1002/wics.1629
Lu XZhu LSun Y(2023)Multi-view Subspace Clustering with Complex Noise ModelingProceedings of the 2023 9th International Conference on Computing and Data Engineering10.1145/3589845.3589854(48-57)Online publication date: 6-Jan-2023
https://dl.acm.org/doi/10.1145/3589845.3589854
Show More Cited By

Index Terms

Some properties of Rényi entropy and Rényi entropy rate
1. Mathematics of computing
  1. Probability and statistics

Index terms have been assigned to the content through auto-classification.

Recommendations

Rényi entropy rate for Gaussian processes

In this paper, we introduce the definition of the conditional Renyi entropy for continuous random variables and show that the so-called chain rule holds. Then, we use this rule to obtain another relation for getting the rate of Renyi entropy. Using this ...
Swiveled Rényi entropies

This paper introduces "swiveled Rényi entropies" as an alternative to the Rényi entropic quantities put forward in Berta et al. (Phys Rev A 91(2):022333, 2015). What distinguishes the swiveled Rényi entropies from the prior proposal of Berta et al. is ...
A formulation of Rényi entropy on $C^{*}$ -algebras
Abstract
The entropy of probability distribution defined by Shannon has several extensions. Renyi entropy is one of the general extensions of Shannon entropy and is widely used in engineering, physics, and so on. On the other hand, the quantum analogue of ... $^{}$ $^{}$

Comments

Information & Contributors

Information

Published In

Copyright © Elsevier Inc. © 2009.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 15 June 2009

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu CLi YXu LHan G(2024)Rényi Entropy Rate of Stationary Ergodic ProcessesIEEE Transactions on Information Theory10.1109/TIT.2023.331826570:1(1-15)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIT.2023.3318265
Megahed MMohammed A(2024)A comprehensive review of generative adversarial networksWIREs Computational Statistics10.1002/wics.162916:1Online publication date: 21-Jan-2024
https://dl.acm.org/doi/10.1002/wics.1629
Lu XZhu LSun Y(2023)Multi-view Subspace Clustering with Complex Noise ModelingProceedings of the 2023 9th International Conference on Computing and Data Engineering10.1145/3589845.3589854(48-57)Online publication date: 6-Jan-2023
https://dl.acm.org/doi/10.1145/3589845.3589854
Aouraghe IKhaissidi GMrabti M(2022)A literature review of online handwriting analysis to detect Parkinson’s disease at an early stageMultimedia Tools and Applications10.1007/s11042-022-13759-282:8(11923-11948)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1007/s11042-022-13759-2
Zheng MLiu LLing C(2019)On the Polarization of Rényi Entropy2019 IEEE International Symposium on Information Theory (ISIT)10.1109/ISIT.2019.8849848(2094-2098)Online publication date: 7-Jul-2019
https://dl.acm.org/doi/10.1109/ISIT.2019.8849848
Rastegin A(2019)Individual attacks with generalized discrimination and inadequacy of some information measuresQuantum Information Processing10.1007/s11128-019-2385-418:9(1-24)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s11128-019-2385-4
Rastegin A(2016)Rényi and Tsallis formulation of noise-disturbance trade-off relationsQuantum Information & Computation10.5555/3179448.317945516:3-4(313-331)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.5555/3179448.3179455
Atar RMerhav N(2015)Information–Theoretic Applications of the Logarithmic Probability Comparison BoundIEEE Transactions on Information Theory10.1109/TIT.2015.246437861:10(5366-5386)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1109/TIT.2015.2464378
Kovačević MStanojević IŠenk V(2013)Some properties of Rényi entropy over countably infinite alphabetsProblems of Information Transmission10.1134/S003294601302001449:2(99-110)Online publication date: 1-Apr-2013
https://dl.acm.org/doi/10.1134/S0032946013020014
Dewaele GHimura YBorgnat PFukuda KAbry PMichel OFontugne RCho KEsaki H(2010)Unsupervised host behavior classification from connection patternsInternational Journal of Network Management10.1002/nem.75020:5(317-337)Online publication date: 1-Sep-2010
https://dl.acm.org/doi/10.1002/nem.750

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents