Shannon entropy inequalities

Random variables in this paper are measurable maps \(X : \Omega \to S\) from a probability space \(\Omega\) to a measurable space \(S\), and called \(S\)-valued random variables. In many cases we will assume that singletons in \(S\) are measurable. Often we will restrict further to the case when \(S\) is finite with the discrete \(\sigma\)-algebra, which of course implies that \(S\) has measurable singletons.

Entropy to Submodularity

• Entropy
• Entropy and relabeling
• Jensen bound
• Uniform distribution
• Uniform distributions exist
• Entropy of uniform random variable
• Entropy of uniform random variable, II
• Bounded entropy implies concentration
• Commutativity and associativity of joint entropy
• Conditioned event
• Conditional entropy
• Conditional entropy and relabeling
• Chain rule
• Conditional chain rule
• Mutual information
• Alternative formulae for mutual information
• Nonnegativity of mutual information
• Subadditivity
• Conditioning reduces entropy
• Submodularity

Alternate form of submodularity to Entropy of conditionally…

• Alternate form of submodularity
• Independent random variables
• Vanishing of mutual information
• Additivity of entropy
• Conditional mutual information
• Alternate formula for conditional mutual information
• Nonnegativity of conditional mutual information
• Conditionally independent random variables
• Vanishing conditional mutual information
• Entropy of conditionally independent variables

The dependency explorer needs JavaScript and its graph data.

Download the complete graph data.