JJ II

(1)

volume 4, issue 5, article 93, 2003.

Received 03 April, 2003;

accepted 21 October, 2003.

Communicated by:S.S. Dragomir

Abstract Contents

JJ II

J I

Home Page Go Back

Close Quit

Journal of Inequalities in Pure and Applied Mathematics

AN ENTROPY POWER INEQUALITY FOR THE BINOMIAL FAMILY

PETER HARREMOËS AND CHRISTOPHE VIGNAT

Department of Mathematics, University of Copenhagen, Universitetsparken 5, 2100 Copenhagen, Denmark.

EMail:moes@math.ku.dk

University of Copenhagen and Université de Marne la Vallée, 77454 Marne la Vallée

Cedex 2, France.

EMail:vignat@univ-mlv.fr

c

2000Victoria University ISSN (electronic): 1443-5756 043-03

(2)

An Entropy Power Inequality for the Binomial Family

Peter Harremoës and Christophe Vignat

Title Page Contents

JJ II

J I

Go Back Close

Quit Page2of13

J. Ineq. Pure and Appl. Math. 4(5) Art. 93, 2003

http://jipam.vu.edu.au

Abstract

In this paper, we prove that the classical Entropy Power Inequality, as derived in the continuous case, can be extended to the discrete family of binomial random variables with parameter1/2.

2000 Mathematics Subject Classification:94A17

Key words: Entropy Power Inequality, Discrete random variable

The first author is supported by a post-doc fellowship from the Villum Kann Ras- mussen Foundation and INTAS (project 00-738) and Danish Natural Science Coun- cil.

This work was done during a visit of the second author at Dept. of Math., University of Copenhagen in March 2003.

1. Introduction

The continuous Entropy Power Inequality

(1.1) e^2h(X)+e^2h(Y⁾ ≤e^2h(X+Y⁾

was first stated by Shannon [1] and later proved by Stam [2] and Blachman [3].

Later, several related inequalities for continuous variables were proved in [4], [5] and [6]. There have been several attempts to provide discrete versions of the Entropy Power Inequality: in the case of Bernoulli sources with addition modulo 2, results have been obtained in a series of papers [7], [8], [9] and [11].

In general, inequality (1.1) does not hold when X and Y are discrete random variables and the differential entropy is replaced by the discrete entropy: a simple counterexample is provided whenX andY are deterministic.

In what follows, Xn ∼ B n,¹₂

denotes a binomial random variable with parametersnand ¹₂,and we prove our main theorem:

Theorem 1.1. The sequenceX_nsatisfies the following Entropy Power Inequal- ity

∀m, n≥1, e^2H(Xⁿ⁾+e^2H(X^m⁾ ≤e^2H(Xⁿ^+X^m⁾.

With this aim in mind, we use a characterization of the superadditivity of a function, together with an entropic inequality.

(4)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page4of13

2. Superadditivity

Definition 2.1. A functionnyYnis superadditive if

∀m, n Y_m+n≥Y_m+Y_n.

A sufficient condition for superadditivity is given by the following result.

Proposition 2.1. If ^Y_nⁿ is increasing, thenY_nis superadditive.

Proof. Takemandnand supposem≥n. Then by assumption Y_m+n

m+n ≥ Y_m m or

Ym+n ≥Ym+ n mYm. However, by the hypothesism≥n

Y_m m ≥ Y_n

n so that

Y_m+n≥Y_m+Y_n.

In order to prove that the function

(2.1) Yn=e^2H^(Xⁿ⁾

is superadditive, it suffices then to show that functionn y ^Y_nⁿ is increasing.

(5)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page5of13

3. An Information Theoretic Inequality

Denote asB ∼Ber(1/2)a Bernoulli random variable so that

(3.1) X_n+1 =X_n+B

and

(3.2) PXn+1 =PXn ∗PB = 1

2(PXn+PXn+1), whereP_X_n ={pⁿ_k}denotes the probability law ofX_nwith

(3.3) pⁿ_k = 2⁻ⁿ

n k

.

A direct application of an equality by Topsøe [12] yields (3.4) H P_X_n+1

= 1

2H(P_X_n₊₁) + 1

2H(P_X_n) + 1

2D P_X_n₊₁||P_X_n+1 +1

2D P_X_n||P_X_n+1 .

Introduce the Jensen-Shannon divergence (3.5) J SD(P, Q) = 1

2D

P

P +Q 2

+1

2D

Q

P +Q 2

and remark that

(3.6) H(PXn) = H(PXn+1),

(6)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page6of13

since each distribution is a shifted version of the other. We conclude thus that

(3.7) H P_X_n+1

=H(P_X_n) +J SD(P_X_n₊₁, P_X_n),

showing that the entropy of a binomial law is an increasing function ofn.Now we need the stronger result that ^Y_nⁿ is an increasing sequence, or equivalently that

(3.8) log Y_n+1

n+ 1 ≥logY_n n or

(3.9) J SD(P_X_n₊₁, P_X_n)≥ 1

2logn+ 1 n .

We use the following expansion of the Jensen-Shannon divergence, due to B.Y.

Ryabko and reported in [13].

Lemma 3.1. The Jensen-Shannon divergence can be expanded as follows J SD(P, Q) = 1

2

∞

X

ν=1

1

2ν(2ν−1)∆_ν(P, Q) with

∆_ν(P, Q) =

n

X

i=1

|p_i −q_i|^2ν (pi +qi)^2ν−1.

This lemma, applied in the particular case whereP =P_X_n andQ =P_X_n₊₁ yields the following result.

(7)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page7of13

Lemma 3.2. The Jensen-Shannon divergence betweenP_X_n₊₁ andP_X_n can be expressed as

J SD(P_X_n₊₁, P_X_n) =

∞

X

ν=1

1

ν(2ν−1)· 2^2ν−1 (n+ 1)^2νm_2ν

B

n+ 1,1 2

,

where m2ν B n+ 1,¹₂

denotes the order2ν central moment of a binomial random variableB n+ 1,¹₂

.

Proof. DenoteP =p_i, Q=p⁺_i andp¯_i = (p_i+p⁺_i )/2. For the term∆_ν(P_X_n₊₁, P_X_n) we have

∆_ν(P_X_n₊₁, P_X_n) =

n

X

i=1

p⁺_i −p_i

2ν

p⁺_i +p_i2ν−1 = 2

n

X

i=1

p⁺_i −pi

p⁺_i +p_i 2ν

¯ p_i

and

p⁺_i −p_i p⁺_i +pi

= 2⁻ⁿ _i−1ⁿ

−2^{−n n}_i 2⁻ⁿ _i−1ⁿ

+ 2^{−n n}_i = 2i−n−1 n+ 1 so that

∆_ν(P_X_n₊₁, P_X_n) = 2

n

X

i=1

2i−n−1 n+ 1

2ν

¯ p_i

= 2 2

n+ 1 2ν n

X

i=1

i− n+ 1 2

2ν

¯ p_i

= 2^2ν+1 (n+ 1)^2νm2ν

B

n+ 1,1 2

.

(8)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page8of13

Finally, the Jensen-Shannon divergence becomes J SD(PXn+1, PXn) = 1

4

+∞

X

ν=1

1

ν(2ν−1)∆ν(PXn+1, PXn)

=

+∞

X

ν=1

1

ν(2ν−1)· 2^2ν−1 (n+ 1)^2νm_2ν

B

n+ 1,1 2

.

(9)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page9of13

4. Proof of the Main Theorem

We are now in a position to show that the function n y ^Y_nⁿ is increasing, or equivalently that inequality (3.9) holds.

Proof. We remark that it suffices to prove the following inequality

(4.1)

3

X

ν=1

1

ν(2ν−1) · 2^2ν−1 (n+ 1)^2νm_2ν

B

n+ 1,1 2

≥ 1 2log

1 + 1

n

since the terms ν > 3in the expansion of the Jensen-Shannon divergence are all non-negative. Now an explicit computation of the three first even central moments of a binomial random variable with parametersn+ 1and ¹₂ yields

m₂ = n+ 1

4 , m₄ = (n+ 1) (3n+ 1)

16 and m₆ = (n+ 1) (15n²+ 1)

64 ,

so that inequality (4.1) becomes 1

60

30n⁴+ 135n³+ 245n²+ 145n+ 37

(n+ 1)⁵ ≥ 1

2log

1 + 1 n

.

Let us now upper-bound the right hand side as follows log

1 + 1

n

≤ 1 n − 1

2n² + 1 3n³ so that it suffices to prove that

1

60· 30n⁴+ 135n³+ 245n²+ 145n+ 37

(n+ 1)⁵ − 1

2 1

n − 1

2n² + 1 3n³

≥0.

(10)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page10of13

Rearranging the terms yields the equivalent inequality 1

60· 10n⁵−55n⁴−63n³−55n²−35n−10 (n+ 1)⁵n³ ≥0 which is equivalent to the positivity of polynomial

P (n) = 10n⁵−55n⁴−63n³−55n²−35n−10.

Assuming first thatn ≥7,we remark that P(n)≥10n⁵−n⁴

55 + 63 6 + 55

6² + 35 6³ +10

6⁴

=

10n− 5443 81

n⁴

whose positivity is ensured as soon asn ≥7.

This result can be extended to the values1≤n ≤6by a direct inspection at the values of functionn y ^Y_nⁿ as given in the following table.

n 1 2 3 4 5 6

e^2H^(Xⁿ⁾

n 4 4 4.105 4.173 4.212 4.233 Table 1: Values of the functionny ^Y_nⁿ for1≤n≤6.

(11)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page11of13

5. Acknowledgements

The authors want to thank Rudolf Ahlswede for useful discussions and pointing our attention to earlier work on the continuous and the discrete Entropy Power Inequalities.

(12)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page12of13

References

[1] C.E. SHANNON, A mathematical theory of communication, Bell Syst.

Tech. J., 27 (1948), pp. 379–423 and 623–656.

[2] A.J. STAM, Some inequalities satisfied by the quantities of information of Fisher and Shannon, Inform. Contr., 2 (1959), 101–112.

[3] N. M. BLACHMAN, The convolution inequality for entropy powers, IEEE Trans. Inform. Theory, IT-11 (1965), 267–271.

[4] M.H.M. COSTA, A new entropy power inequality, IEEE Trans. Inform.

Theory, 31 (1985), 751–760.

[5] A. DEMBO, Simple proof of the concavity of the entropy power with re- spect to added Gaussian noise, IEEE Trans. Inform. Theory, 35 (1989), 887–888.

[6] O. JOHNSON, A conditional entropy power inequality for dependent vari- ables, Statistical Laboratory Research Reports, 20 (2000), Cambridge University.

[7] A. WYNER AND J. ZIV, A theorem on the entropy of certain binary sequences and applications: Part I, IEEE Trans. Inform. Theory, IT-19 (1973), 769–772.

[8] A. WYNER, A theorem on the entropy of certain binary sequences and applications: Part II, IEEE Trans. Inform. Theory, IT-19 (1973), 772–777.

[9] H.S. WITSENHAUSEN, Entropy inequalities for discrete channels, IEEE Trans. Inform. Theory, IT-20 (1974), 610–616.

(13)

Title Page Contents

JJ II

J I

Go Back Close

Quit Page13of13

[10] R. AHLSWEDE AND J. KÖRNER, On the connection between the en- tropies of input and output distributions of discrete memoryless channels, Proceedings of the Fifth Conference on Probability Theory, Brasov, Sept.

1974, 13–22, Editura Academiei Republicii Socialiste Romania, Bucuresti 1977.

[11] S. SHAMAI ANDA. WYNER, A binary analog to the entropy-power in- equality, IEEE Trans. Inform. Theory, IT-36 (1990), 1428–1430.

[12] F. TOPSØE, Information theoretical optimization techniques, Kyber- netika, 15(1) (1979), 8–27.

[13] F. TOPSØE, Some inequalities for information divergence and related measures of discrimination, IEEE Tr. Inform. Theory, IT-46(4) (2000), 1602–1609.