The Mathematical Theory of Communication by Claude E Shannon & Warren Weaver

Author:Claude E Shannon & Warren Weaver
Language: eng
Format: mobi, epub
Publisher: University of Illinois Press
Published: 1998-09-01T07:00:00+00:00

where is the duration of the sth symbol leading from state i to state j and the Bi satisfy

then H is maximized and equal to C.

By proper assignment of the transition probabilities the entropy of symbols on a channel can be maximized at the channel capacity.

9. The Fundamental Theorem for a Noiseless Channel

We will now justify our interpretation of H as the rate of generating information by proving that H determines the channel capacity required with most efficient coding.

Theorem 9: Let a source have entropy H(bits per symbol) and a channel have a capacity C (bits per second). Then it is possible to encode the output of the source in such a way as to transmit at the average rate symbols per second over the channel where ∊ is arbitrarily small. It is not possible to transmit at an average rate greater than .

The converse part of the theorem, that cannot be exceeded, may be proved by noting that the entropy of the channel input per second is equal to that of the source, since the transmitter must be non-singular, and also this entropy cannot exceed the channel capacity. Hence H′ ≤ C and the number of symbols per second = H′/H ≤ C/H.

The first part of the theorem will be proved in two different ways. The first method is to consider the set of all sequences of N symbols produced by the source. For N large we can divide these into two groups, one containing less than 2(H+η)N members and the second containing less than 2RN members (where R is the logarithm of the number of different symbols) and having a total probability less than μ. As N increases η and μ approach zero. The number of signals of duration T in the channel is greater than 2(C–θ)T with θ small when T is large. If we choose

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.