Is there a proof that equal bandwidths have equal information-carrying capacity?
Consider a continuous channel, band-limited to the $[f_{\text{min}},\,f_{\text{max}}]$ frequency window. A well-known statement says that, as far as the dependence on the frequencies, the information-carrying capacity of such a channel depends only on the bandwidth $B=f_{\text{max}}-f_{\text{min}}$, and not on the magnitudes of $f_{\text{min}}$ and $f_{\text{max}}$ separately. There are also dependencies on other, non-frequency-related variables (in particular, on the signal-to-noise ratio), but this question is about how the carrying capacity depends on the frequencies, while any and all other relevant variables are held fixed.
The rigorous underpinning of all statements connecting the information-carrying capacity and the bandwidth seems to be the Shannon–Hartley theorem: $C = B \log_2 \left( 1+\frac{S}{N} \right),$ where $C$ is the information-carrying capacity and $S/N$ is the signal-to-noise ratio. However, in every (at least somewhat) careful statement and proof of the Shannon–Hartley I've seen (e.g., this one; go one page up for the definition of $W$), the bandwidth always refers to the case when $f_{\text{min}}=0$. The reason seems to be that the proof relies on the Nyquist–Shannon sampling theorem; and in the latter, the relevant bandwidth indeed always starts from zero, since that theorem is all about the rate of sampling. (This was what I had thought originally, but it turned out to be wrong. See Update 1 below.)
Now, in order to be able to use a variety of frequencies for transmitting information, one typically (always?) resorts to some kind of a modulation scheme, say of the amplitude, or of the frequency, or of the phase, or something more complicated (QAM, ASK, PSK, OFDM...). And for each known modulation scheme, it can (presumably) be shown that, indeed, what matters (as far as the relation between the information-carrying capacity and the frequencies used) is the bandwidth around the carrier signal.
But perhaps this is just because we are not sufficiently clever. Perhaps there is a scheme, as yet undiscovered, that would manage to pack, within the same bandwidth, more information in a frequency band with $f_{\text{min}}>0$ than the Shannon–Hartley allows to be packed in a band with $f_{\text{min}}=0$ (assuming the same signal-to-noise ratio in all cases).
Is there a proof that there could not be such a scheme? (As it turns out, probably yes. See Update 2 below.) In other words, is there a proof that the bandwidth that enters the Shannon–Hartley can indeed be $f_{\text{max}}-f_{\text{min}}$ with $f_{\text{min}}$ any arbitrary positive number? Equivalently: is there some generalization of the Shannon–Hartley that allows $f_{\text{min}}$ to be any positive number? (Almost certainly yes. See Update 1 below.)
Update 1: As pointed out by Omegaman and Peter K., the Nyquist–Shannon sampling theorem in fact deals with the case of $f_{\text{min}}>0$ as well. As explained on the Wikipedia entry for that theorem, "For example, in order to sample the FM radio signals in the frequency range of 100-102 MHz, it is not necessary to sample at 204 MHz (twice the upper frequency), but rather it is sufficient to sample at 4 MHz (twice the width of the frequency interval)." It would seem that this is the key to at least one way of answering this question.
Update 2: Jim Clay and especially Jason R. suggest the following line of reasoning: the theorem on reversibility states that "a reversible operation, transforming a signal into one or more waveforms, may be inserted between the channel output and the receiver without affecting the minimum attainable probability of error" (Wozencraft and Jacobs, p. 488, 1965). Now, for the purposes of the (sought-after generalized version of the) Shannon-Hartley, any sort of transformation of the signal to a channel with $f_{\text{min}}>0$ would have to be lossless, i.e. reversible. We would than need to prove the following theorem: "a reversible transformation from one channel, band-limited to $[f^{(1)}_{\text{min}},\,f^{(1)}_{\text{max}}]$, to another channel, band-limited to $[f^{(2)}_{\text{min}},\,f^{(2)}_{\text{max}}]$, preserves the signal-to-noise ratio if and only if the bandwidths of the two channels are the same, i.e. $f^{(1)}_{\text{max}}-f^{(1)}_{\text{min}}=f^{(2)}_{\text{max}}-f^{(2)}_{\text{min}}$." The required generalization of the Shannon-Hartley then follows.
At least in W&J, the theorem on reversibility, at least superficially, seems to be given in somewhat different terms than those usually used to state the Shannon-Hartley. Is the theorem on reversibility really applicable to our attempt to generalize the Shannon-Hartley?
Does anyone know a proof (that does not invoke the generalized Shannon-Hartley) of the required theorem, i.e. that a reversible transformation preserves the signal-to-noise ratio if and only if it is between channels whose bandwidths are equal? In fact, one should be able to prove that, under a reversible transformation, $\big(1+\frac{S^{(1)}}{N^{(1)}}\big)^{f^{(1)}_{\text{max}}-f^{(1)}_{\text{min}}}=\big(1+\frac{S^{(2)}}{N^{(2)}}\big)^{f^{(2)}_{\text{max}}-f^{(2)}_{\text{min}}}$. It certainly seems that a statement like this should be provable directly, i.e. without invoking the Shannon-Hartley, but I don't have a proof yet.
It also occurs to me that a direct proof of the statement $\big(1+\frac{S^{(1)}}{N^{(1)}}\big)^{f^{(1)}_{\text{max}}-f^{(1)}_{\text{min}}}=\big(1+\frac{S^{(2)}}{N^{(2)}}\big)^{f^{(2)}_{\text{max}}-f^{(2)}_{\text{min}}}$ could be a key part of a nice complement to the usual information-theoretic treatment of the information-carrying capacity. For example, you could say that, since there is a strong intuition that a reversible transformation should not affect information content, the statement above suggests that the information-carrying capacity $C$ must be a function of $\big(1+\frac{S}{N}\big)^{f_{\text{max}}-f_{\text{min}}}$. Then we demand that the capacity of two channels combined must be the sum of the capacities of the individual channels, from which, by usual arguments, it then follows that the function must be a log.
bandwidth information-theory
linguisticturn
linguisticturnlinguisticturn
I'm not a mathematician so I won't pretend to claim that this is anything like a proof, but at an intuitive level I think that the fact that you can deterministically transform baseband to passband and back again by multiplying with complex sinusoids implies that the two are equivalent.
Jim ClayJim Clay
$\begingroup$ Thanks. However, isn't any lossless compression algorithm a counterexample to this intuitive principle? $\endgroup$ – linguisticturn Jan 7 '16 at 2:31
$\begingroup$ @linguisticturn, generic data cannot be losslessly compressed. data that shows certain patterns (like it's text) can be losslessly compressed. but you cannot just take some arbitrary data stream and hope to losslessly compress it. $\endgroup$ – robert bristow-johnson Jan 7 '16 at 3:06
$\begingroup$ and i dunno that Jim's answer really speaks to the question. $\endgroup$ – robert bristow-johnson Jan 7 '16 at 3:08
$\begingroup$ Yes, I know, but typicality is part of the game here... The proposed intuitive principle is, "If there is any reversible map between two media, then their information-carrying capacities are the same." Counterexample: consider e.g. the (uncompressed) .wav and the (losslessly compressed) .flac music files. Although one can "deterministically transform" one into another, they are not equivalent: a typical digitized piece of music in .flac will be only about half the size it would be in .wav. $\endgroup$ – linguisticturn Jan 7 '16 at 4:42
$\begingroup$ @linguisticturn sorry for mistaking that Fourier argument. Now coming to your claim of $b(k)=w(Wk/B)$ you seem to ignore noise from your model of the channel. Look at the formula, if there is no noise then S/N is inifite and capacity is infinite, and then you are right, you can transmit the most complex functions in the smallest possible bandwidth by the frequency-scaling you propose. But if you assume any nonzero noise, then such a scaling should I geuss change the signal to noise ratio of the channel and assumption of everything being the same will be violated. $\endgroup$ – Fat32 Jan 8 '16 at 0:26
You can probably find the proof you need looking at Nyquist's original paper.
In short, the analysis comes down to converting the information signal into a series of Fourier components, each a sinusoid of a given frequency and phase. Equal bandwidths, by definition, accommodate an equal number of sinusoids of different frequencies regardless of their offset in the spectrum.
Shannon's contribution was to essentially figure out the limit on how many levels of those sinusoids can be distinguished in the presence of noise.
OmegamanOmegaman
$\begingroup$ Thank you! As mentioned in another answer below, the following is found on Wikipedia: "For example, in order to sample the FM radio signals in the frequency range of 100-102 MHz, it is not necessary to sample at 204 MHz (twice the upper frequency), but rather it is sufficient to sample at 4 MHz (twice the width of the frequency interval)." This must indeed be the key. After I have thought it through some more, I'll make the appropriate edits. $\endgroup$ – linguisticturn Jan 7 '16 at 21:14
Your statement
The reason seems to be that the proof relies on the Nyquist–Shannon sampling theorem; and in the latter, the relevant bandwidth indeed always starts from zero,
is incorrect.
A similar result is true if the band does not start at zero frequency but at some higher value, and can be proved by a linear translation (corresponding physically to single-sideband modulation) of the zero-frequency case. In this case the elementary pulse is obtained from sin(x)/x by single-side-band modulation.
Peter K.♦Peter K.
$\begingroup$ Thank you! I discovered that very passage on Wikipedia a moment ago myself... "For example, in order to sample the FM radio signals in the frequency range of 100-102 MHz, it is not necessary to sample at 204 MHz (twice the upper frequency), but rather it is sufficient to sample at 4 MHz (twice the width of the frequency interval)." This must indeed be the key. After I have digested it fully, I'll make the appropriate edit to the original question, and give your answer the vote. $\endgroup$ – linguisticturn Jan 7 '16 at 21:07
$\begingroup$ @linguisticturn You're welcome! $\endgroup$ – Peter K.♦ Jan 8 '16 at 0:00
$\begingroup$ @downvoter: what's the problem? $\endgroup$ – Peter K.♦ Jan 8 '16 at 0:00
$\begingroup$ Incidentally, I'm trying to upvote this answer, but I think that, as a newbie, I lack reputation points for my upvote to take effect... (I hope it's clear that it wasn't me who downvoted. :) ) $\endgroup$ – linguisticturn Jan 8 '16 at 0:05
$\begingroup$ @linguisticturn : Ya, I figured you weren't downvoting! :-) No worries; I have a reasonable rep. I just get annoyed with anonymous downvoters when I'm pretty sure I have the right of it. $\endgroup$ – Peter K.♦ Jan 8 '16 at 12:39
Not the answer you're looking for? Browse other questions tagged bandwidth information-theory or ask your own question.
Will the capacity of a channel becomes unbounded if i increase its signal-to-noise ratio $S/N$ without limit?
What is the definition of the bandwidth of a signal?
"Capacity" concept in Shannon Theorem
How to reconcile "bandwidth" in the Shannon-Hartley Theorem with a spread-spectrum principle
Why Shannon Theorem has nothing to do with frequency?