text:
WO2011050853A1 - Coding of multi-channel signals - Google Patents
Coding of multi-channel signals Download PDF
signal components
Juha Petteri OJANPERÄ
2009-10-30 Application filed by Nokia Corporation filed Critical Nokia Corporation
2009-10-30 Priority to PCT/EP2009/064380 priority Critical patent/WO2011050853A1/en
238000004458 analytical methods Methods 0 claims description 354
230000015572 biosynthetic process Effects 0 claims description 83
238000003786 synthesis Methods 0 claims description 83
230000002194 synthesizing Effects 0 claims description 83
230000001131 transforming Effects 0 claims description 20
238000000513 principal component analysis Methods 0 description 30
238000000354 decomposition Methods 0 description 2
G10—MUSICAL INSTRUMENTS; ACOUSTICS
G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
G10L19/008—Multichannel audio signal coding or decoding, i.e. using interchannel correlation to reduce redundancies, e.g. joint-stereo, intensity-coding, matrixing
A system comprises: first and second base stations; and a plurality of relay nodes, each of said relay nodes connected to the first base station, each of said relay nodes being connected to at least one other relay node, whereby at least one relay node is configured to at least one of receive and send information for another of said relays nodes; wherein when at least one of the plurality of relay nodes is handed over to a second base station the at least one relay node is configured to receive and / or send information via another of the relay nodes connected to the first base station.
Coding of Multi-Channel Signals
FIELD OF THE DISCLOSURE
This invention relates to the field of coding of multi-channel signals.
A multi-channel audio signal can be captured using multiple microphones within common acoustic space or created in a synthetical manner by combining a number of possibly unrelated audio signals. A multi-channel audio signal may comprise any number of channels, common channel
configurations are for example "traditional two-channel stereo", 5.1 or 7.2 channel configurations commonly used in consumer electronics. Typically, coding of multi-channel audio signals at high-quality requires high bit-rate, which may not be feasible in all applications and operational scenarios .
An emerging new type of use case for audio capture and further processing is multiview audio, which is a concept that provides different aural views to an audio scene, from which (e.g.) a user can select the one he/she prefers.
In principle, "traditional" multi-channel audio can be seen as a subset of multiview audio, implying that generic multi-channel audio coding techniques may be applied to multiview audio to introduce relatively low bit-rates. SUMMARY OF SOME EMBODIMENTS OF THE INVENTION A first method is described, which comprises performing a concatenated component analysis based on a time-frequency representation of a multi-channel signal for determining a down-sampled representation of the multi-channel signal, wherein said concatenated component analysis comprises applying a first component analysis for determining a first set of signal components representing the multi-channel signal, and applying a second component analysis based on the first set of signal components for determining a second set of signal components representing the multi-channel signal.
Moreover, a first apparatus is described, which comprises means for performing a concatenated component analysis based on a time-frequency representation of a multi-channel signal for determining a down-sampled representation of the multi-channel signal, wherein said concatenated component analysis is performed by means for applying a first component analysis for determining a first set of signal components representing the multi-channel signal and by means for applying a second component analysis based on the first set of signal components for determining a second set of signal components representing the multi-channel signal.
The means of this apparatus can be implemented in hardware and/or software. They may comprise for instance a processor for executing computer program code for realizing the required functions, a memory storing the program code, or both. Alternatively, they could comprise for instance a circuit that is designed to realize the required functions, for instance implemented in a chipset or a chip, like an integrated circuit. Moreover, a second apparatus is described, which comprises at least one processor and at least one memory including computer program code, the at least one memory and the computer program code, with the at least one processor, configured to cause the apparatus at least to perform the actions of the presented first method.
Moreover, a computer readable storage medium is described, in which computer program code is stored. The computer program code causes an apparatus to realize the actions of the presented first method when executed by a processor.
The computer readable storage medium could be for example a disk or a memory or the like. As an example, the memory may represent a memory card such as SD and micro SD cards or any other well-suited memory cards or memory sticks. The computer program code could be stored in the computer readable storage medium in the form of instructions encoding the
computer-readable storage medium. The computer readable storage medium may be intended for taking part in the operation of a device, like an internal or external hard disk of a computer, or be intended for distribution of the program code, like an optical disc.
A second method is described, which comprises performing a concatenated reconstruction of a multi-channel signal based on a down-sampled representation of the multi-channel signal, wherein said concatenated reconstruction comprises applying a first component synthesis for determining a first set of reconstructed signal components representing the
multi-channel signal; and applying a second component synthesis based on the first set of reconstructed signal components for determining reconstructed representation of the multi-channel signal.
Moreover, a third apparatus is described, which comprises means for performing a concatenated reconstruction of a multi-channel signal based on a down-sampled representation of the multi-channel signal, wherein said concatenated reconstruction is performed by means for applying a first component synthesis for determining a first set of reconstructed signal components representing the
multi-channel signal and means for applying a second component synthesis based on the first set of reconstructed signal components for determining reconstructed
representation of the multi-channel signal. The means of this third apparatus can be implemented in hardware and/or software. They may comprise ' for instance a processor for executing computer program code for realizing the required functions, a memory storing the program code, or both. Alternatively, they could comprise for instance a circuit that is designed to realize the required functions, for instance implemented in a chipset or a chip, like an integrated circuit.
Moreover, a fourth apparatus is described, which comprises at least one processor and at least one memory including computer program code, the at least one memory and the computer program code, with the at least one processor, configured to cause the apparatus at least to perform the actions of the presented second method.
Moreover, a computer readable storage medium is described, in which computer program code is stored. The computer program code causes an apparatus to realize the actions of the presented second method when executed by a processor.
Moreover, a system is described, comprising an apparatus according to one of the first and second apparatus and comprising a further apparatus according to one of the third and fourth apparatus .
The multi-channel signal may comprise at least two channels, wherein each of the at least two channels is associated with a signal.
For instance, the multi-channel signal may represent a "traditional two-channel stereo" signal, or a 5.1 or 7.2 channel configuration or any other multi-channel
configuration. Furthermore, as an example, the multi-channel signal may represent a multi-view signal, wherein different channels of the multi-channel signal are associated with different aural views to an audio scene. Thus, each of said different channels is associated with a signal being associated with the respective aural view of the different aural views .
As an example, the multi-channel signal may represent a mixture of a multi-view signal and at least one further signal. Furthermore, the multi-channel signal may represent a multi-channel audio signal or a multi-channel video signal or any other kind of multi-channel signal.
For instance, the multi-channel signal may represent the signal of closely spaced microphones all pointing toward a different angle relative to the forward axis which may be used to record an audio scene in accordance with a multi-view audio system. For instance, the multi-channel signal may be in the time-frequency domain. In this case, the multi-channel signal may be directly processed by the first apparatus.
As another example, the first or second apparatus may comprise a converter configured to transform the multi-channel signal into the time- frequency representation.
The first component analysis may represent an analysis procedure which is configured to perform a first
decorrelation of the time- frequency representation of the multi-channel signal. For instance, this first decorrelation may be directed to each of at least one channel of the at least two channels of the multi-channel signal, i.e. the first decorrelation may be performed to decorrelate each channel of at least one channel of the at least two channels separately, thereby representing an intra-channel
decorrelation. Futhermore, as an example, this first decorrelation may perform a decorrelation between at least two channels of the at least two channels of the multi -channel signal, thereby representing an inter-channel decorrelation. Furthermore, as an example, the first component analysis may determine a set of first analysis components, wherein this set of first analysis components is configured to be used in combination with the first set of signal components for reconstruction of the multi-channel signal. For instance, the multi-channel signal may be reconstructed by a linear combination of the first analysis components and the first set of signal components. The first set of signal components represents the
multi-channel signal after this first component analysis has been performed. Thus the multi-channel signal (or an approximation of the multi-channel signal) may be
reconstructed by means of a corresponding component synthesis applied to the first set of signal components, the first component synthesis representing the inverse operation of the first component analysis.
For instance, the first component analysis may by performed by means of a principal component analysis (PCA) or an independent component analysis (ICA), but any other well-suited component analysis may also be used for carrying out the first component analysis. Due to the first component analysis the output data rate of the determined first set of signal components may be reduced compared to the input data rate of the time-frequency domain representation lof the multi-channel signal. The second component analysis may be different compared to first component analysis. For instance, the first component analysis may perform a first decorrelation of the time-frequency representation of the multi-channel signal, as mentioned above, and the second component analysis may perform a second decorrelation of the multi-channel signal based on the first set of signal components, wherein the first decorrelation differs from the second decorrelation. As an example, the second decorrelation may be one of the above-mentioned inter-channel decorrelation and
intra-channel decorrelation and the first decorrelation may be remaining inter-channel or intra-channel decorrelation.
Furthermore, as an example, the second component analysis may determine a set of second analysis components, wherein this set of second analysis components is configured to be used in combination with the second set of signal components for reconstruction of the first set of signal components. For instance, the first set of signal components may be reconstructed by a linear combination of the second analysis components and the second set of signal components. The second set of signal components represents the multi-channel signal after this second component analysis has been performed. Thus the first set of signal component (or an approximation of the first set of signal components) may be reconstructed by means of a corresponding component synthesis applied to the second set of signal components the second component synthesis representing the inverse operation of the second component analysis.
For instance, the second component analysis may by performed by means of a principal component analysis (PCA) or an independent component analysis (ICA) , but any other well-suited component analysis may also be used for carrying out the second component analysis.
Thus, due to the concatenated component analysis different statistical properties of the multi-channel signal may be used for a two-stage component analysis in order to reduce the data rate of the multi-channel signal, wherein the first component analysis is directed to exploit one of the different statistical properties and the second component analysis is directed to exploit another of the different statistical properties .
For instance, the multi-channel signal is associated with at least two channels. Furthermore, as an example, the first component analysis comprises for each channel of at least one channel of the at least two channels applying an intra-channel component analysis for determining a subset of signal components representing the signal of the respective channel. The down-sampled representation of the multi-channel signal may comprise the second set of signal components. Due to the two-stage decomposition of the multi-channel signal the data rate of the down-sampled representation of the multi-channel signal is reduced compared to the inputted multi-channel signal.
The third or fourth apparatus may be configured to have access to a signal which may represent or comprise the down-sampled representation of the multi-channel signal. This access to the signal may be represented any well-suited access, i.e. this signal may represent any kind of accessed signal representing or comprising the down-sampled representation of the multi-channel signal. As an example, the signal may be stored in a kind of memory, or it may be transmitted from another functional entity to the third or fourth apparatus. For instance, the third or fourth apparatus may be configured to receive the signal. As an example, the third or fourth apparatus may be configured to receive the signal after being transmitted over a channel.
The first component synthesis may represent an inverse operation of the second component analysis. Thus the first set of signal components (or an approximation of the first set of signal components) may be reconstructed by means of the first component synthesis applied to the second set of signal components, which are included in the down-sampled representation of the multi-channel signal of the accessed signal .
Furthermore, as an example, the first component synthesis may be performed based on the set of second analysis components which may be used in combination with the accessed second set of signal components.
The second component synthesis may represent an inverse operation of the first component analysis. Thus, for instance, the time-frequency domain representation of the multi-channel signal may be reconstructed by means of the second component synthesiser as reconstructed representation of the multi-channel signal based on the first set of reconstructed signal components .
Furthermore, as an example, the second component synthesis may be performed based on the set of first analysis components which may be used in combination with the first set of reconstructed signal components for reconstruction of the multi-channel signal. For instance, the third or fourth apparatus may be used in a receiver in order to reconstruct the multi-channel signal.
According to a further aspect, the multi-channel signal is associated with at least two channels, the first component analysis comprises for each channel of at least one channel of the at least two channels applying an intra-channel component analysis for determining a subset of signal components of said first set of signal components
representing the signal of the respective channel, and the second component analysis represents an inter-channel component analysis of at least two channels of the at least two channels associated with the multi-channel signal.
For instance, one of these at least two audio channels may be selected and an intra-channel component analysis may be applied for determining a subset of signal components representing the signal of the respective channel. The intra-channel component analysis may be carried out as explained above, for example based on PCA or any other well-suited component analysis. Thus, a decorrelation of the signal of the respective channel may be performed.
Furthermore, as an example, the intra-channel component analysis may determine a subset of first analysis components, wherein this subset of first analysis components is configured to be used in combination with the subset of signal components for reconstruction of the signal of the respective channel of the multi-channel signal, e.g. by means of a linear combination. The set of first analysis components may comprise this subset of first analysis components. Then it may be checked whether a component analysis is to be performed for a next channel of the at least two channels. If there is a next channel for component analysis the method may proceed with selecting this channel and applying the intra-channel component analysis for determining a subset of signal components representing the signal of the selected channel .
The signal components of each determined subset of signal components being associated with the respective channel are signal components of the first set of signal components. Accordingly, the first set of components comprises the at least one subset of signal components determined by the respective intra-channel component analysis of the
respective channel.
For instance, this may be performed for each of the at least one channel of the at least two channels of the multi-channel signal in order to determine at least one subset of signal components of the first set of signal components.
Furthermore, as an example, the time-frequency
representation may comprise at least two sets of
time-frequency representatives, wherein each set of the at least two sets of time-frequency representatives is associated with one channel of the at least two channels of the multi-channel signal. Furthermore, for instance, in case the intra-channel component analysis is not applied for at least one channel of the at least two channels of the multi-channel signal, the signal of this at least one channel may be represented by the respective time-frequency representatives. For instance, the set of first signal components may comprise at least one subset of signal components being associated with said at least one channel for which no intra-channel component analysis has been applied, wherein each of this at least one subset of signal components comprises the respective set of time-frequency representatives being associated with the respective channel.
Accordingly, as an example, the intra-channel component analysis may be performed based on the set of time-frequency representatives of the at least two sets of time-frequency representatives associated with the respective channel.
For instance, the inter-channel component analysis may be performed on the basis of the determined at least one subset of signal components of the first set of signal components which has been determined by the first component analysis as mentioned above. As an example, the inter-channel component analysis may be applied for at least two channels of the at least two channels of the multi-channel signal in order to determine signal components of the second set of signal components
representing the signals of these at least two channels of the at least two channels of the multi-channel signal. This determining may be performed on basis of the respective subset of signal components of the first set of signal components being associated with the at least two channels of the at least two channels of the multi-channel signal.
Furthermore, as another example, in case the inter-channel component analysis is not applied for at least one channel of the at least two channels of the multi-channel signal, the signal of this at least one channel may be represented by the respective signal components of the first set of signal components .
The inter-channel component analysis may be carried out as explained above, for example based on PCA or any other well-suited component analysis. Thus, a decorrelation of the at least two channels of the at least two channels of the multi-channel signal may be performed.
Furthermore, as an example, the inter-channel component analysis may determine analysis components of the set of second analysis components, wherein these determined analysis components are configured to be used in combination with the second set of signal components for reconstruction of the signal of the at least one channel of the multi-channel signal, e.g. by means of a linear combination. According to a further aspect, the first component synthesis represents an inter-channel synthesis of at least two channels of the at least two channels associated with the multi-channel signal, and the second component synthesis comprises for each channel of at least one channel of the at least two channels applying an intra-channel component synthesis for determining a reconstructed signal of the respective channel. This reconstructed signal of the respective channel may represent the above-mentioned set of time-frequency representatives being associated with the respective channel.
According to a further aspect, each channel of the at least one channel of the at least two channels is associated with a frequency band, and the intra-channel component analysis for a respective channel comprises applying a component analysis for each subband of at least one subband of the frequency band of the respective channel for determining a subsubset of signal components associated with the respective subband of the respective channel for each subband of the at least one subband, a subsubset of signal components representing signal components of the first set of signal components representing the signal in the respective subband of the respective channel of the multi-channel signal. Each channel of the at least one channel of the at least two channels is associated with a frequency band. The width of the frequency band of one channel may depend on the signal associated with the respective channel and can either be fixed or variable.
Furthermore, the frequency band of one channel of the at least one channel is associated with at least one subband of the frequency band of the respective channel. A component analysis is applied for each subband of at least one subband of the frequency band of the respective channel for determining a subsubset of signal components for each subband of the at least one subband of the respective channel. A subsubset of signal components represents signal components of the first set of signal components representing the signal in the respective subband of the respective channel of the multi-channel signal.
Furthermore, as an example, the component analysis associated with a subband of the at least one subband of the frequency band of the respective channel may determine a subsubset of first analysis components, wherein this subsubset of first analysis components is configured to be used in combination with the respective subsubset of signal components for reconstruction of the signal of respective subband of the respective channel of the multi-channel signal, e.g. by means of a linear combination. The set of first analysis components may comprise this subsubset of first analysis components.
This determining may be performed based on the time-frequency representatives of the set of time-frequency representatives of the respective channel and being associated with the respective subband of the respective channel. For instance, each set of time-frequency representatives associated with one channel may comprise at least one subset of time-frequency representatives, wherein each subset of time-frequency representatives may be associated with one subband of the respective channel and may comprise time-frequency
representatives representing the signal in the respective subband of the respective channel. For instance, the above-mentioned subset of signal components being associated with the respective channel may comprise the at least one subsubset of signal components determined by means of a component analysis for one channel. Accordingly, each subsubset of signal components comprises signal components representing the signal in the subband of the respective channel determined by the applied component analysis .
Thus, for example, a plurality of intra-channel component analysis may be performed for different subbands of a frequency band of a respective channel. For instance, a decorrelation of signals in different subbands of a respective channel may be performed, which may enhance the quality of decorrelation.
According to a further aspect, the intra-channel component synthesis for a respective channel comprises applying a component synthesis for each subband of at least one subband of the frequency band of the respective channel for determining a subsignal of reconstructed signal components associated with the respective subband of the respective channel for each subband of the at least one subband.
For instance, this reconstructed subsignal may represent reconstructed time-frequency representatives representing the signal in the respective subband of the respective channel.
Furthermore, as an example, the intra-channel first component synthesis for a respective channel comprises applying a component synthesis may be performed based on a respective subset of first analysis components. According to a further aspect, the inter-channel component analysis comprises applying at least one component analysis for determining at least one subset of signal components of the second set of signal components, wherein each of the at least one component analysis is associated with at least two channels of the at least two channels of the multi-channel signal and with a subband of a frequency band associated with the respective at least two channels of the at least two channels, each determined subset of signal components representing signal components of the second set of signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband.
Thus, each determined subset of signal components may comprise signal components of the second set of signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband.
The determining is based on the first set of signal components. Accordingly, signal components of the first set of signal components representing the signals of the respective at least two channels in the respective subband may be used as basis for the component analysis of the inter-channel component analysis.
For instance, under the assumption of the exemple notation regarding the subsubset of signal components of the first set of signal components, one component analysis of the at least one component analysis of the inter-channel component analysis may be applied based on the respective at least two subsubsets of signal components of the first set of signal components, wherein a subsubset of signal components of the at least two subsubsets of signal components is associated with the respective subband of the respective channel of the respective at least two channels.
As an example, the inter-channel component analysis may comprise a component analysis for each subband of a frequency band associated with the at least two channels.
Furthermore, as an example, a component analysis of the at least one component analysis of the inter-channel component analysis may determine a subset of second analysis components of the set of second analysis components, wherein the subset of second analysis components is configured to be used in combination with the respective subset of second set of signal components for reconstruction of the respective at least two subsubsets of signal components of the first set of signal components .
According to a further aspect, the inter-channel component synthesis comprises applying at least one component synthesis for determining at least one subset of reconstructed signal components of the first set of reconstructed signal components, wherein each of the at least one component synthesis is associated with at least two channels of the at least two channels of the multi-channel signal and with a subband of a frequency band associated with the respective at least two channels of the at least two channels, each determined subset of reconstructed signal components representing reconstructed signal components of the first set of reconstructed signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband.
For instance, each determined subset of reconstructed signal components may represent a reconstructed subsubset of signal components associated with the respective subband of the respective channel of the respective at least two channels.
Furthermore, as an example, the first component synthesis may be performed based on the set of second analysis components, as explained above.
According to a further aspect, converting a time-domain representation of the multichannel signal to said
time-frequency representation of the multi-channel signal is performed, the time-frequency representation comprising at least two sets of time-frequency representatives, each set of the at least two sets of time-frequency representatives being associated with one channel of the at least two channels .
For instance, the multi-channel signal may represent this time-domain representation of the multi-channel signal. The time-frequency representation comprises at least two sets of time-frequency representatives, each set of the at least two sets of time-frequency representatives being associated with one channel of the at least two channels of the multi-channel signal. Furthermore, each set of the at least two sets of time-frequency representatives may comprise at least one subset of time-frequency representatives, wherein each time-frequency representative of a subset of the at least one subset is associated with a frequency component of the frequency band of the respective channel and with different point in times. For instance, the time-frequency
representatives of a subset of the at least one subset are associated with the respective frequency component and a time frame .
For instance, the frequency component of the frequency band of the respective channel may correspond to the
above-mentioned subband of the frequency band of the respective channel, but, as another example providing a higher resolution in frequency, one subband of the frequency band of the respective channel my be associated with at least two frequency components .
For instance, this converting may be based on a Fourier Transformation, e.g. implemented by means of a Fast Fourier Transformation (FFT) or a Discrete Fourier Transformation (DFT) or any other well-suited Fourier Transformation, and/or by means of a Discrete Cosine Transformation (DCT) and/or by means of any other suited transformation.
Accordingly, the time-frequency representation of the multi-channel signal may be used for one or more of the above-mentioned component analysis.
As an example, applying a component analysis for a respective subband of a respective channel for determining a subsubset of signal components of the first set of signal components may be performed based. on the set of time-frequency representatives associated with this respective channel and with at least one subset of time-frequency representatives of this set of time-frequency representatives, the at least one subset of time-frequency representatives being
asscociated with the respective subband. I.e., the at least one frequency component associated with the at least one subset of time-frequency representatives are associated with the respective subband of the respective channel.
Furthermore, with respect to the second method or to the third or fourth apparatus, one of these apparatuses may comprise a converter representing an inverse converter with respect to above mentioned conversion.
According to a further aspect, said converting comprises for each of the at least two channels transforming the time-domain representation of the respective channel of the time-domain representation into a frequency domain representation; and performing a two-dimensional discrete cosine transformation based on the frequency domain representation associated with the respective channel in order to determine the set of time-frequency representatives associated with the
For instance, the time-domain representation of a channel m of the at least two channels may be represented by xm(k) . A frame 1 of time-domain representations xm(t) may converted by a first transformer to a respective frequency domain representation Xm[k,l], where k is the frequency component index (e.g. k may represent a frequency bin index) and wherein TF may represent a corresponding time-to-frequency operator:
Xm [k,l] = TF (xmJ ) (1) For instance, this may be performed for each of the channels.
As an example, a Modified Discrete Cosine Transformation may be used for the first transformer, as exemplarily explained in the sequel :
For instance and as non-limiting example, the TF operator may be applied to each signal segment according to
Xm[k ]= TF(xmJ.) (2) where m is the channel index, k is the frequency bin index, 1 is time frame index, T is the hop size between successive time frames, and TF the time-to-frequency operator. A MDCT may be used as the TF operator as follows
TF(xmJj)=2
(3) xin{n)=w{n)-xm(n +hT) where w(n) is the N-point analysis window such as sinusoidal or Kaiser-Bessel Derived (KBD) window. In MDCT, the hop size is T=N/2.
For instance, assuming that the at least two channels are M channels, the frequency domain representation of the multi-channel signal is represented by m[&,/] with 0 < m ≤
M. A second transformer may be configured, for each channel of the at least two channels, to perform a two-dimensional discrete cosine transformation (2D-DCT) based on the frequency domain representation associated with the respective channel in order to determine the set of time-frequency representatives associated with the respective channel .
For instance, the set-of time-frequency representative of channel m may be represented by matrix 7m [£,t] , where t is the time-index. Accordingly, k is the index of the respective frequency component and all representatives of ^"[A;,/] with a fixed k and a fixed m represent the above-mentioned subset of the time-frequency representatives being associated with channel m and frequency component k .
As an example, the operation of the second transformer may be applied to the frequency domain representation based on a 2D-.DCT as follows:
start < u < t end t _ start = grpldx■ TF _ size
t end - t start + TF size
grpldx = 0,1,2,3,...
where TF_size is the size of the 2D time-frequency plane. The size of matrix Ym may therefore be TF_size x A. According to one aspect, the first method comprises extracting from each of at least one set of signal components of the first set of signal components a separate subset of relevant signal components, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the first set of signal components in accordance with a first accuracy criteria; and extracting from each of at least one set of signal components of the second set of signal components a separate subset of relevant signal components, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the second set of signal components in accordance with a second accuracy criteria .
Each of the at least one set of signal components may be associated with one component analysis of the first component analysis . For instance, one set of the at least one set of signal components may represent a subset of signal components of the first set of signal components determined by means of the intra-channel component analysis. In this exemple case, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective channel of the multi-channel signal. Or, as another example, one set of the at least one set of signal components may represent a subsubset of signal components of the first set of signal components determined by means of the component analysis for a respective subband of a respective channel. In this exemple case, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective subband of the respective channel of the multi-channel signal. The first accuracy criteria may represent any well-suited criteria configured to determine the quality of the part of the multi-channel signal reconstructed by means of the respective set of relevant signal components. Furthermore, for instance, first extraction information may be provided indicative which signal components of a set of the at least one set of signal components have been extracted to the respective subset of relevant signal components. Further, as an example, this first extraction information may be used in order to reconstruct the first signal components before the second component synthesis is performed.
Thus, for instance, only those signal components of a set of the at least one set of signal components are selected which are sufficient to represent the respective part of the multi-channel signal at desired accuracy, whereas the remaining signal components may be discarded. These selected signal components may define the corresponding subset of relevant signal components. For instance, the first extraction information may comprise the number of selected components of a set of the at least one set of signal components .
According to an aspect of the invention, said extracting comprises at least one of (a) determining a measure of relevance for each signal component of each of the at least one set of signal components of the first set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the first set of signal components; and (b) determining a measure of relevance for each signal component of each of the at least one set of signal components of the first set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the first set of signal components. For instance, the measure of relevance associated with a signal component may represent a variance. For instance, this variance may be computed by means of the respective component analysis when determining the first set of signal components, or when determining a subset of signal components of the first set of signal components, or when determining a subsubset of signal components of the first set of signal components.
For instance, Vm (/') may represent the variance of the i-th signal component of a set of the at least one set of signal components of the first set of signal components, the set of the at least one set being associated with the m-th channel and subband fb. Accordingly, for this example, a set of the at least one set represents a subsubset of signal components as mentioned above. Furthermore, it may be assumed that variances Vm fh () have been sorted in decreasing order. Then, the subset of relevant components of the respective set of the at least one set of signal components may be extracted based on the following pseudo-code: aSum = 0
2 length(Vm <],, )~\
tSum = ∑VmJb (i)
3 for i=0 to length ( Vm β ) -1
4 aSum = aSum + Vm β{ί) ;
6 if aSum / tSum > thr_ind
7 exit for loop
Vldxm,jb = i Thus, this exemple code may determine how many signal components of the set of the at least one set of signal components of the first set of signal components are needed such that the accumulated variance divided by the sum of all variances exceeds the first accuracy criteria thr__ind. For instance, this first accuracy criteria thr_ind may be set to 0.9999, but any other well-suited threshold may also be used. This first accuracy criteria may indicate that signal components with large associated variances represent significant dynamics in the audio/video scene, while those with lower variances represent less detailed information and can be discarded. Accordingly, only the signal components of the set of the at least one set of signal components being associated with the vldxm β highest variances are selected for the respective subset of relevant signal components.
For instance, in the set of first signal components at least one set of the at least one set of signal components of the first set of signal components may be replaced by the respective at least one subset of relevant signal component.
Furthermore, a second embodiment of an extracting method may be performed comprising extracting from each of at least one set of signal components of the second set of signal components a separate subset of relevant signal components, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the second set of signal components in accordance with a second accuracy criteria .
Each of the at least one set of signal components of the second set of signal components may be associated with one component analysis of the second component analysis. For instance, one set of the at least one set of signal components of the second set of signal components may represent a subset of signal components of the second set of signal components determined by means of the inter-channel component analysis. In this exemple case, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective at least two channels of the multi-channel signal. Or, as another example, one set of the at least one set of signal components of the second set of signal components may represent a subsubset of signal components of the second set of signal components determined by means of the component analysis for a respective subband of respective at least two channels. In this exemple case, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective subband of the respective at least two channel of the multi-channel signal.
Furthermore, for instance, second extraction information may be provided indicative which signal components of a set of the at least one set of signal components of the second set of signal components have been extracted to the respective subset of relevant signal components.
Further, as an example, this second extraction information may be used by in order to reconstruct the second signal components before the first component synthesis is performed.
The second accuracy criteria may represent any well-suited criteria configured to determine the quality of the part of the multi-channel signal reconstructed by means of the respective set of relevant signal components.
Thus, for instance, only those signal components of a set of the at least one set of signal components are selected which are sufficient to represent the respective part of the multi-channel signal at desired accuracy, whereas the remaining signal components may be discarded. These selected signal components may define the corresponding subset of relevant signal components. The number of selected components of a set of the at least one set of signal components of the second set of signal components may represent an extraction information associated with the respective set of the at least one set of signal components.
For instance, a measure of relevance may be determined for each signal component of each of the at least one set of signal components of the second set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the second set of signal components. As an example, the measure of relevance associated with a signal component may represent a variance. For instance, this variance may be computed by means of the respective component analysis when determining the second set of signal
components, or when determining a subset of signal components of the second set of signal components, or when determining a subsubset of signal components of the second set of signal components .
For instance, V_mv^{t) may represent the variance of the i-th signal component of a set of the at least one set of signal components of the second set of signal components, the set of the at least one set being associated with at least two channels of the at least two channels and with subband fb. Accordingly, for this example, a set of the at least one set represents a subsubset of signal components as mentioned above. As a non-limiting example, it may be assumed that the set of the at least one set of signal components of the second set of signal components is associated with all channels of the at least two channels of the multi-channel signal.
Furthermore, it may be assumed that variances Vm jb {i have been sorted in decreasing order.
Then, the subset of relevant components of the respective set of the at least one set of signal components may be extracted based on the following pseudo-code:
1 aSum = 0
2 lengfh(V_mv
tSum = V _ mvfh (/')
3 for i=0 to length ( V __mv Λ ) -1
6 if aSum / tSum > thr_ind2
10 vldx _mvfb = i
Thus, this exemple code may determine how many signal components of the set of the at least one set of signal components of the second set of signal components are needed such that the accumulated variance divided by the sum of all variances exceeds the second accuracy criteria thr__ind2. For instance, this second accuracy criteria thr_ind2 may be set to 0.9995, but any other well-suited threshold may also be used. This second accuracy criteria may indicate that signal components with large associated variances represent significant dynamics in the audio/video scene, while those with lower variances represent less detailed information and can be discarded. Accordingly, only the signal components of the set of the at least one set of signal components being associated with the vldx^mv^ highest variances are selected for the respective subset of relevant signal components of the second set of signal components.
For instance, in the set of second signal components at least one set of the at least one set of signal components of the second set of signal components may be replaced by the respective at least one subset of relevant signal components. The second embodiment of an extracting method may be performed after the second component analysis is performed. For instance, applying the second component analysis may be based on the at least one extracted subset of relevant signal components of the first set of signal components.
For instance, the down-sampled representation of the multi-channel signal may comprises the at least one extracted subset of relevant signal components of the second set of signal components.
Accordingly, a further data rate reduction may be performed based on the described extracting.
According to a further aspect, the first method comprising at least one of: applying the second component analysis based on the at least one extracted subset of relevant signal components of the first set of signal components; and the down-sampled representation of the multi-channel signal comprises the at least one extracted subset of relevant signal components of the second set of signal components.
According to a further aspect, the first method comprising determining signal scene information, the signal scene information comprising the down-sampled representation of the multi-channel signal, a first set of analysis components associated with the first component analysis, wherein the first set of analysis components is configured to be used in combination with the first set of signal components for reconstruction of the multi-channel signal, and a second set of analysis components associated with the second component analysis, wherein the second set of analysis components is configured to be used in combination with the second set of signal components for reconstruction of the first set of signal components .
Furthermore, for instance, the signal scene information may comprise the first and/or second extraction information.
According to a further aspect, the second method comprising accessing signal scene information, the signal scene information comprising the down-sampled representation of the multi-channel signal, a first set of analysis components associated with the second component synthesis, wherein the first set of analysis components is configured to be used in combination with the first set of reconstructed signal components for reconstruction of the multi-channel signal; and a second set of analysis components associated with the first component synthesis, wherein the second set of analysis components is configured to be used in combination with down-sampled representation of the multi-channel signal for reconstruction of the second set of signal components.
The first set of analysis components may represent the first set of analysis components as mentioned above and is configured to be used in combination with the first set of signal components for reconstruction of the multi-channel signal . The second set of analysis components may represent the second set of analysis components as mentioned above and is configured to be used in combination with the second set of signal components for reconstruction of the first set of signal components.
It has to be understood that a corresponding receiver for reconstructing the down-sampled multi-channel signal also falls within in the scope of the protection. This receiver may be configured to apply any described detail with respect to the first method in reverse in order to reconstruct the multi-channel signal.
Further aspects of the invention will be apparent from and elucidated with reference to the detailed description presented hereinafter.
In the figures show:
Fig. 1 is a schematic block diagram which illustrates a first embodiment of an apparatus; Fig. 2 is a flow chart illustrating a first embodiment of a method;
Fig. 3a is a first embodiment of a method of the first component analysis;
Fig. 3b is first embodiment of a method of an inter-channel component analysis;
Fig. 4a a first embodiment of a method of a component
analysis for one channel;
Fig. 4b a second embodiment of a method of a component analysis for one channel;
Fig. 5a is a second embodiment of a method of an
inter-channel component analysis;
Fig. 5b is a third embodiment of a method of an inter-channel component analysis;
Fig. 6a is a schematic block diagram which illustrates a second embodiment of an apparatus;
Fig. 6b a schematic block diagram which illustrates an embodiment of a converter;
Fig. 7a is a first embodiment of an extracting method; Fig. 7b is a second embodiment of an extracting method; Fig. 8 is a schematic block diagram which illustrates a third embodiment of an apparatus;
Fig. 9 is a schematic block diagram which illustrates a fourth embodiment of an apparatus;
Fig. 10 is a flow chart illustrating a second embodiment of a method;
Fig. 11 is a schematic block diagram which illustrates a fifth embodiment of an apparatus; and
Fig. 12 is a schematic block diagram which illustrates a sixth embodiment of an apparatus.
DETAILED DESCRIPTION OF NO -LIMITI G EMBODIMENTS OF THE INVENTION
In the following detailed description, non-limiting embodiments of the present invention will be described in the context of embodiment of methods and apparatuses.
Figure 1 is a schematic block diagram which illustrates a first embodiment of an apparatus 100. This first embodiment of an apparatus 100 will be described in conjunction with the flow chart of a first embodiment of a method depicted in figure 2.
Apparatus 100 is fed by a multi-channel signal 105. This multi-channel signal 105 may comprise at least two channels, wherein each of the at least two channels is associated with a signal .
For instance, the multi-channel signal 105 may represent a "traditional two-channel stereo" signal, or a 5.1 or 7.2 channel configuration or any other multi-channel
configuration. Furthermore, as an example, the multi-channel signal 105 may represent a multi-view signal, wherein different channels of the multi-channel signal 105 are associated with different aural views to an audio scene. Thus, each of said different channels is associated with a signal being associated with the respective aural view of the different aural views.
For instance, the multi-channel signal 105 may represent a mixture of a multi-view signal and at least one further signal . The apparatus 100 is configured to perform a concatenated component analysis based on a time-frequency representation 105' of the multi-channel signal 105 for determining a down-sampled representation 130 of the multi-channel signal 105.
For instance, the multi-channel signal 105 may be in the time-frequency domain. In this case, the multi-channel signal 105 may be directly fed to first component analyser.
As another example, the apparatus 100 may comprise a converter (not depicted in figure 1) configured to transform the multi-channel signal 105 into the time-f equency
representation 105'. The dashed line 105' in figure 1 indicates that there may be some further signal processing with respect to the multi-channel signal 105 before being fed to the first component analyser 110.
The first component analyser 110 is configured to perform a first component analysis for determining a first set of signal components 115 representing the multi-channel signal, as exemplarily indicated by reference 210 in figure 2.
decorrelation of the time-frequency representation of the multi-channel signal. For instance, this first decorrelation may be directed to each of at least one channel of the at least two channels of the multi-channel signal 105, i.e. the first decorrelation may be performed to decorrelate each channel of at least one channel of the at least two channels separately, thereby representing an intra-channel decorrelation. Futhermore, as an example, this first decorrelation may perform a decorrelation between at least two channels of the at least two channels of the multi-channel signal 105, thereby representing an inter-channel
decorrelation.
Furthermore, as an example, the first component analysis may determine a set of first analysis components, wherein this set of first analysis components is configured to be used in combination with the first set of signal components for reconstruction of the multi-channel signal. For instance, the multi-channel signal may be reconstructed by a linear combination of the first analysis components and the first set of signal components.
The first set of signal components 115 represents the multi-channel signal after this first component analysis has been performed. Thus the multi-channel signal (or an approximation of the multi-channel signal) may be
reconstructed by means of a corresponding component synthesis applied to the first set of signal components 115, the first component synthesis representing the inverse operation of the first component analysis. For instance, the first component analysis may by performed by means of a principal component analysis (PCA) or an independent component analysis (ICA), but any other well-suited component analysis may also be used for carrying out the first component analysis.
Due to the first component analysis the output data rate of the first component analyser 110 may be reduced compared to the input data rate of the time-frequency domain
representation 105' of the multi-channel signal.
The second component analyser 120 is configured to perform a second component analysis based on the first set of signal components for determining a second set of signal components 125 representing the multi-channel signal, as exemplarily indicated by reference 220 in figure 2. The second component analysis may be different compared to first component analysis. For instance, the first component analysis may perform a first decorrelation of the
time-frequency representation of the multi-channel signal, as mentioned above, and the second component analysis may perform a second decorrelation of the multi-channel signal based on the first set of signal components, wherein the first decorrelation differs from the second decorrelation. As an example, the second decorrelation may be one of the above-mentioned inter-channel decorrelation and
Furthermore, as an example, the second component analysis may determine a set of second analysis components, wherein this set of second analysis components is configured to be used in combination with the second set of signal components for reconstruction of the first set of signal components. For instance, the first set of signal components may be reconstructed by a linear combination of the second analysis components and the second set of signal components. The second set of signal components 125 represents the multi-channel signal after this second component analysis has been performed. Thus the first set of signal component 115 (or an approximation of the first set of signal components 115) may be reconstructed by means of a corresponding component synthesis applied to the second set of signal components 125, the second component synthesis representing the inverse operation of the second component analysis. For instance, the second component analysis may by performed by means of a principal component analysis (PCA) or an independent component analysis (ICA), but any other well-suited component analysis may also be used for carrying out the second component analysis.
Thus, due to the concatenated component analysis different statistical properties of the multi-channel signal may be used for a two-stage component analysis in order to reduce the data rate of the multi-channel signal, wherein the first component analysis is directed to exploit one of the different statistical properties and the second component analysis is directed to exploit another of the different statistical properties . For instance, the multi-channel signal is associated with at least two channels. Furthermore, as an example, the first component analysis comprises for each channel of at least one channel of the at least two channels applying an intra-channel component analysis for determining a subset of signal components representing the signal of the respective channel. The down-sampled representation 130 of the multi-channel signal may comprise the second set of signal components 125. Due to the two-stage decomposition of the multi-channel signal the data rate of the down-sampled representation 130 of the multi-channel signal is reduced compared to the inputted multi-channel signal 105. The dashed arrow with respect to reference sign 130 indicates that there may be performed further signal processing to second set of signal components 125.
Figure 9 is a schematic block diagram which illustrates a fourth embodiment of an apparatus 1000. This fourth embodiment of an apparatus 1000 will be described in conjunction with the flow chart of a second embodiment of a method depicted in figure 10.
The fourth embodiment of an apparatus 1000 is configured to have access to signal 1130 which may represent or comprise the down-sampled representation 130 of the multi-channel signal. For instance, as a non-limiting example, signal 1130 may represent the down-sampled representation 130 of the multi-channel after being transmitted over a channel.
The apparatus 1000 is configured to perform a concatenated reconstruction of the multi-channel signal based on the down-sampled representation 130 of the multi-channel signal. The apparatus 1000 comprises a first component synthesiser 1010 configured to apply a first component synthesis for determining a first set of reconstructed signal components 1115 representing the multi-channel signal, as indicated by reference sign 1210 in figure 10, and a second component synthesizer 1020 configured to apply a second component synthesis based on the first set of reconstructed signal components 1115 for determining a reconstructed
representation 1005' of the multi-channel signal. The first component synthesis may represent an inverse operation of the second component analysis. Thus the first set of signal components 115 (or an approximation of the first set of signal components 115) may be reconstructed, indicated by reference sign 1115 in figure 9, by means of the first component synthesis applied to the second set of signal components 125, which are included in the down-sampled representation 130 of the multi-channel signal of accessed signal 1130. Furthermore, as an example, the first component synthesis may be performed based on the set of second analysis components which may be used in combination with the accessed second set of signal components. The second component synthesis may represent an inverse operation of the first component analysis. Thus the time-frequency domain representation 105' of the
multi-channel signal may be reconstructed by means of the second component synthesiser 1020 as reconstructed representation 1005' of the multi-channel signal based on the first set of reconstructed signal components 1115.
Furthermore, as an example, the second component synthesis may be performed based on the set of first analysis components which may be used in combination with the first set of reconstructed signal components 1115 for reconstruction of the multi-channel signal. For instance, as a non-limiting example, the fourth embodiment of an apparatus 1000 may be used in a receiver in order to reconstruct the multi-channel signal.
Figure 3a depicts a first embodiment of a method of the first component analysis for this intra-channel component analysis. For instance, this first embodiment of a method of the first component analysis may be used for the first component analyser 110 depicted in figure 1 and for step 210 depicted in figure 2.
One of these at least two audio channels is selected, as indicated by reference sign 310 in figure 3a, and an intra-channel component analysis is applied for determining a subset of signal components representing the signal of the respective channel, as indicated by reference sign 320 in figure 3a. The intra-channel component analysis may be carried out as explained above, for example based on PCA or any other well-suited component analysis. Thus, a
decorrelation of the signal of the respective channel may be performed.
Furthermore, as an example, the intra-channel component analysis may determine a subset of first analysis components, wherein this subset of first analysis components is configured to be used in combination with the subset of signal components for reconstruction of the signal of the respective channel of the multi-channel signal, e.g. by means of a linear combination. The set of first analysis components may comprise this subset of first analysis components. Then it may be checked whether a component analysis is to be performed for a next channel of the at least two channels, as indicated by reference sign 330 in figure 3a. If there is a next channel for component analysis the method proceeds with selecting this channel, as indicated by reference sign 340, and applying the intra-channel component analysis for determining a subset of signal components representing the signal of the selected channel, as indicated by reference sign 320.
The signal components of each determined subset of signal components being associated with the respective channel are signal components of the first set of signal components. Accordingly, the first set of components comprises the at least one subset of signal components determined by the respective intra-channel component analysis (indicated by reference sign 320) of the respective channel.
Accordingly, the loop depicted in figure 3a may be performed for each of the at least one channel of the at least two channels of the multi-channel signal in order to determine at least one subset of signal components of the first set of signal components. Furthermore, as an example, the time-frequency
time-frequency representatives, wherein each set of the at least two sets of time-frequency representatives is associated with one channel of the at least two channels of the multi-channel signal. Furthermore, in case the intra-channel component analysis is not applied for at least one channel of the at least two channels of the multi-channel signal, the signal of this at least one channel may be represented by the respective time-frequency representatives. For instance, the set of first signal components may comprise at least one subset of signal components being associated with said at least one channel for which no intra-channel component analysis has been applied, wherein each of this at least one subset of signal components comprises the respective set of
time-frequency representatives being associated with the respective channel .
Accordingly, as an example, the intra-channel component analysis is performed based on the set of time-frequency representatives of the at least two sets of time-frequency representatives associated with the respective channel.
It has to be understood that the embodiment of a method depicted in figure 3 is not limited to the strict structure of the flowchart's loop. For example, applying the intra-channel component analysis for each of the at least one channel of the at least two channels may be performed in parallel, or, as another example, partially in parallel and partially sequentially.
With respect to the fourth embodiment of an apparatus depicted in figure 9 and the second embodiment of a method depicted in figure 10 as counterpart to the embodiment of a method depicted in figure 3a, for instance, the second component synthesis may comprise for each channel of at least one channel of the at least two channels applying an intra-channel component synthesis for determining a reconstructed signal of the respective channel. This reconstructed signal of the respective channel may represent the above-mentioned set of time-frequency representatives being associated with the respective channel.
Figure 3b depicts a first embodiment of a method of the second component analysis representing an inter-channel component analysis. For instance, this first embodiment of a method of the second component analysis may be used for the second component analyser 120 depicted in figure 1 and for step 220 depicted in figure 2.
For instance, this inter-channel component analysis may be performed on the basis of the determined at least one subset of signal components of the first set of signal components which has been determined by the first embodiment of a method of the first component analysis as explained with respect to figure 3a.
The inter-channel component analysis, indicated by reference sign 350, is performed for determining the second set of signal components representing the multi-channel signal. For instance, the inter-channel component analysis may be applied for at least two channels of the at least two channels of the multi-channel signal in order to determine signal components of the second set of signal components
Furthermore, as an example, the inter-channel component analysis may determine analysis components of the set of second analysis components, wherein these determined analysis components are configured to be used in combination with the second set of signal components for reconstruction of the signal of the at least one channel of the multi-channel signal, e.g. by means of a linear combination. With respect to the fourth embodiment of an apparatus depicted in figure 9 and the second embodiment of a method depicted in figure 10 as counterpart to the embodiment of a method depicted in figure 3b, for instance, the first component synthesis may represent an inter-channel synthesis of at least two channels of the at least two channels associated with the multi-channel signal, the inter-channel synthesis representing the inverse operation of the inter-channel component analysis.
Figure 4a depicts a first embodiment of a method of a component analysis for one channel. For instance, this embodiment of a method of a component analysis may be used for the above-mentioned intra-channel component analysis for determining a subset of signal components of a respective channel, as indicated by reference sign 320 in figure 3. Thus, the first embodiment of a method of a component analysis for one channel may be inserted between the reference signs 315 and 325 in figure 3a and may represent a part of the first embodiment of a method of inter-channel component analysis depicted in figure 3a.
Each channel of the at least one channel of the at least two channels is associated with a frequency band. The width of the frequency band of one channel may depend on the signal associated with the respective channel and can either be fixed or variable.
Furthermore, the frequency band of one channel of the at least one channel is associated with at least one subband of the frequency band of the respective channel.
A component analysis is applied for each subband of at least one subband of the frequency band of the respective channel for determining a subsubset of signal components for each subband of the at least one subband of the respective channel, as indicated by reference sign 410 in figure 4a. A subsubset of signal components represents signal components of the first set of signal components representing the signal in the respective subband of the respective channel of the multi-channel signal.
Furthermore, as an example, the component analysis associated with a subband of the at least one subband of the frequency band of the respective channel may determine a subsubset of first analysis components, wherein this subsubset of first analysis components is configured to be used in combination with the respective subsubset of signal components for reconstruction of the signal of respective subband of the respective channel of the multi-channel signal, e.g. by means of a linear combination. The set of first analysis components may comprise this subsubset of first analysis components. This determining may be performed based on the time-frequency representatives of the set of time-frequency representatives of the respective channel and being associated with the respective subband of the respective channel. For instance, each set of time-frequency representatives associated with one channel may comprise at least one subset of time-frequency representatives, wherein each subset of time-frequency representatives may be associated with one subband of the respective channel and may comprise time-frequency
representatives representing the signal in the respective subband of the respective channel.
For instance, the above-mentioned subset of signal components being associated with the respective channel may comprise the at least one subsubset of signal components determined by means of the first embodiment of a method of a component analysis for one channel depicted in figure 4a. Accordingly, each subsubset of signal components comprises signal components representing the signal in the subband of the respective channel determined by the applied component analysis. Thus, for example, a plurality of intra-channel component analysis may be performed for different subbands of a frequency band of a respective channel. For instance, a decorrelation of signals in different subbands of a respective channel may be performed, which may enhance the quality of decorrelation.
With respect to the fourth embodiment of an apparatus depicted in figure 9 and the second embodiment of a method depicted in figure 10 as counterpart to the embodiment of a method depicted in figure 4a, for instance, the second component synthesis may represent the above-mentioned intra-channel component channel, wherein this intra-channel component synthesis comprises for a respective channel applying a component synthesis for each subband of at least one subband of the frequency band of the respective channel for determining a reconstructed subsignal of the multi-channel signal associated with the respective subband of the respective channel for each subband of the at least one subband .
For instance, this reconstructed subsignal may represent reconstructed time-frequency representatives representing the signal in the respective subband of the respective channel .
Figure 4b depicts a second embodiment of a method of a component analysis for one channel which may be used for performing the first embodiment of a method of a component analysis for one channel depicted in figure 4a.
One subband of the at least one subband of the respective channel is selected, as indicated by reference sign 420 in figure 4b.
Then a component analysis is applied for the respective subband of the respective channel for determining a subsubset of signal components of the first set of signal components, as indicated by reference sign 430. This component analysis may be performed as explained above.
Then it may be checked whether there is another subband of the at least one subband associated with the respective channel, as indicated by reference sign 440. If there is a next subband the method proceeds with selecting this next subband, as indicated by reference sign 450, and applying the component analysis for the selected subband for determining the subsubset of signal component being associated with the selected subband and the respective channel (indicated by reference sign 430) .
Accordingly, the loop depicted in figured 4b may be performed fore each of the at least one subband associated with the respective channel in order to determine at least one subsubset of signal components of the first set of signal components. Thus, this at least one subsubset of signal components may represent the subset of signal components of the first set of signal components representing the signal in the respective channel in accordance with the applied at least one component analysis performed by step 430. Figure 5a depicts a second embodiment of a method of an inter-channel component analysis. For instance, this second embodiment of a method of on inter-channel component analysis may be used for the first embodiment of a method of inter-channel component analysis depicted in figure 3b and may be used for the second component analyser 130 depicted in figure 1 and for step 230 depicted in figure 2. For instance, this second embodiment of a method of an inter-channel component analysis may be used in combination with an inter-channel component analysis based on one of the embodiment of a methods depicted in figures 4a and 4b.
The second embodiment of a method of inter-channel component analysis comprises applying at least one component analysis for determining at least one subset of signal components of the second set of signal components, as indicated by reference sign 510 in figure 5a. Each of the at least one component analysis is associated with at least two channels of the at least two channels of the multi-channel signal and with a subband of a frequency band associated with the respective at least two channels of the at least two channels.
Thus, each determined subset of signal components comprises signal components of the second set of signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband . The determining is based on the first set of signal components. Accordingly, signal components of the first set of signal components representing the signals of the respective at least two channels in the respective subband are used as basis for the component analysis of the inter-channel component analysis. For instance, under the assumption of the exemple notation regarding the subsubset of signal components of the first set of signal components, one component analysis of the at least one component analysis of the inter-channel component analysis may be applied based on the respective at least two subsubsets of signal components of the first set of signal components, wherein a subsubset of signal components of the at least two subsubsets of signal components is associated with the respective subband of the respective channel of the respective at least two channels.
The second embodiment of a method of inter-channel component analysis may comprise a component analysis for each subband of a frequency band associated with the at least two channels. Furthermore, as an example, a component analysis of the at least one component analysis of the inter-channel component analysis may determine a subset of analysis components of the set of second analysis components, wherein the subset of analysis components is configured to be used in combination with the respective subset of second set of signal components for reconstruction of the respective at least two subsubsets of signal components of the first set of signal components.
With respect to the fourth embodiment of an apparatus depicted in figure 9 and the second embodiment of a method depicted in figure 10 as counterpart to the embodiment of a method depicted in figure 5b, for instance, the first component synthesis may represent an inter-channel synthesis of at least two channels of the at least two channels associated with the multi-channel signal, the inter-channel synthesis comprising applying at least one component synthesis for determining at least one subset of reconstructed signal components of the first set of reconstructed signal components, wherein each of the at least one component synthesis is associated with at least two channels of the at least two channels of the multi-channel signal and with a subband of a frequency band associated with the respective at least two channels of the at least two channels, each determined subset of reconstructed signal components representing a reconstructed signal components of the first set of reconstructed signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband.
Figure 5b depicts a third embodiment of a method of an inter-channel component analysis which may be used for performing the second embodiment of a method of an
inter-channel component analysis. At least two channels of the multi-channel signal are selected, as indicated by reference sign 520, and a subband of a frequency band associated with these at least two channels is selected, as indicated by reference sign 530.
Then a component analysis is applied for determining a subset of signal components of the second set of signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband (indicated by reference sign 540), as explained above .
Then it is checked whether there is a further subband associated with these at least two channels, as indicated by reference sign 550, and if there is a further subband, this subband is selected (indicated by reference sign 560) and a component analysis is applied for determining a subset of signal components of the second set of signal components representing the signals of the respective at least two channels of the multi-channel signal in the selected subband, as indicated by reference sign 540.
If there is no further subband, the method process with checking whether there are further at least two channels of the multi-channel signal to be used for inter-channel component analysis, as indicated by reference sign 570, and if there are further at least two channels, these at least two channels are selected (indicated by reference sign 580) and the method proceeds with selecting a subband of the frequency band associated with the selected at least two channels . For instance, according to method depicted in figure 5b, the inter-channel component analysis may be performed for a plurality of sets of at least two channels, thereby applying component analysis for subbands of the frequency bands associated with the at least two channels of each set of the plurality of sets of at least two channels.
Furthermore, the inner loop regarding the subbands and the outer loop regarding the selection of at least two channels may be exchanged, and, as another example, parts of the loops or the complete loops may be performed in parallel.
As another example, the loop regarding the selection of at least two channels may be discarded. For instance, all the channels of the at least two channels of the multi-channnel signal may be selected.
Fig. 6a depicts a schematic block diagram which illustrates a second embodiment of an apparatus 600. This second embodiment of an apparatus 600 is based on the first embodiment of an apparatus 100. Accordingly, the explanations presented with the respect to first embodiment of an apparatus 100 also hold for the second embodiment of an apparatus 600. Compared to the first embodiment of an apparatus 100, the second embodiment of an apparatus 600 comprises a converter 140 configured to convert a time-domain multichannel representation of the multi-channel signal to the
time-frequency representation 105' of the multi-channel signal. For instance, the multi-channel signal 105 may represent this time-domain representation of the
multi-channel signal. The time-frequency representation comprises at least two sets of time-frequency representatives, each set of the at least two sets of time-frequency representatives being associated with one channel of the at least two channels of the multi-channel signal. Furthermore, each set of the at least two set of time-frequency representatives may comprise at least one subset of time-frequency representatives, wherein each time-frequency representative of a subset of the at least one subset is associated with a frequency component of the frequency band of the respective channel and with different point in times. For instance, the time-frequency
representatives of a subset of the at least one subset are associated with the respective frequency component and a time frame.
above-mentioned subband of the frequency band of the respective channel, but, as another example providing a higher resolution in frequency, one subband of the frequency band of the respective channel my be associated with at least two frequency components. For instance, this converting may be based on a Fourier
Transformation, e.g. implemented by means of a Fast Fourier Transformation (FFT) or a Discrete Fourier Transformation (DFT) or any other well-suited Fourier Transformation, and/or by means of a Discrete Cosine Transformation (DCT) and/or by means of any other suited transformation. Accordingly, the time-frequency representation 105' of the multi-channel signal may be used for one or more of the above-mentioned component analysis. As an example with respect to the embodiment of a methods depicted in figures 4a and 4b, applying a component analysis for a respective subband of a respective channel for determining a subsubset of signal components of the first set of signal components may be performed based on the set of time-frequency representatives associated with this respective channel and with at least one subset of
time-frequency representatives of this set of time-frequency representatives, the at least one subset of time-frequency representatives being asscociated with the respective band. I.e., the at least one frequency component associated with the at least one subset of time-frequency representatives are associated with the respective subband of the respective channel . Fig. 11 depicts a schematic block diagram which illustrates a fifth embodiment of an apparatus 1600. This fifth embodiment of an apparatus 1600 is based on the fourth embodiment of an apparatus 1000. Accordingly, the explanations presented with the respect to fourth embodiment of an apparatus 1000 also hold for the second embodiment of an apparatus 600.
Compared to the fourth embodiment of an apparatus 1000, the fifth embodiment of an apparatus comprises a converter 1040 configured to convert a time-frequency representation 1005' of the multi-channel signal to time-domain multichannel representation 1005 of the multi-channel signal. Thus, converter 1040 may represent an inverse converter with respect to the converter 140 of the second embodiment of an apparatus 600 depicted in figure 6a and/or to the converter 140' depicted in figure 6b.
Fig. 6b depicts a schematic block diagram which illustrates an embodiment of a converter 140' comprising a first transformer 620 and a second transformer 630. For instance, this converter 140' may be used as converter 140 depicted in figure 6a.
The first transformer 620 is configured, for each channel of the at least two channels, to transform the time-domain representation of the respective channel of the time-domain representation into a frequency domain representation.
As an example, the time-domain representation of a channel m of the at least two channels is represented by xm(k) . A frame 1 of time-domain representations xm(t) is converted by the first transformer 620 to a respective frequency domain representation Xm[k,l], where k is the frequency component index (e.g. k may represent a frequency bin index) and wherein TF may represent a corresponding time-to-frequency operator: Xm [k, l] = TF (xmJ ) (4)
This is performed for each of the channels.
As an example, a Modified Discrete Cosine Transformation may be used for the first transformer 620, as exemplarily explained in the sequel: The TF operator is applied to each signal segment according to Xm [k,l] = TF{xmJ ,r) (5) where m is the channel index, k is the frequency bin index, 1 is time frame index, T is the hop size between successive time frames, and TF the time-to-frequency operator. For instance, MDCT may be used as the TF operator as follows
2^^)= 2 ·∑¾(»)■ cos n +— + 0.5 \ - (k + 0. , 0 < k < 1
n=0 N { 4 J ' 2
(n) = w(n)- xm (n + l where w(n) is the N-point analysis window such as sinusoidal or Kaiser-Bessel Derived (KBD) window. In MDCT, the hop size is T=N/2.
Accordingly, assuming that the at least two channels are M channels, the frequency domain representation 625 of the multi-channel signal is represented by Xm[ ,/] with 0 < m ≤ M.
The second transformer 630 is configured, for each channel of the at least two channels, to perform a two-dimensional discrete cosine transformation (2D-DCT) based on the frequency domain representation associated with the respective channel in order to determine the set of time-frequency representatives associated with the respective channel.
For instance, the set-of time-frequency representative of channel m may be represented by matrix Fm [&,/], where t is the time-index. Accordingly, k is the index of the respective frequency component and all representatives of ^"[ ] with a fixed k and a fixed m represent the above-mentioned subset of the time-frequency representatives being associated with channel m and frequency component k.
As an example, the operation of the second transformer may be applied to the frequency domain representation based on a 2D-DCT as follows:
■(2 - j + l)
start < u < t end t _ start = grpldx■TF_
t _ end = t _ start + TF
Α = , B - t _end - 1 _start, C(i) -
1, otherwise
where TF_size is the size of the 2D time-frequency plane. The size of matrix Ym may therefore be TF_size x A. Figure 7a depicts a first embodiment of an extracting method which may be applied for one of the preceding embodiments of a method. This first embodiment of an extracting method comprises extracting from each of at least one set of signal components of the first set of signal components a separate subset of relevant signal components, as indicated by reference sign 720, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the first set of signal components in accordance with a first accuracy criteria . Each of the at least one set of signal components may be associated with one component analysis of the first component analysis .
For instance, one set of the at least one set of signal components may represent a subset of signal components of the first set of signal components determined by means of the intra-channel component analysis. In this example, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective channel of the multi-channel signal.
Or, as another example, one set of the at least one set of signal components may represent a subsubset of signal components of the first set of signal components determined by means of the component analysis for a respective subband of a respective channel. In this example, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective subband of the respective channel of the multi-channel signal.
The first accuracy criteria may represent any well-suited criteria configured to determine the quality of the part of the multi-channel signal reconstructed by means of the respective set of relevant signal components.
Furthermore, for instance, first extraction information may be provided indicative which signal components of a set of the at least one set of signal components have been extracted to the respective subset of relevant signal components.
Further, as an example, this first extraction information may be used by the fourth embodiment of an apparatus 1000 in order to reconstruct the first signal components before the second component synthesis is performed.
Thus, for instance, only those signal components of a set of the at least one set of signal components are selected which are sufficient to represent the respective part of the multi-channel signal at desired accuracy, whereas the remaining signal components may be discarded. These selected signal components may define the corresponding subset of relevant signal components. For instance, the first extraction information may comprise the number of selected components of a set of the at least one set of signal components . For instance, a measure of relevance may be determined for each signal component of each of the at least one set of signal components of the first set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the first set of signal components.
As an example, the measure of relevance associated with a signal component may represent a variance. For instance, this variance may be computed by means of the respective component analysis when determining the first set of signal components, or when determining a subset of signal components of the first set of signal components, or when determining a subsubset of signal components of the first set of signal components.
For instance, Vm fl) (z) represents the variance of the i-th signal component of a set of the at least one set of signal components of the first set of signal components, the set of the at least one set being associated with the m-th channel and subband fb. Accordingly, for this example, a set of the at least one set represents a subsubset of signal components as mentioned above. Furthermore, it may be assumed that variances Vm ^ {i have been sorted in decreasing order.
3 for i=0 to length ( Vm fb ) -1
4 aSum = aSum + Vm jb (i^) ; 5
10 vldx"tfi = i
Thus, this example of a code may determine how many signal components of the set of the at least one set of signal components of the first set of signal components are needed such that the accumulated variance divided by the sum of all variances exceeds the first accuracy criteria thr_ind. For instance, this example of a first accuracy criteria thr_ind may be set to 0.9999, but any other well-suited threshold may also be used. This first accuracy criteria may indicate that signal components with large associated variances represent significant dynamics in the audio/video scene, while those with lower variances represent less detailed information and can be discarded. Accordingly, only the signal components of the set of the at least one set of signal components being associated with the vldxm ^ highest variances are selected for the respective subset of relevant signal components.
The first embodiment of an extracting method depicted in figure 7a may be performed before the second component analysis is performed. With respect to the flowchart depicted in figure 2, the first embodiment of an extracting method depicted in figure 7a may be inserted between applying the first component analysis and applying the second component analysis . Then, for instance, applying the second component analysis may be based on the at least one extracted subset of relevant signal components of the first set of signal components.
Figure 7b depicts a second embodiment of an extracting method which may be applied for one of the preceding embodiments of a method.
This second embodiment of an extracting method comprises extracting from each of at least one set of signal components of the second set of signal components a separate subset of relevant signal components, as indicated by reference sign 730, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the second set of signal components in accordance with a second accuracy criteria .
Each of the at least one set of signal components of the second set of signal components may be associated with one component analysis of the second component analysis.
For instance, one set of the at least one set of signal components of the second set of signal components may represent a subset of signal components of the second set of signal components determined by means of the inter-channel component analysis. In this exemple case, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective at least two channels of the multi-channel signal.
Or, as another example, one set of the at least one set of signal components of the second set of signal components may represent a subsubset of signal components of the second set of signal components determined by means of the component analysis for a respective subband of respective at least two channels. In this exemple case, the subset of relevant signal components may represent the signal (or an approximation of this signal) of the respective subband of the respective at least two channel of the multi-channel signal.
Furthermore, for instance, second extraction information may be provided indicative which signal components of a set of the at least one set of signal components of the second set of signal components have been extracted to the respective subset of relevant signal components. Further, as an example, this second extraction information may be used by the fourth embodiment of an apparatus 1000 in order to reconstruct the second signal components before the first component synthesis is performed. The second accuracy criteria may represent any well-suited criteria configured to determine the quality of the part of the multi-channel signal reconstructed by means of the respective set of relevant signal components. Thus, for instance, only those signal components of a set of the at least one set of signal components are selected which are sufficient to represent the respective part of the multi-channel signal at desired accuracy, whereas the remaining signal components may be discarded. These selected signal components may define the corresponding subset of relevant signal components. The number of selected components of a set of the at least one set of signal components of the second set of signal components may represent an extraction information associated with the respective set of the at least one set of signal components. For instance, a measure of relevance may be determined for each signal component of each of the at least one set of signal components of the second set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the second set of signal components.
As an example, the measure of relevance associated with a signal component may represent a variance. For instance, this variance may be computed by means of the respective component analysis when determining the second set of signal
components, or when determining a subset of signal components of the second set of signal components, or when determining a subsubset of signal components of the first set of signal components.
For instance, V_mvfi (z) may represent the variance of the i-th signal component of a set of the at least one set of signal components of the second set of signal components, the set of the at least one set being associated with at least two channels of the at least two channels and with subband fb. Accordingly, for this example, a set of the at least one set represents a subsubset of signal components as mentioned above. As a non-limiting example, it may be assumed that the set of the at least one set of signal components of the second set of signal components is associated with all M channels of the at least two channels of the multi-channel signal.
Furthermore, it may be assumed that variances Vm (z) have been sorted in decreasing order.
2 !engt (
tSum =
3 for i=0 to len th ( V mvfb ) -1
10 vldx mv Λ = i
Thus, this example of a code may determine how many signal components of the set of the at least one set of signal components of the second set of signal components are needed such that the accumulated variance divided by the sum of all variances exceeds the second accuracy criteria thr_ind2. For instance, this second accuracy criteria thr_ind2 may be set to 0.9995, but any other well-suited threshold may also be used. This second accuracy criteria may indicate that signal components with large associated variances represent significant dynamics in the audio/video scene, while those with lower variances represent less detailed information and can be discarded. Accordingly, only the signal components of the set of the at least one set of signal components being associated with the vldx_mvfi! highest variances are selected for the respective subset of relevant signal components of the second set of signal components.
For instance, in the set of second signal components at least one set of the at least one set of signal components of the second set of signal components may be replaced by the respective at least one subset of relevant signal components.
The second embodiment of an extracting method depicted in figure 7b may be performed after the second component analysis is performed. With respect to the flowchart depicted in figure 2, the second embodiment of an extracting method depicted in figure 7b may be placed after applying the second component analysis .
Then, for instance, applying the second component analysis may be based on the at least one extracted subset of relevant signal components of the first set of signal components.
For instance, the down-sampled representation of the multi-channel signal may comprise the at least one extracted subset of relevant signal components of the second set of signal components. Accordingly, a further data rate reduction may be performed based on the first and/or second embodiment of an extracting method . Fig. 8 depicts a schematic block diagram which illustrates a third embodiment of an apparatus 100' . This third embodiment of an apparatus 100' is based on the first embodiment of an apparatus 100. Accordingly, the explanations presented with the respect to first embodiment of an apparatus 100 also hold for the third embodiment of an apparatus 100' , and the explanations given with respect to the third embodiment of an apparatus 100' may also hold for the second embodiment of an apparatus 600 depicted in figure 6a. The third apparatus 100' is configured to output a signal scene information 150 which comprises the down-sampled representation 130 of the multi-channel signal, a first set of analysis components 111 associated with the first component analysis, and a second set of analysis components 121 associated with the second component analysis.
The first set of analysis components may represent the first set of analysis components as mentioned above and is configured to be used in combination with the first set of signal components for reconstruction of the multi-channel signal .
The second set of analysis components may represent the second, set of analysis components as mentioned above and is configured to be used in combination with the second set of signal components for reconstruction of the first set of signal components. Furthermore, for instance, the signal scene information 150 may comprise the first and/or second extraction information. Fig. 12 depicts a schematic block diagram which illustrates a sixth embodiment of an apparatus 1000' . This sixth embodiment of an apparatus 1000' is based on the fourth embodiment of an apparatus 1000. Accordingly, the
explanations presented with the respect to fourth embodiment of an apparatus 1000 also hold for the sixth embodiment of an apparatus 1000' , and the explanations given with respect to the sixth embodiment of an apparatus 1000' may also hold for the fifth embodiment of an apparatus 1600 depicted in figure 11.
The fifth apparatus 1000' may be configured to have access to signal scene information 150, the signal scene information which comprises the down-sampled representation 1130 of the multi-channel signal, the first set of analysis components 1111 associated with the second component synthesis, wherein the first set of analysis components 1111 is configured to be used in combination with the first set of reconstructed signal components 1115 for reconstruction of the
multi-channel signal, and the second set of analysis components 1121 associated with the first component synthesis, wherein the second set of analysis components 1121 is configured to be used in combination with the down-sampled representation 1130 of the multi-channel signal for reconstruction of the second set of signal components.
For instance, it has to be understood that a corresponding apparatus for reconstructing the down-sampled multi-channel signal, for instance implemented by one of the fourth, fifth and sixth embodiment of an apparatuses, also falls within in the scope of the protection. As an example, this corresponding apparatus may represent a kind of receiver.
In the sequel a further embodiment of a method will be explained. In this further embodiment of a method both the first component analysis and the second component analysis are based on a PCA.
Furthermore, for instance, it is assumed that the first component analyser 110 performs an intra-channel component analysis and that the second component analyser 120 performs an inter-channel component analysis. For instance, it is assumed that the at least two channels are M channels, and that the frequency domain representation
625 of the multi-channel signal is represented by OT[&,/] with
0 < m ≤ M, as exemplarily explained with respect to the converter 140'.
As an example, the intra-channel PCA may decorrelate the channels of the multi-channel signal and determines the set of first signal components according to following steps: Step 1: Ym[fb_start(fb),0 Υ",[β_ start (fb), TF _ size - 1
Jm,fl (7)
Ym[fb_end(fb)-\,Q] ■■■ Ym[fb_end(fb)-l,TF_size-l fb _start = 0,fWidth,2-fWidth,3- fWidth,...,A
fb_end(fb) = β _ tart (fb + \)-fb_ start (β), <β< length (fl _ tart) where length () returns the length of the specified input vector and pea () is a function that returns the principal components ~PCmJb and associated variances Vm fb for the given input signal. The principal components PC^ may represent a subsubset of first analysis components of the set of first analysis components as mentioned above, wherein this subsubset of first analysis components is associated with subband fb and with channel m. In the end of this specification a detailed pseudo-code listing for the pea () function in simplified Matlab code and a mathematical formulation of PCA will be presented. The size of matrix Zm is fSize x TF__size where
fSize=fb_end (fb) -fb_start (fb) . Furthermore, the dimensions of PC " and V » are fSize x fSize and fSize x 1,
respectively. Thus, the intra-channel PCA is determined on a subband fb where the width of the subband (fWidth) can either be fixed or variable.
For instance, as a non-limiting example, the width of a subband may be set to 6, i.e., 6 successive frequency components (e.g. frequency bins) of the m-th channel may represent a subset of time-frequency representatives of the m-th set of time-frequency representatives. As another example, for instance, the width of a subband may be variable. For example, the width of a subband may follow the boundaries of Equivalent Rectangular Bandwidth (ERB) .
Step 2: The matrix Zmy¾ may now projected using the principal components to obtain the decorrelated signal components according to
where represents the determined subsubset of signal components of the first set of signal components associated with subband fb and with channel m.
Next, the subset of relevant components of the subsubset of signal components most relevant decorrelated signal components may be extracted from Dm according to following pseudo-code and as explained with respect to the first embodiment of an extracting method:
2 kngth(V"l h )-\
tSum= ∑Vmfb(i)
4 aSum = aSum + Vmjb{);
7 exit for loop 8 End
10 vldx, = ι
Thus, the above pseudo-code may determine how many principal components are needed such that the accumulated variance divided by the sum of all variances exceeds the example threshold thr_ind. For instance, this threshold value may be set to 0.9995. This value may indicate that principal components with large associated variances represent significant dynamics in the signal, while those with lower variances represent noise and can be discarded.
Thus, the intra-channel PCA of step 1 and step 2 may correspond to the component analysis applied for a respective subband (fb) of a respective channel (m) for determining a subsubset of signal components ( Dm β ) as indicated by reference sign 430 in figure 4b and explained with respect to the embodiment of a method depicted in figure 4b. The intra-channel PCA and the first embodiment of an extracting may be performed for each subband of each of the M channels; for instance by means of the loops depicted in figure 4b or by means of the embodiment of a method depicted in figure 4a.
For instance, the inter-channel PCA may analyse the correlations across channels of the multi-channel signal, and thereby decorrelate the channels of the multi-channel signal and extracts the most relevant signal components according to following: P( _mv ; . Y mv
where M is the number of channels of the multi-channel signal.
The size of matrix is P x TF_size, where P = vldxl The principal components PC_mv/¾ may represent a subset of second analysis components of the set of second analysis components, wherein this subset of second analysis components is associated with subband fb and with all M channels.
The matrix may now projected using the principal components to obtain the decorrelated signal components corresponding to the channels of the input signal according to
(10) where represents the determined subset of signal components of the second set of signal components being associated with subband fb and with all M channels. Accordingly, the inter-channel PCA may corresponds to the component analysis for determining a subset of signal components R^of the second set of signal components indicated by reference sign 540 depicted in figure 5b, wherein the loop for selecting the channels is discarded and selected at least two channels represent the M channels.
The, for instance, the most relevant decorrelated signal components of the channels of the input signal may be extracted from according to above-mentioned example of second extracting method:
2 length(V nv β )-1
tSum = ^ V _ mvjb (/)
3 for i=0 to len th ( V_mvfb ) -1
10 vldx_mvjb = i
Thus, the above pseudo-code may determine how many principal components are needed such that the accumulated variance divided by the sum of all variances exceeds the threshold thr_ind2. For instance, this threshold value may be set to 0.9999. This value may indicates that principal components with large associated variances represent significant dynamics in the audio scene, while those with lower variances represent less detailed information and can be discarded.
For instance, the down-sampled representation for each subband fb of the plurality of subbands of the multi-channel signal be obtained as follows
Sfi = [RJb
(11) Furthermore, for instance, the signal scene information may be represented by the following elements for each subband fb of the plurality of subbands:
PC0ifi , - PCM_hfb, PC_mvfl
vldx0 fb , · · · vIdxM_l ftl, vldx _mv β
Vector S ^
Thus, for instance, the multi-channel signal scene may be represented by M sets of intra-PCA components (representing the first set of analysis information) together with the information, regarding the number of intra-PCA components (representing the first extraction information) included in respective set, a set of inter-PCA components (second set of analysis information) together with the information regarding the number of intra-PCA components (representing the second extraction information) , and a down sampled signal representation of the multi-channel signal.
For instance, any of the above mentioned embodiment of a method and/or embodiment of an apparatus may be applied to a transmitter configured to transmit the down-sampled representation of the multi-channel signal or the signal scene information.
For instance, at the reconstruction side, i.e. by one of the fourth, fifth or sixth embodiments of an apparatus, the reverse of the operations are performed.
First, as an example, the inter PCA synthesis may be applied to recover the individual decorrelated signals on each of the channels as following step:
= PC_mv fl (12) where PC mv ^ contains the accessed principal components of the respective subband fb (as part of the second set of analysis information) for the decorrelated multi-channel signal, i.e. PC mv β may represent an accessed subset of second analysis components of the set of second analysis components. Furthermore,
M -l
MV =∑vIdx,",fl where S contains the accessed down sampled signal representation of the multi-channel signal and 0xy
represents a zero valued matrix with x rows and y columns, which may be determined based on the accessed second extraction information.
Next, for instance, the intra PCA synthesis may be applied to recover the signals on each of the channels for the 2D time-frequency plane as following step:
where PCm fh contains the accessed principal components for the individual channels of the multi-channel signal, i.e. PCm fb may represent an accessed subsubset of first analysis components of the set of first analysis components. Steps 3 and 4 may be repeated for 0≤ fb < length(jb _ start) in the same manner as done in the analysis side, i.e. for each subband of the plurality of subbands . The 2D time-frequency samples Ym are transferred to frequency domain samples via 2D-IDCT according to
Λ' = F
15' ij0,0]r, ... Zm[0,TF_size-l] Zm - ■■■ Zm[A-\,TF_size-
Finally, the frequency domain samples may be transformed to time domain signals xm via inverse TF, in this case via IMDCT as follows
•(«+ 0.5)1 0<£<N-1
N N
xm [k + 1■ Τ] = xx m [k,l]+ xx" + k,l-l 0≤k<
Thus, for instance, xm may represent the reconstructed multi-channel signal 1005.
Now, as an example, the mathematical background of PCA will be explained. In general, the PCA analysis may be
mathematically as follows: Let the data set be X, an m n matrix, where m is the number of measurement types and n is the number of samples. The goal is summarized as follows. Find some orthonormal matrix P where Y = PX such that
•YY is diagonalized .
Y n-\
The rows of P are the principal components of X . Begin by rewriting Cy in terms of our variable of choice P
Cv = — PAP7 (16)
* n-1
A new matrix A≡XXr has been defined where A is symmetric, Furthermore,
A = EDE7' (17) where D is a diagonal matrix and E is a matrix of eigenvectors of A arranged as columns. Let the matrix P to be a matrix where each row p. is an eigenvector of XX7. By this selection,
P≡E7. Substituting into Equation (17), we find A = P7DP. With this relation and P_I=Prwe can finish evaluating as follows
Cv=— D (18)
* 77-1
It is evident that the choice of P diagonalizes C^ . This was the goal for PCA. The results of PCA in the matrices P and C,7 can be summarized as follows • The principal components of X are the eigenvectors of XX ; or the rows of P
• The itfl diagonal value of Cy is the variance of X along Pi
In practice computing PCA of a data set X may entails (5) subtracting off the mean of each measurement type and (6) computing the eigenvectors of XX7. For instance, a PCA may be performed by the following exemple Matlab code for calculating the PCA of a signal : function [PC,V] = pea (X) % Perform PCA using covariance.
M = rows (X) ;
N = columns (X) ; % subtract off the mean for each dimension
mn = mean (X, 2 ) ;
data = X - repmat (mn, 1 , N) ;
% calculate the covariance matrix
covariance = 1 / (N-l) * X * X' ;
% find the eigenvectors and eigenvalues.
% produces a diagonal matrix V of eigenvalues and a full matrix PC whose columns are the corresponding eigenvectors so that covariance *V = V*PC.
[PC, V] = eig (covariance) ; % extract main diagonal of matrix V as vector
V = diag (V) ; % sort the variances V in decreasing order
[junk, rindices] = sort(-l*V);
V = V ( rindices ) ;
% principal components
PC = PC ( : , rindices) ;
Furthermore, it is readily clear for a person skilled in the art that the logical blocks in the schematic block diagrams as well as the flowchart and algorithm steps presented in the above description may at least partially be implemented in electronic hardware and/or computer software, wherein it may depend on the functionality of the logical block, flowchart step and algorithm step and on design constraints imposed on the respective devices to which degree a logical block, a flowchart step or algorithm step is implemented in hardware or software. The presented logical blocks, flowchart steps and algorithm steps may for instance be implemented in one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable devices. The computer software may be stored in a variety of
computer-readable storage media of electric, magnetic, electro-magnetic or optic type and may be read and executed by a processor, such as for instance a microprocessor. To this end, the processor and the storage medium may be coupled to interchange information, or the storage medium may be included in the processor. Any presented connection in the described embodiments is to be understood in a way that the involved components are operationally coupled. Thus, the connections can be direct or indirect with any number or combination of intervening elements, and there may be merely a functional relationship between the components.
Any of the processors mentioned in this text could be a processor of any suitable type. Any processor may comprise but is not limited to one or more microprocessors, one or more processor (s) with accompanying digital signal processor (s), one or more processor (s) without accompanying digital signal processor (s), one or more special-purpose computer chips, one or more field-programmable gate arrays (FPGAS), one or more controllers, one or more application-specific integrated circuits (ASICS), or one or more computer (s) . The relevant structure/hardware has been programmed in such a way to carry out the described function.
Any of the memories mentioned in this text could be implemented as a single memory or as a combination of a plurality of distinct memories, and may comprise for example a read-only memory, a random access memory, a flash memory or a hard disc drive memory etc.
Moreover, any of the actions described or illustrated herein may be implemented using executable instructions in a general-purpose or special-purpose processor and stored on a computer-readable storage medium (e.g., disk, memory, or the like) to be executed by such a processor. References to 'computer-readable storage medium' should be understood to encompass specialized circuits such as FPGAs, ASICs, signal processing devices, and other devices.
It will be understood that all presented embodiments represent non-limiting examples, that features of these embodiments may be omitted or replaced and that other features may be added. Any mentioned element and any mentioned method step can be used in any combination with all other mentioned elements and all other mentioned method step, respectively. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.
A method comprising:
performing a concatenated component analysis based on a time-frequency representation of a multi-channel signal for determining a down-sampled representation of the multi-channel signal,
wherein said concatenated component analysis comprises: applying a first component analysis for determining a first set of signal components representing the multi-channel signal; and
applying a second component analysis based on the first set of signal components for determining a second set of signal components representing the multi-channel signal.
The method according to claim 1, the multi-channel signal being associated with at least two channels, the first component analysis comprising for each channel of at least one channel of the at least two channels applying an intra-channel component analysis for determining a subset of signal components of said first set of signal components representing the signal of the respective channel, and the second component analysis representing an inter-channel component analysis of at least two channels of the at least two channels associated with the multi-channel signal.
3. The method according to claim 2, wherein each channel of the at least one channel of the at least two channels is associated with a frequency band, and the intra-channel component analysis for a respective channel comprises applying a component analysis for each subband of at least one subband of the frequency band of the respective channel for determining a subsubset of signal components associated with the respective subband of the respective channel for each subband of the at least one subband, a subsubset of signal components representing signal components of the first set of signal components representing the signal in the respective subband of the respective channel of the multi-channel signal .
The method according to claim 3, wherein the
inter-channel component analysis comprises applying at least one component analysis for determining at least one subset of signal components of the second set of signal components, wherein each of the at least one component analysis is associated with at least two channels of the at least two channels of the multi-channel signal and with a subband of a frequency band associated with the respective at least two channels of the at least two channels, each determined subset of signal components representing signal components of the second set of signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband.
The method according to one of claim 1 to 4, comprising converting a time-domain representation of the multichannel signal to said time-frequency
representation of the multi-channel signal, the time-frequency representation comprising at least two sets of time-frequency representatives, each set of the at least two sets of time-frequency representatives being associated with one channel of the at least two channels.
The method according to claim 5, said converting comprising for each of the at least two channels:
transforming the time-domain representation of the respective channel of the time-domain representation into a frequency domain representation; and
performing a two-dimensional discrete cosine
transformation based on the frequency domain
representation associated with the respective channel in order to determine the set of time-frequency
representatives associated with the respective channel.
The method according to one of the preceding claims, comprising at least one of:
extracting from each of at least one set of signal components of the first set of signal components a separate subset of relevant signal components, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the first set of signal components in accordance with a first accuracy criteria; and
extracting from each of at least one set of signal components of the second set of signal components a separate subset of relevant signal components, wherein each subset of relevant signal components represents a part of the multi-channel signal associated with the respective set of signal components of the second set of signal components in accordance with a second accuracy criteria .
The method according to claim 7, wherein said extracting comprises at least one of:
determining a measure of relevance for each signal component of each of the at least one set of signal components of the first set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the first set of signal components; and
determining a measure of relevance for each signal component of each of the at least one set of signal components of the first set of signal components, the measure of relevance indicating the relevance of the associated signal component with respect to the part of multi-channel signal associated with the respective set of signal components of the first set of signal components .
The method according to claim 7 or 8, comprising at least one of:
applying the second component analysis based on the at least one extracted subset of relevant signal components of the first set of signal components; and
the down-sampled representation of the multi-channel signal comprises the at least one extracted subset of relevant signal components of the second set of signal components . The method according to one of the preceding claims, comprising determining signal scene information, the signal scene information comprising:
the down-sampled representation of the multi-channel signal ;
a first set of analysis components associated with the first component analysis, wherein the first set of analysis components is configured to be used in combination with the first set of signal components for reconstruction of the multi-channel signal; and a second set of analysis components associated with the second component analysis, wherein the second set of analysis components is configured to be used in combination with the second set of signal components for reconstruction of the first set of signal components.
performing a concatenated reconstruction of a
multi-channel signal based on a down-sampled
representation of the multi-channel signal,
wherein said concatenated reconstruction comprises: applying a first component synthesis for determining a first set of reconstructed signal components
representing the multi-channel signal; and
applying a second component synthesis based on the first set of reconstructed signal components for determining reconstructed representation of the multi-channel signal . 12. The method according to claim 11, the multi-channel signal being associated with at least two channels, the first component synthesis representing an inter-channel synthesis of at least two channels of the at least two channels associated with the multi-channel signal, and the second component synthesis comprising for each channel of at least one channel of the at least two channels applying an intra-channel component synthesis for determining a reconstructed signal of the respective channel .
The method according to claim 12, wherein each channel of the at least one channel of the at least two channels is associated with a frequency band, and the
intra-channel component synthesis for a respective channel comprises applying a component synthesis for each subband of at least one subband of the frequency band of the respective channel for determining a subsignal of reconstructed signal components associated with the respective subband of the respective channel for each subband of the at least one subband.
The method according to claim 13, wherein the
inter-channel component synthesis comprises applying at least one component synthesis for determining at least one subset of reconstructed signal components of the first set of reconstructed signal components, wherein each of the at least one component synthesis is associated with at least two channels of the at least two channels of the multi-channel signal and with a subband of a frequency band associated with the respective at least two channels of the at least two channels, each determined subset of reconstructed signal components representing reconstructed signal components of the first set of reconstructed signal components representing the signals of the respective at least two channels of the multi-channel signal in the respective subband.
The method to one of claims 11 to 14, comprising accessing signal scene information, the signal scene information comprising :
a first set of analysis components associated with the second component synthesis, wherein the first set of analysis components is configured to be used in combination with the first set of reconstructed signal components for reconstruction of the multi-channel signal; and
a second set of analysis components associated with the first component synthesis, wherein the second set of analysis components is configured to be used in combination with down-sampled representation of the multi-channel signal for reconstruction of the second set of signal components.
An apparatus comprising:
means for performing a concatenated component analysis based on a time-frequency representation of a
multi-channel signal for determining a down-sampled representation of the multi-channel signal,
wherein said concatenated component analysis is performed by:
means for applying a first component analysis for determining a first set of signal components representing the multi-channel signal; and means for applying a second component analysis based on the first set of signal components for determining a second set of signal components representing the multi-channel signal.
The apparatus according to claim 16, wherein the apparatus is one of:
a chip;
an integrated circuit;
an audio device; and
a video device.
An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code, with the at least one processor, configured to cause the apparatus at least to perform:
performing a concatenated component analysis based on a time-freguency representation of a multi-channel signal for determining a down-sampled representation of the multi-channel signal,
The apparatus according to claim 18, the multi-channel signal being associated with at least two channels, wherein the at least one memory and the computer program code, with the at least one processor, are configured to cause the apparatus to perform the first component analysis by applying for each channel of at least one channel of the at least two channels an intra-channel component analysis for determining a subset of signal components of said first set of signal components representing the signal of the respective channel, and the second component analysis representing an
inter-channel component analysis of at least two channels of the at least two channels associated with the multi-channel signal.
The apparatus according to one of claims 18 to 19, wherein the apparatus is one of:
means for performing a concatenated reconstruction of a multi-channel signal based on a down-sampled
wherein said concatenated reconstruction is performed by :
means for applying a first component synthesis for determining a first set of reconstructed signal components representing the multi-channel signal; and means for applying a second component synthesis based on the first set of reconstructed signal components for determining reconstructed representation of the multi-channel signal.
22. The apparatus according to claim 21, wherein the apparatus is one of:
- an integrated circuit;
23. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code, with the at least one processor, configured to cause the apparatus at least to perform:
applying a second component synthesis based on the first set of reconstructed signal components for determining reconstructed representation of the multi-channel signal .
24. The apparatus according to claim 23, the first component synthesis representing an inter-channel synthesis of at least two channels of the at least two channels associated with the multi-channel signal, and the second component synthesis comprising for each channel of at least one channel of the at least two channels applying an intra-channel component synthesis for determining a reconstructed signal of the respective channel.
25. The apparatus according to one of claims 23 to 24, wherein the apparatus is one of :
26. A computer program code causing an apparatus to perform the following when executed on a processor:
applying a second component analysis based on the first set of signal components for determining a second set of signal components representing the multi-channel signal. 27. A computer readable storage medium in which the computer program code according to claim 26 is stored.
- performing a concatenated reconstruction of a
representation of the multi-channel signal, wherein said concatenated reconstruction comprises: applying a first component synthesis for determining a first set of reconstructed signal components
29. A computer readable storage medium in which the computer program code according to claim 28 is stored.
30. A system, comprising:
a first apparatus according to one of claims 16 to 20; and
a second apparatus according to one of claims 21 to 25.
PCT/EP2009/064380 2009-10-30 2009-10-30 Coding of multi-channel signals WO2011050853A1 (en)
PCT/EP2009/064380 WO2011050853A1 (en) 2009-10-30 2009-10-30 Coding of multi-channel signals
EP09751867A EP2494547A1 (en) 2009-10-30 2009-10-30 Coding of multi-channel signals
CN200980162219.9A CN102598120B (en) 2009-10-30 2009-10-30 Coding of multi-channel signals
US13/502,367 US9269359B2 (en) 2009-10-30 2009-10-30 Coding of multi-channel signals
US20120019616A1 (en) * 2010-07-26 2012-01-26 Chen wei hong 3D image capturing and playing device
US9812150B2 (en) 2013-08-28 2017-11-07 Accusonus, Inc. Methods and systems for improved signal decomposition
US20150264505A1 (en) 2014-03-13 2015-09-17 Accusonus S.A. Wireless exchange of data between devices in live events
US10468036B2 (en) 2014-04-30 2019-11-05 Accusonus, Inc. Methods and systems for processing and mixing signals using signal decomposition
US9883308B2 (en) * 2014-07-01 2018-01-30 Electronics And Telecommunications Research Institute Multichannel audio signal processing method and device
US20080175394A1 (en) * 2006-05-17 2008-07-24 Creative Technology Ltd. Vector-space methods for primary-ambient decomposition of stereo audio signals
TW327223B (en) 1993-09-28 1998-02-21 Sony Co Ltd Methods and apparatus for encoding an input signal broken into frequency components, methods and apparatus for decoding such encoded signal
2009-10-30 EP EP09751867A patent/EP2494547A1/en active Pending
AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, 20 May 2006 (2006-05-20) - 23 May 2006 (2006-05-23), XP040373142 *
AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, 5 October 2006 (2006-10-05) - 8 October 2006 (2006-10-08), XP040373293 *
GOODWIN M M ET AL: "Primary-Ambient Signal Decomposition and Vector-Based Localization for Spatial Audio Coding and Enhancement", 2007 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING 15-20 APRIL 2007 HONOLULU, HI, USA, IEEE, PISCATAWAY, NJ, USA, 15 April 2007 (2007-04-15), pages I - 9, XP031462785, ISBN: 978-1-4244-0727-9 *
GOODWIN MICHAEL M ET AL: "A Frequency-domain Framework for Spatial Audio Coding Based on Universal Spatial Cues", AUDIO ENGINEERING SOCIETY CONVENTION PAPER; [12056], NEW YORK, NY; US, vol. 120TH, no. 6751, 20 May 2006 (2006-05-20), pages 1 - 12, XP002477914 *
US7516074B2 (en) 2009-04-07 Extraction and matching of characteristic fingerprints from audio signals
KR101178060B1 (en) 2012-08-30 Multichannel Decorrelation in Spatial Audio Coding
TWI307248B (en) 2009-03-01 Apparatus and method for generating multi-channel synthesizer control signal and apparatus and method for multi-channel synthesizing
KR101531239B1 (en) 2015-07-06 Apparatus For Decoding multi-object Audio Signal
RU2512103C2 (en) 2014-04-10 Noise background, apparatus for processing noise background, method of providing noise background parameters, method of providing spectral representation of audio signal noise background, computer program and encoded audio signal
KR101143225B1 (en) 2012-05-21 Complex-transform channel coding with extended-band frequency coding
RU2536679C2 (en) 2014-12-27 Time-deformation activation signal transmitter, audio signal encoder, method of converting time-deformation activation signal, audio signal encoding method and computer programmes
JP2012078866A (en) 2012-04-19 Audio coding system using characteristics of decoded signal to adapt synthesized spectral components
US8326638B2 (en) 2012-12-04 Audio compression
KR100978018B1 (en) 2010-08-25 Parametric representation of spatial audio
US9093120B2 (en) 2015-07-28 Audio fingerprint extraction by scaling in time and resampling
US8831958B2 (en) 2014-09-09 Method and an apparatus for a bandwidth extension using different schemes
US20110249821A1 (en) 2011-10-13 encoding of multichannel digital audio signals
EP2490215A2 (en) 2012-08-22 Method and apparatus to extract important spectral component from audio signal and low bit-rate audio signal coding and/or decoding method and apparatus using the same
EP1575032A2 (en) 2005-09-14 Lossless audio coding/decoding method and apparatus
US20150332679A1 (en) 2015-11-19 Method and apparatus for compressing and decompressing a higher order ambisonics representation for a sound field
EP3246919A1 (en) 2017-11-22 Improved harmonic transposition
JP2015527610A (en) 2015-09-17 Method and apparatus for improving rendering of multi-channel audio signals
US20060064299A1 (en) 2006-03-23 Device and method for analyzing an information signal
CN101826071B (en) 2011-09-14 Adaptive hybrid transform for signal analysis and synthesis
US20120150890A1 (en) 2012-06-14 Method of searching for multimedia contents and apparatus therefor
ES2627212T3 (en) 2017-07-27 Modification of code words in dictionary used for efficient coding of digital media spectral data
JP2009524101A (en) 2009-06-25 Encoding / decoding apparatus and method
JP5371931B2 (en) 2013-12-18 Encoding device, decoding device, and methods thereof
KR20110113765A (en) 2011-10-18 Audio signal encoding or decoding
Country of ref document: US
Ref document number: 4254/CHENP/2012
Country of ref document: IN