Video compression is used in many current and emerging products. It is at the heart of digital television set-top boxes (“STB”), digital satellite systems (“DSS”), high definition television (“HDTV”) decoders, digital versatile disk (“DVD”) players, video conferencing, Internet video and multimedia content, and other digital video applications. Without video compression, the number of bits required to represent digital video content can be extremely large, making it difficult or even impossible for the digital video content to be efficiently stored, transmitted, and/or viewed.
The digital video content comprises a stream of pictures that can be displayed as an image on a television receiver, computer monitor, or other electronic device capable of displaying digital video content. A picture that is displayed in time before a particular picture is in the “backward direction” in relation to the particular picture. Likewise, a picture that is displayed in time after a particular picture is in the “forward direction” in relation to the particular picture.
Video compression is accomplished in a video encoding, or coding, process in which each picture is encoded as either a frame or as two fields. Each frame comprises a number of lines of spatial information. For example, a frame may contain 480 horizontal lines. Each field contains half the number of lines in the frame. For example, if the frame comprises 480 horizontal lines, each field comprises 240 horizontal lines. In a typical configuration, one of the fields comprises the odd numbered lines in the frame and the other field comprises the even numbered lines in the frame. The field that comprises the odd numbered lines will be referred to as the “top” field hereinafter and in the appended claims, unless otherwise specifically denoted. Likewise, the field that comprises the even numbered lines will be referred to as the “bottom” field hereinafter and in the appended claims, unless otherwise specifically denoted. The two fields can be interlaced together to form an interlaced frame.
The general idea behind video coding is to remove data from the digital video content that is “non-essential.” The decreased amount of data then requires less bandwidth for broadcast or transmission of the representation of the original signal.
After the compressed (encoded) video data has been transmitted, it must be decoded, or decompressed. In this process, the transmitted video data is processed to generate approximation data that is substituted into the video data to replace the “non-essential” data that was removed in the above-mentioned coding process.
Thus, video coding transforms the digital video content into a compressed form that can be stored using less space and transmitted using less bandwidth than uncompressed digital video content. It does so by taking advantage of temporal and spatial redundancies in the pictures of the video content. The resultant digital video content can then be stored in a storage medium such as a hard drive, DVD, or some other non-volatile storage unit.
There are numerous video coding methods that compress the digital video content. Consequently, video coding standards have been developed to standardize the various video coding methods so that the compressed digital video content is rendered in formats that a majority of video encoders and decoders can recognize. For example, the Motion Picture Experts Group (“MPEG”) and International Telecommunication Union (“ITU-T”) have developed video coding standards that are in wide use today. Examples of these standards include the MPEG-1, MPEG-2, MPEG-4, ITU-T H.261, and ITU-T H.263 standards. However, with the current increased demand for higher resolutions, more complex graphical content, and faster transmission time, there exist a need for better video compression methods. To this end, a new video coding standard is currently being developed. This new video coding standard is called the MPEG-4 Part 10 Advanced Video Coding (AVC)/H.264 standard.
Most modern video coding standards, including the MPEG-4 Part 10 AVC/H.264 standard, are based in part on a temporal prediction with a motion compensation (“MC”) algorithm and a transform domain coding algorithm. Temporal prediction with motion compensation is used to remove temporal redundancy between successive pictures in a digital video broadcast. The temporal prediction with motion compensation algorithm typically utilizes one or two reference pictures to encode a particular picture. Thus, by comparing the particular picture that is to be encoded with one of the reference pictures, the temporal prediction with motion compensation algorithm can take advantage of the temporal redundancy that exists between the reference picture and the particular picture that is to be encoded, and encode the picture with a higher amount of compression than if the picture were encoded without using the temporal prediction with motion compensation algorithm. One of the reference pictures is in the backward direction in relation to the particular picture that is to be encoded. The other reference picture is in the forward direction in relation to the particular picture that is to be encoded.
Transform domain coding is used to remove spatial redundancy within each picture or temporally predicted residual picture. A residual picture is the difference between a picture and a picture that is temporally predicted from that picture. Each picture or temporally predicted residual picture comprises a number of blocks of pixels. Each block refers to an N by M group of pixels where N refers to the number of columns of pixels in the block and M refers to the number of rows of pixels in the block. Each block in the picture or temporally predicted residual picture is represented by an N by M array of luminance and chrominance coefficients which correspond to each pixel in the blocks' N by M grid of pixels. Each luminance coefficient represents the brightness level, or luminance, of its corresponding pixel. Each block in the picture or temporally predicted residual picture is also represented by an N by M array of chrominance coefficients which correspond to each pixel in the blocks' N by M grid of pixels. Each chrominance coefficient represents the color content, or chrominance, of its corresponding pixel. The term “picture” will be used hereinafter and in the appended claims, unless otherwise specifically denoted, to mean either a picture or a temporally predicted residual picture.
Most pictures have smooth color variations, with the fine details being represented as sharp edges in between the smooth variations. The smooth variations in color can be termed as low frequency variations and the sharp variations as high frequency variations. The smooth variations in color, or low frequency components of the picture, constitute the base of an image, and the edges which give detail to the picture, or the high frequency components, add upon the smooth variations in color to refine the picture. The combination of the low and high frequency components results in a detailed image.
Typically, the values of the luminance coefficients only vary slightly between the most of the pixels in a particular picture. Consequentially, in many pictures, most pixels contain more of the low frequency component than the high frequency component. In other words, most of the energy of a signal containing the digital video content lies at low frequencies.
Transform domain coding takes advantage of the fact that most of the energy of a signal containing the digital video content lies at low frequencies. Transform domain coding transforms the luminance coefficients in each N by M array from the spatial domain to the frequency domain. The transformed N by M array comprises coefficients which represent energy levels in the frequency domain. As used hereinafter and in the appended claims, unless otherwise denoted, the coefficients of the transformed N by M array will be referred to as “frequency coefficients.” Once the luminance coefficients have been transformed into frequency coefficients, various compression techniques can then be performed on the contents of picture in the frequency domain that would otherwise be impossible to perform in the spatial domain.
The N by M array of frequency coefficients is two dimensional and must be converted into a one dimensional array of frequency coefficients so that the encoder or decoder can use the frequency coefficients to encode or decode the picture. The encoder generates the one dimensional array of frequency coefficients by scanning the two dimensional array of frequency coefficients using a particular scanning path. The scanning path refers to the order in which the frequency coefficients in the two dimensional array are scanned and output by the encoder into the one dimensional array.
A common scanning path that is used by an encoder to scan the frequency coefficients is a zig-zag scanning path. FIG. 1 illustrates two variations of zig-zag scanning paths that are currently used to scan a four by four array of frequency coefficients. As shown in FIG. 1, the first zig-zag scanning path 100 goes in a zig-zag order starting with an upper left coefficient (0) and ending with a lower right coefficient (15) of the array of frequency coefficients. The second zig-zag scanning path 101 is similar to the first in that it starts with the upper left coefficient (0) and ends with the lower right coefficient (15). However, as shown in FIG. 1, the two zig-zag scanning paths 100, 101 differ slightly in the order that the coefficients are scanned. FIG. 1 also shows one non-zig-zag scanning path 102 that is also prior art. Other prior art scanning paths for an 8 by 8 array of frequency coefficients can be found in MPEG-2 (Generic Coding of Moving Pictures and Associated Audio, Draft of International Standard, ISO/IEC 13818-2, March 1994).
It is preferable for the encoder to first scan the high-energy low frequency coefficients and then scan the low-energy high frequency coefficients. Scanning the low frequency coefficients before the high frequency coefficients places the low frequency coefficients before the high frequency coefficients in the resulting one dimensional array of coefficients. This particular order allows efficient coding and compression of the picture.
The zig-zag scanning path scans the two dimensional array of frequency coefficients without any significant bias towards the horizontal or vertical frequency directions. However, for interlaced material, energy tends to be concentrated along the vertical direction.
Thus, there is a need in the art for scanning paths that allow for more compression than do traditional zig-zag scanning paths. In particular, there is a need for a scanning path(s) which is biased in the vertical direction such that non-zero frequency coefficients would be captured first, thereby allowing for better compression efficiency.