1. Field of the Invention
This invention pertains in general to use video compression technology to encode and decode video frames, and in particular, to dynamically encoding a remotely displayed user applications based on feedback from a decoder and/or network conditions.
2. Description of the Related Art
Video compression is useful for transmission of digital video over a variety of bandwidth-limited networks, or for storage constrained applications. For example, the broadcast transmission of digital video at 24-bit per pixel sampled at 720 by 480 spatial resolution and 30 frames per second (fps) temporal resolution would require a bit rate of above 248 Mbps! Taking another example of supporting web browser applications with rich media content in a client-server architecture within a wireless network, bandwidth limitations of the wireless network itself may comprise one of the major limiting factors in fully utilizing the client-server architecture. Client devices, such as mobile phones, may additionally be resource-constrained with respect to the device's capabilities, including processing power, memory and battery life limitations. Compounding this, web browser applications are continually embracing rich media content, such as digital video and audio, which in turn poses further challenges for a client-server architecture. For applications such as digital television broadcasting, satellite television, Internet video streaming, video conferencing and video security over a variety of networks, limited transmission bandwidth or storage capacity stresses the demand for higher video compression ratios.
To improve compression efficiency, currently available coding standards, such as MPEG-1, MPEG-2, MEPG4 and H.264/AVC etc., remove information redundancy spatially within a video frame and temporally between video frames. The goal of video compression systems is to achieve the best fidelity (or the lowest distortion D) given the capacity of a transmission channel, subject to the coding rate constraint R(D). Most currently available coding standards employ some rate control mechanism to achieve such a goal. Prior art rate control algorithms are designed to dynamically adjust encoder parameters to achieve a target bitrate. They allocate a budget of bits to each group of pictures, individual picture and/or sub-pictures, in a video sequence. However, this optimization task is complicated by the fact that various coding options show varying efficiency at different bit rates and with different scene content. It is further complicated by lack of real time decoder feedback to guide the encoder parameters adjustment.
As alluded to above, existing video coding systems in use today, such as video broadcasting, are often designed to be open-loop systems, where there is no mechanism for a decoder to signal back to encoder. Most conventional encoders use a first-in-first-out (FIFO) on encoding output with a rate control method such as leaky bucket. As the FIFO buffer fills up, the encoder turns up a quantizer to reduce the bit rate. Other rate control techniques known to those skilled in the art may be used as well. Although the conventional rate control method keeps the peak and/or average bit rate controlled, it does not have the capability to dynamically tune the encoder according to available network bandwidth provided by decoder feedback or other source.
Another challenge faced by a video coding system is to maintain data integrity across noisy network channels. Transmission error detection and correction becomes more critical when compressed data are transmitted. A single transmission error is able to render a large amount of data useless. Conventional encoders deal with transmission loss/errors in three primary ways. First, a video frame is segmented into blocks. If a block gets corrupted, only part of the frame will be lost. This minimizes the effects of the error. Second, traditional encoders send periodic intra frames (I-frames) to clean up any previous transmission errors and/or due to a rigid group of pictures (GOP) structures. Third, encoders use redundant information available. This extra data can be used to replace corrupted data. However, a conventional encoder has to make intelligent error recovery decisions without feedback from decoder. For example, to use redundant information effectively, encoder has to send a lot of redundant data over the network. This increases the overall data transmitted on the network, and increases the likelihood of more errors. Periodically sending I-frames to the decoder sometimes may also seem unnecessary and wastes bandwidth when there are no errors. Intra frames are conventionally intra coded without using any other references frames for temporal prediction. Consequently, intra frames require more bandwidth to send over network than temporally predicted frames. If a decoder were able to signal back to encoder when and where an error is detected, the encoder can then decide whether a re-transmission is needed, or to change the current/next frame to be transmitted to allow decoder to move past the bad data.
In addition to rate and error control described above, a video encoding system often needs to be able to respond to requests from user or decoder itself. For example, one important encoding characteristic is resolution. A user may want to change the apparent or physical resolution of their screen for certain applications or due to network congestion. In another example, decoder input buffer fullness affects how many encoded frames a decoder can take, which in turn affects encoder rate control. In yet another example, a user may want to change his/her current mono sound to stereo sound for audio source configuration, or change the current audio sample rate (in KHz) for special sound effect. To respond to such requests, an encoder needs to change its encoding parameters. Conventional encoders lack such capability due to receive and process decoder feedback.
Hence, there is, inter alia, a lack of a system and method that provides decoder feedback to encoder within a video processing system.