Data pruning for video compression is an emerging technology in the area of video compression. There has been some prior work on data pruning to improve video coding efficiency.
One approach is vertical and horizontal line removal as in D. T. Vo, J. Sole, P. Yin, C. Gomila, and T. Nguyen, “Data Pruning-Based Compression Using High Order Edge-Directed Interpolation”, Thomson Research technical report, submitted to the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2009. This approach removes vertical and horizontal lines in video frames before encoding, and recovers the lines by non-linear interpolation after decoding. The positions of the lines are optimized to minimize the recovery error. There are a number of drawbacks to this approach. First, the positions of the lines have to be sent though a side channel and losslessly encoded, which could decrease overall coding efficiency. Second, the removal of lines may result in loss of information, leading to aliasing artifacts during recovery. For example, if there is a one-pixel horizontal line in the video frame, it may be lost due to line removal and not recoverable at the decoder side. Third, in a video shot, the line positions have to be the same for all frames in the shot. If there is fast motion in the shot, it may result in false removal of the lines containing important information. Although line removal can be designed to adapt to object motion in a shot, removing different lines in different frames in a shot may create artificial motion, therefore it could adversely affect the video coding process that involves motion estimation and motion compensation.
Another category of approaches is based on block or region removal, such as in P. Ndjiki-Nya, T. Hinz, A. Smolic, and T. Wiegand, “A generic and automatic content-based approach for improved H.264/MPEG4-AVC video coding,” IEEE International Conference on Image Processing (ICIP), 2005; Chunbo Zhu, Xiaoyan Sun, Feng Wu, and Houqiang Li, “Video Coding with Spatio-Temporal Texture Synthesis,” IEEE International Conference on Multimedia and Expo (ICME), 2007; and Chunbo Zhu, Xiaoyan Sun, Feng Wu, and Houqiang Li, “Video coding with spatio-temporal texture synthesis and edge-based inpainting,” IEEE International Conference on Multimedia and Expo (ICME), 2008. These approaches are similar to line removal except that blocks or regions are removed instead of lines. The removed blocks or regions are recovered at the decoder side by interpolation or inpainting. The drawbacks of these approaches are similar to those of line removal. However, using blocks rather than lines may gain more flexibility for data pruning, therefore alleviating the problem of information loss, but other problems are the same as those of the line removal approach.
Image or video epitome, such as in N. Jojic, B. Frey, and A. Kannan, “Epitomic analysis of appearance and shape,” IEEE International Conference on Computer Vision (ICCV), 2003, and V. Cheung, B. Frey, and N. Jojic, “Video epitomes,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2005 is another technique that could potentially be used for data pruning for compression. Epitome based approaches divide an image (or video) into patches, and represent an image as a small miniature containing representative patches and a surjective map that maps the patches in the image to those in the epitome miniature. The small miniature (i.e. epitome) can be deemed as a compressed version of the original image or video, therefore epitome can potentially be used for compression purposes. However, the problem of applying epitome to data pruning is that the surjective map also has to be sent to a decoder as side information. Furthermore, the surjective map has to be losslessly encoded to avoid correspondence error and artifacts, which could substantially reduce encoding efficiency.