1. Field of the Invention
This invention relates to an image coding apparatus and an image decoding apparatus for use with a system which performs high efficiency coding or decoding of moving pictures to perform efficient transmission or storage of images, and more particularly to an image coding apparatus and an image decoding apparatus which can be applied to processing of, for example, a digital broadcasting system which is performed using a satellite or a ground wave or cable communication network, a digital video disk, a mobile video phone, a PHS video phone or a data base for images.
2. Description of the Prior Art
As a representative one of conventional high efficiency coding systems, the MPEG2 is known which is an international standard system recommended by the ISO/IEC/JTC1/SC29/WG11. For example, “Image Information Engineering and Broadcasting Techniques”, Journal of the Television Engineering Society of Japan, April, 1995 explains the MPEG as a theme of special editing. A coding system of the MPEG2 is disclosed in “3-2 Video Compression” of the same document, pp. 29-60.
The coding system of the MPEG2 is described below.
FIG. 31 is a block diagram showing a basic construction of an ordinary encoder of the MPEG2, and FIG. 32 is a block diagram showing a basic construction of an MPEG2 decoder. Referring to FIGS. 31 and 32, reference numeral 1 denotes a frame re-arranging unit, 2 a subtracting unit, reference characters 3a and 3b denote each an inter(interframe)/intra(intraframe) switching selector, reference numeral 4 denotes a converting unit, 5 a quantizing unit, 6 a reverse quantizing unit, 7 a reverse converting unit, 8 an adder, 9 a first frame memory, 10 a second frame memory, 11 a forward direction motion compensating unit, 12 a bidirection motion compensating unit, 13 a backward direction motion compensating unit, 151 a motion estimating unit, 16 a coding control unit, 17 a variable length coding unit, and 18 a buffer.
Further, reference numeral 100 denotes input image data in the form of digital data, 101 re-arranged input image data, 102 a predictive error image, 103 an original input image or predictive error image, 104 a conversion coefficient, 105 a quantization coefficient, 106 a reverse quantized conversion coefficient, 107 reversed converted image data, 108 a locally decoded image, 109 a reference image from the first frame memory, 110 a reference image from the second frame memory, 111 a forward direction motion predicted image, 112 a bidirection motion predicted image, 113 a backward direction motion predicted image, 115 a determined predicted image, 117 a control signal to the selector, 118 a control signal to the converting unit 4, 119 an adaptive quantization value, 120 a variable length coder, 121 a bit stream, 123 a motion vector, 124 a reference image, and 125 an intra/inter switching signal.
Operation of the conventional image encoder is described below with reference to FIG. 31.
First, an input image signal 100 in the form of a digital signal is inputted to the frame re-arranging unit 1, by which picture frames to be coded are re-arranged.
FIG. 33 illustrates such re-arrangement. Referring to FIG. 33, reference character I denotes an intra (intraframe) coded picture, P an interframe coded picture, and B a bidirectional predictive coded picture. It is to be noted that reference numerals 1 to 10 represent an order in time in which they are displayed.
The first frame is first coded as an I picture, and then the fourth frame is coded as a P picture, whereupon the already coded I picture is used as a reference frame for prediction.
Then, the second frame is coded as a B picture. Thereupon, the I picture of the first frame and the P picture of the fourth frame coded already are used as reference frames for the prediction. In FIG. 33, each arrow mark represents a direction in which prediction is performed.
Thereafter, coding is performed in the construction of I, B, B, P, B, B, P, . . . by similar processing. Accordingly, the action of the frame re-arranging unit 1 is to re-arrange the input image signal 100, in which the picture frames are arranged in order of time, so that they appear in order of coding in order to allow the processing described above.
Subsequently, since predictive coding is not performed for the I picture mentioned above, when the re-arranged image 101 is inputted as it is to the selector 3a, it is transmitted as a selector output 103 to the converting unit 4. On the other hand, for predictive coding for the P picture or the B picture mentioned above, the re-arranged image 101 is subtracted from a predicted image 115 by the subtracting unit 2, and a predictive error image 102 is transmitted as the selector output 103 to the converting unit 4.
Then, the selector output 103 is inputted to the converting unit 4, and a conversion coefficient 104 is outputted from the converting unit 4. The conversion coefficient 104 passes the quantizing unit 5, and a quantization coefficient 105 is obtained from the quantizing unit 5. The quantization coefficient 105 is coded into a variable length code by the variable length coding unit 17, and a variable length coded word 120 is outputted from the variable length coding unit 17.
The quantization coefficient 105 is, on the other hand, inputted to the reverse quantizing unit 6, and a quantization coefficient 106 is outputted from the reverse quantizing unit 6.
Further, the quantization coefficient 106 is reverse converted back to an image level by the reverse converting unit 7, and image data 107 is outputted from the reverse converting unit 7. The image data 107 is, where it is data of the I picture, added to a predicted image 116 selected by the adding unit 8, and a locally decoded image 108 is outputted from the adding unit 8.
It is to be noted that the locally decoded image 108 is written as it is into the first frame memory 9 when it is an I picture, but, when it is a P picture, it is written into the second frame memory 10.
On the other hand, when the locally decoded image 108 is a B picture, it is written into neither the first frame memory 9 nor the second frame memory 10.
Thereafter, when the locally decoded image 108 is a P picture, since it is used only for forward direction prediction, a reference image 124 in the first frame memory 9 is read out, and motion prediction is performed for each macroblock (basic unit for processing of 16 pixels×16 lines) by the motion estimating unit 151. The motion estimating unit 151 thus selects one of the macroblocks which has a value nearest to that of the current macroblock as a predicted image, and simultaneously outputs a motion vector 123 therefrom.
The motion vector 123 is inputted to the motion compensating units 11, 12 and 13 surrounded by a dotted line in FIG. 31, and motion predictive pictures are outputted from the motion compensating units 11, 12 and 13.
In this instance, the forward direction motion compensating unit 11 produces a forward direction motion predicted image 111 using a reference image 109 from the first frame memory 9 and outputs a thus determined predicted image 115.
Further, as described hereinabove, the locally decoded images 108 of all macroblocks in a P picture are written into the second frame memory. However, even with the P picture mentioned above, when the macroblocks thereof are intraframe (intra) coded, the frame re-arranged image 101 is outputted directly as the selector output.
Meanwhile, for a B picture, the procedure of coding processing is similar to that for a P picture described above, but different from the processing for a P picture, in that two reference frames are used for prediction.
The motion estimating unit 151 performs forward direction prediction using the reference image 109 from the first frame memory 9, backward direction prediction using a reference image 110 from the second frame memory 10, and bidirection prediction using both of the reference images 109 and 110 to select one of the prediction modes with which a value nearest to that of the current macroblock is obtained, and then outputs a motion vector 123.
In accordance with the thus determined prediction mode, in the motion compensating unit, one of the motion compensating units 11, 12 and 13 which corresponds to the determined prediction mode produces and outputs a predicted picture.
For example, when bidirection motion prediction is selected, the bidirection motion compensating unit 12 produces and outputs a predicted image 115 determined using a bidirection predicted image 112.
After coding of the B pictures of the second and third frames shown in FIG. 33 is completed, image data written in the second frame memory is transferred to the first frame memory. Thereafter, the P picture of the seventh frame is coded, and a decoded picture is written into the second frame memory.
Thereafter, B pictures (fifth and sixth frames) are coded by similar processing to that described above.
When the macroblocks are intraframe (intra) coded, the image 101 after frame re-arrangement is directly outputted as the selector output similarly as in the case of a P picture.
FIG. 32 is a block diagram of a conventional decoder. Referring to FIG. 32, reference character 22 denotes a variable length decoding unit, 107(a) an intra (intraframe) coded picture, and 107(b) a prediction error picture.
Subsequently, operation of the conventional decoder will be described.
A bit stream 121 is stored for a certain period of time into the receiving buffer 18, and a variable length coded word 120 is variable length decoded by the variable length decoding unit 22 and outputted as a quantization coefficient 105.
The processing procedure after this is quite similar to the local decoding processing of the encoder described hereinabove.
When the macroblock is intra decoded, a reverse converted image 107 makes an image 107(a) without passing the adding unit 8, but when the macroblock is inter (interframe) decoded, the reverse converted image data 107 makes an image 107(b). The image 107(b) is added to a predicted image 115 by the adding unit 8, and a decoded image 108 is outputted from the adding unit 8. The decoded image 108 is processed by the displayed frame re-arranging unit 38 such that such decoded images are re-arranged so that they appear in order of time, and finally, an output image 137 is outputted the displayed frame re-arranging unit 38.
The example of a conventional image coder and image decoder described above is a representative apparatus of a type which performs forward direction, bidirection and backward direction prediction coding in combination.
In the example, for coding of a P picture, only forward direction prediction is performed using the first frame memory to perform predictive coding. On the other hand, for coding of a B picture, one of the modes of forward direction prediction, backward direction prediction and bidirection prediction with which a minimum predictive error is provided is selected using the first and second frame memories.
Accordingly, as coding processing proceeds, decoded pictures written in the frame memories are erased. Consequently, for example, even if one of coded pictures processed in the past is similar to a picture of the currently coded frame, since the past decoded pictures have already been erased from the frame memories, the similar coded picture cannot be used for reference, resulting in a problem of lower image processing efficiency.