mode for most blocks in the b2 picture in other words after obtaining the motion vector of each block in the specified picture p 200 in the inter mode from the long term reference picture p 50 belonging to the same scene set b 2 the motion vectors of direct mode in the b2 picture are calculated from the motion vector of the co located block in the specified picture p 200 since the b2 picture and the specified picture p 200 belong to the scene set b 2 and the similarity between the scene set b 1 to which the reference picture p 50 belongs and the scene set b 2 is very high the direct mode can be selected as a coding mode for most blocks in the b2 picture accordingly the coding efficiency for the b2 picture is improved on the other hand if the motion vector of the co located block in the specified picture points to a short term reference picture the b picture is coded using the conventional expressions 1 and 2 s 153 in this time since the short term reference picture stored in the short term reference buffer belongs to the same scene set as the b picture and another scene set does not exist between the specified picture and the short term reference picture the forward motion vector and the backward motion vector of direct mode are determined using the conventional expressions 1 and 2 related to trd and trb representing time distance after one block of a b picture is coded.