Recently, a person and others are photographed in a studio called a virtual studio in a television station and others, the photographed picture and a picture generated by computer graphics (CG) are synthesized and the synthesized picture is broadcast.
A weather forecast program will be described in detail as an example below. First, a performer is instructed to stand in a blue background and is photographed with a television camera. Video data from the television camera and the video data of an object generated by CG such as a Japanese map including weather information are synthesized so as to generate synthetic video data related to a synthetic picture to be provided to television (TV) viewers.
FIG. 3 shows a concrete example of a virtual studio.
A performer 101 is photographed in front of a blue background 102. Blue and inconspicuous markers (patterns) M1 to M9 are added to the background 102. Each position on three dimensions of the markers M1 to M9 is measured beforehand on the basis of a specific fixed coordinate system (world coordinates)
Next, a picture generator 103 will be described. The picture generator 103 is provided with a mobile television (TV) camera 104 for photographing the performer 101 in front of the background 102. The markers M1 to M9 can be slightly recognized by the TV camera 104 though the markers are inconspicuous because they are located in the blue background 102 as described above. Therefore, the three-dimensional positions of the markers M1 to M9 on the basis of the TV camera 104 can be measured.
The picture generator 103 is also provided with a markers' positional information storage 105 for storing the three-dimensional positional information PDW of markers M1 to M9 on the basis of the world coordinates. The picture generator 103 is also provided with a camera parameter calculator 106 for acquiring the three-dimensional position and the orientation of the TV camera 104 on the basis of the world coordinates based upon the three-dimensional positional information PDT obtained with the TV camera 104 of the markers M1 to M9 on the basis of the TV camera 104 and three-dimensional positional information PDW read from the markers' positional information storage 105.
Suppose that relationship between the three-dimensional positions of the markers M1 to M9 on the basis of the world coordinates and the three-dimensional positions of the markers M1 to M9 on the basis of the TV camera 104 is equivalent to relationship between rotational transform R and translational transform T. According to the camera parameter calculator 106, the three-dimensional position of the TV camera 104 on the basis of the world coordinates is T and the orientation is R. The acquired information of the three-dimensional position and the orientation is output from the camera parameter calculator 106 as the viewpoint information POV of the TV camera 104.
The picture generator 103 is also provided with a CG information storage 107 for storing the three-dimensional positional information PCG of an object generated by CG on the basis of the world coordinates, for example, the information of in what position in three-dimensional space "Japanese map including weather information" generated by CG virtually exists. In this case, in the CG information storage 107, the three-dimensional positional information PCG on the basis of the world coordinates of multiple objects generated by CG is stored. The three-dimensional positional information PCG of a suitable object generated by CG is selectively read from the CG information storage 107 by a control unit described later and used. That is, the three-dimensional positional information PCG of a predetermined object generated by CG is read according to a reading request signal RRQ supplied to the CG information storage 107 from the control unit.
The picture generator 103 is also provided with a CG data generator 108 for generating two-dimensional video data VDc corresponding to the projected picture of a predetermined object generated by CG to which perspective transformation viewed from the viewpoint of the TV camera 104 is applied based upon the viewpoint information POV of the TV camera 104 output from the camera parameter calculator 106 and the three-dimensional positional information PCG of the predetermined object generated by CG read from the CG information storage 107. In this case, in the CG data generator 108, two-dimensional video data (corresponding to the picture to which perspective transformation is applied) VDc for "Japanese map including weather information" for example which looks as if it were viewed from the viewpoint (the three-dimensional position and the orientation) of the TV camera 104 is generated.
The picture generator 103 is also provided with a delay unit 109 for adjusting time for delaying video data VDa output from the TV camera 104. In this case, delay time by the delay unit 109 is set so that when two-dimensional video data VDc in a case viewed from the viewpoint of the TV camera 104 at time T1 is output from the CG data generator 108, video data VDa obtained by photographing with the TV camera 104 at the time T1 is output from the delay unit 109 as video data VDb.
That is, it takes some time for calculation in the camera parameter calculator 106 and the CG data generator 108 and when it takes t seconds since three-dimensional positional information PDT at certain time is supplied from the TV camera 104 to the camera parameter calculator 106 until two-dimensional video data VDc corresponding to the three-dimensional positional information PDT is output from the CG data generator 108, delay time by the delay unit 109 is t seconds.
If two-dimensional video data VDc output from the CG data generator 108 as described later and video data VDb output from the delay unit 109 are synthesized by adjusting time in the delay unit 109 as described above, a synthetic picture which looks as if it is a picture obtained by photographing an object generated by CG and the performer 101 respectively existing in the same space can be acquired.
The picture generator 103 is also provided with an .alpha. blender 110 for mixing two-dimensional video data VDc output from the CG data generator 108 and video data VDb output from the delay unit 109 and a key generator 111 for acquiring the value of .alpha. supplied to the .alpha. blender 110 as a key signal based upon video data VDa output from the TV camera 104.
In the key generator 111, operation for separating a blue picture element and a picture element which is not completely blue in video data VDa is executed. The value of .alpha. is output every picture element of video data VDa from the key generator 111. In this case, .alpha.=0 in a blue picture element, .alpha.=1 in a picture element which is not completely blue and further, 0 &lt;.alpha.&lt;1 in a halftone part.
The TV camera 104 photographs the performer 101 explaining a weather forecast for example using the blue background 102. Therefore,as the projected picture of the background 102 is blue, .alpha.=0 in picture elements corresponding to the background 102 in video data VDa. In the meantime, as the projected picture of the performer 101 is not blue, .alpha.=1 in picture elements corresponding to the performer 101 in video data VDa. Further, as the projected picture of a boundary between the performer 101 and the background 102 is dim because of the distortion of a projection lens and others and blue in the background 102 and color except blue in the performer 101 are mixed, 0&lt;.alpha.&lt;1 in picture elements corresponding to the boundary between the performer 101 and the background 102 in video data VDa.
As described above, when two-dimensional video data VDc in a case viewed from the viewpoint of the TV camera 104 at time T1 is output from the CG data generator 108, video data VDa obtained by photographing with the TV camera 104 at the time T1 is output from the delay unit 109 as video data VDb, however, similarly time is adjusted so that when two-dimensional video data VDc in a case viewed from the viewpoint of the TV camera 104 at the time T1 is output from the CG data generator 108, the value of a acquired based upon video data VDa obtained by photographing with the TV camera 104 at the time T1 is output from the key generator 111.
In the .alpha. blender 110, operation in the following expression (1) is applied to two-dimensional video data VDc generated in the CG data generator 108 and video data VDb output from the delay unit 109 so as to acquire synthetic video data VDd. EQU VDd=.alpha..times.VDb+(1-.alpha.).times.VDc (1)
As .alpha.=0 in picture elements corresponding to the background 102 in video data VDb, two-dimensional video data VDc is output from the .alpha. blender 110 as it is as synthetic video data VDd. As .alpha.=1 in picture elements corresponding to the performer 101 in video data VDb, video data VDb is output from the .alpha. blender 110 as it is as synthetic video data VDd. Further, 0&lt;.alpha.&lt;1 in picture elements corresponding to a boundary between the background 102 and the performer 101 in video data VDb and synthetic video data VDd in which video data VDb and VDc are mixed is output from the .alpha. blender 110. As described above, as synthetic video data VDd in which video data VDb and VDc are mixed is output to picture elements corresponding to the boundary between the background 102 and the performer 101 in video data VDb, a sense of incompatibility on the boundary is removed.
The picture generator 103 is also provided with an output terminal 112 for outputting synthetic video data VDd as data for a synthetic picture to be provided to viewers and a monitor 113 to which the synthetic video data VDd is supplied for displaying a picture based upon the synthetic video data VDd as a picture for assisting the performance of the performer 101.
The picture generator 103 is also provided with a control unit 114 for supplying a reading request signal RRQ to the CG information storage 107 according to an operation by a director. A user interface is normally provided to the control unit 114. That is, video data VDa output from the TV camera 104 is supplied, a picture based upon the video data VDa is displayed on the monitor not shown and information photographed by the TV camera 104 is provided to the director who manages the whole virtual studio. The director selects a suitable picture generated by CG in accordance with the current progress of a weather forecast program for example and inputs the information to the input device not shown of the control unit 114. In the meantime, the control unit 114 supplies the above information to the CG information storage 107 as a reading request signal RRQ.
For example, if a director judges that it is time for a weather forecast in relation to Japan to be broadcast, he/she instructs the control unit 114 to provide "Japanese map including weather information". The control unit 114 supplies the above information to the CG information storage 107 as a reading request signal RRQ. Hereby, the three-dimensional positional information PCG of "Japanese map including weather information" generated by CG is read from the CG information storage 107 and is supplied to the CG data generator 108.
Next, the operation of the picture generator 103 will be described. The performer 101 is photographed in front of the background 102 with the TV camera 104. Video data VDa is acquired from the TV camera 104, is delayed by the delay unit 109 by time corresponding to the processing time of the camera parameter calculator 106 and the CG data generator 108 and is supplied to the a blender 110 as video data VDb.
The three-dimensional positional information PDT of markers M1 to M9 on the basis of the TV camera 104 is output from the TV camera 104 and is supplied to the camera parameter calculator 106. In the meantime, the three-dimensional positional information PDW of the markers M1 to M9 on the basis of the world coordinates is read from the markers' positional information storage 105 and is supplied to the camera parameter calculator 106. In the camera parameter calculator 106, the three-dimensional position and the orientation of the TV camera 104 on the basis of the world coordinates are calculated based upon the three-dimensional positional information PTD and PDW of the markers M1 to M9.
The information of the three-dimensional position and the orientation of the TV camera 104 acquired in the camera parameter calculator 106 is supplied to the CG data generator 108 as the viewpoint information POV of the TV camera 104. In the meantime, the three-dimensional positional information PCG of a predetermined object generated by CG is read from the CG information storage 107 in response to a reading request signal RRQ supplied from the control unit 114 and is supplied to the CG data generator 108. In the CG data generator 108, two-dimensional video data VDc corresponding to the projected picture of the predetermined object generated by CG to which perspective transformation viewed form the viewpoint of the TV camera 104 is applied is generated based upon the viewpoint information POV of the TV camera 104 and the three-dimensional positional information PCG of the predetermined object generated by CG. For example, the two-dimensional video data VDc of "Japanese map including weather information" which looks as if it were viewed from the viewpoint of the TV camera 104 is generated. The two-dimensional video data VDc generated in the CG data generator 108 is supplied to the .alpha. blender 110.
Video data VDa output from the TV camera 104 is supplied to the key generator 111 and on the key generator 111, the value of .alpha. as a key signal for achieving a so-called soft key is generated based upon the video data VDa. The value of .alpha. is supplied to the .alpha. blender 110. In the .alpha. blender 110, operation in the expression (1) is applied to two-dimensional video data VDc output from the CG data generator 108 and video data VDd output from the delay unit 109 and synthetic video data VDb for displaying a synthetic picture obtained by synthesizing the projected picture or a predetermined object generated by CG and the projected picture of the performer 101 is acquired.
Synthetic video data VDd output from the .alpha. blender 110 is output to the output terminal 112 as the data of a synthetic picture to be provided to viewers. The synthetic video data VDd is also supplied to the monitor 113 and a picture based upon the synthetic video data VDd is displayed on the screen of the monitor 113 as a picture for assisting the performance of the performer 101.
Referring to FIGS. 4A to 4D, it will be described what picture is actually generated in each processing of the picture generator 103.
FIG. 4A shows a picture based upon video data VDa output from the TV camera 104 at certain time. Picture elements corresponding to the background 102 are blue. Picture elements corresponding to the performer 101 are not blue. Further, picture elements on a boundary between the performer 101 and the background 102 have color obtained by mixing blue and color except blue. The same reference number is allocated to each picture corresponding to the performer 101, the background 102 and the markers M1 to M9. The following each picture is also similar.
FIG. 4B shows the value of .alpha. output from the key generator 111. ".alpha.=0" in picture elements corresponding to the background 102. ".alpha.=1" in picture elements corresponding to the performer 101. Further, 0&lt;.alpha.&lt;1 in picture elements on a boundary between the performer 101 and the background 102.
FIG. 4C shows a picture based upon two-dimensional video data VDc generated in the CG data generator 108. FIG. 4C shows a case that the three-dimensional positional information PCG of "Japanese map including weather information" generated by CG is read from the CG information storage 107.
FIG. 4D shows a picture based upon synthetic video data VDd output from the .alpha. blender 110. That is, FIG. 4D shows a picture in which the picture shown in FIG. 4A and the picture shown in FIG. 4C are mixed every picture element depending upon the value of .alpha. of each picture element.
Further, a case that the TV camera 104 is moved at the following time will be considered. In this case, even if the performer 101 stands still, a picture corresponding to the performer 101 in a picture based upon video data VDa output from the TV camera 104 is moved on the screen. In the camera parameter calculator 106, as the three-dimensional position and the orientation of the TV camera 104 on the basis of the world coordinates are calculated and the information of the three-dimensional position and the orientation is supplied to the CG data generator 108 as the viewpoint information POV of the TV camera 104, two-dimensional video data VDc output from the CG data generator 108 varies even if the three-dimensional positional information PCG of an object generated by CG read from the CG information storage 107 is similar. Because a projected picture also varies if a viewed position varies in perspective transformation.
The quantity in which a picture based upon video data VDc to which perspective transformation is applied at certain time and at the next time varies is equal to the quantity in which a picture varies by the movement of the TV camera 104. Therefore, the quantity in which a picture based upon video data VDc to which perspective transformation is applied at certain time and at the next time varies corresponds to the quantity in which the projected picture of the performer 101 photographed with the TV camera 104 varies. Therefore, in a picture based upon synthetic video data VDd finally acquired, the projected picture of the performer 101 and the projected picture of an object generated by CG move similarly. That is, a picture based upon synthetic video data VDd finally acquired looks as if the above picture were a picture obtained by photographing an object generated by CG, for example Japanese map including weather information and the performer 101 respectively existing in the same space.
In the above picture generator 103, a picture based upon synthetic video data VDd, that is, a picture obtained by synthesizing the projected picture of the performer 101 and the projected picture of the object generated by CG is displayed on the monitor 113. The performer 101 can grasp in what positional relationship he/she and the object generated by CG are by performing, watching the picture. Hereby, the performer 101 can also point to a specific part in "Japanese map including weather information" generated by CG for example.
As described above, when it takes t seconds for processing in the camera parameter calculator 106 and the CG data generator 108, delay time by the delay unit 109 is t seconds and time is adjusted. Therefore, a picture displayed on the monitor 113 is a picture obtained by photographing an event at time prior by t seconds. Concretely, if the performer 101 raises his/her hand, the projected picture of the performer 101 displayed on the monitor 113 raises his/her hand after t seconds. As described above, there is a problem that it is difficult for the performer 101 to perform, watching his/her motion before t seconds on the monitor 113.
The object of the present invention is to remove the delay of a picture for assisting the performance of a performer so that the performer can readily perform.