Conventionally, in image sensing apparatuses including a video camera, automation and multiple functions have been developed as automatic exposure (AE), automatic focus (AF) and the like, thus excellent image sensing can be easily performed.
Further, in recent years, with downsizing of image sensing apparatuses and development of high-powered optical system, shake of an image sensing apparatus (camera shake) or the like becomes a major factor of degradation of image quality. Accordingly, various blur correction functions to correct blur of an image caused by apparatus, shake or the like have been proposed. In a case where such blur correction function is incorporated in an image sensing apparatus, improved image sensing can be performed.
As an example of blur correction function, optical blur correction has been proposed. First, shake of an image sensing apparatus is detected as an angular velocity by an angular velocity sensor such as a gyroscope attached to an image sensing unit, then a direct current (DC) component of an angular velocity signal obtained by the angular velocity sensor is cut by a direct current cut filter which passes only an alternating current component, i.e. vibration component, and the angular velocity signal which passed the filter is amplified by an amplifier to an appropriate level. Thereafter, the angular velocity signal is subjected to integration processing by an integrator thereby converted to an angular displacement signal, and an optical shake correction unit is driven so as to suppress blur, based on the angular displacement signal obtained from the integrator (for example, Japanese Patent Application Laid-Open No. 9-181959).
The optical blur correction unit is an optical correction mechanism to displace an optical axis to cancel out the camera shake. For example, a method of displacing an optical axis of light incident on an image sensing surface by displacing a blur correction lens on a plane orthogonal to the optical axis has been proposed.
In this manner, by optically canceling out the shake of the image sensing unit, optical blur correction is always performed during image sensing (during exposure), thus an image without blur can be obtained.
On the other hand, a blur correction system which performs blur correction by electrical process has been proposed. As an example, known is a method of performing image sensing with short-time exposure, short enough not to be affected by camera shake, a plurality of times in a short period, then combining (superposing) a plurality of images obtained by the image sensing while correcting shifts of the images, thus obtaining an image (combined image) with appropriate exposure time (for example, Japanese Patent No. 3110797).
Recent image sensing apparatuses such as digital cameras and video cameras are being downsized such that they can be incorporated in hand-held electronic devices (for example, a cellular phone). In this situation, if the above-described optical blur correction system is to be incorporated in a camera, the blur correction optical device must be further downsized or the vibration detection unit must be downsized.
However, in the optical blur correction system, as it is necessary to support the correction lens and drive it with high accuracy, downsizing is limited. Further, as most of currently-used vibration detection units utilize an inertial force, if such vibration detection unit is downsized, the detection sensitivity is degraded and shake correction cannot be performed with high accuracy.
Further, as shake applied to an image sensing apparatus, angular shake about a predetermined axis and shift shake to horizontally shake the image sensing apparatus are known. The angular shake can be corrected by the optical shake prevention system, while the shift shake cannot be prevented without difficulty. Particularly, the shift shake is increased as the camera is downsized.
On the other hand, the above-described blur correction system using electrical coordinate conversion simply has a construction to perform image sensing a plurality of times within a predetermined period. More particularly, as long as a shutter having a higher speed than that of conventional shutters is continuously driven plural times, the mechanism can be easily downsized in comparison with the optical blur correction system. Further, regarding the above-described shift shake, as a plurality of images obtained by image sensing are combined (superposed) into an image (combined image) with appropriate exposure time while shift of images is corrected, shift shake can be corrected. Accordingly, regarding downsizing of an image sensing apparatus, the blur correction system using electrical coordinate conversion is advantageous by virtue of its characteristic.
Next, a conventional image sensing apparatus capable of blur correction using the above-described electrical coordinate conversion will be described.
FIG. 10 is a block diagram showing the construction of a conventional image sensing apparatus which performs blur correction using electrical coordinate conversion.
In FIG. 10, reference numeral 150 denotes a zoom lens; 210, a zoom driving motor; and 220, a zoom controller. When a user operates a zoom operation unit 202 so as to change a view angle, an operation signal is inputted into an image sensing controller 200. The image sensing controller 200 generates a zoom control signal based on the operation signal and sends the signal to the zoom controller 220. The zoom controller 220 converts the zoom control signal into an electrical driving signal to drive the zoom driving motor 210, and drives the zoom driving motor 210 to move the zoom lens 150. Then the zoom lens 150 is stopped in a desired position, thereby the image sensing view angle is changed.
Further, numeral 151 denotes an aperture; 211, an aperture driving motor; 221, an aperture controller; 152, a focus lens; 212, a focus driving motor to directly drive the focus lens 152; and 222, a focus controller. Control operations of the aperture 151 and the focus lens 152 will be described later.
Further, numeral 153 denotes a shutter; 213, a shutter driving actuator; and 223, a shutter controller. The shutter 153 is provided to limit light flux incident on an image sensing device 161 when reading of image signal from the image sensing device 161 is performed after exposure of the image sensing device 161 in correspondence with the user's operation of a release operation unit 201 to be described later. The shutter 153 has a plurality of shutter blades. The shutter blades receive a driving force from the shutter driving actuator 213, to open/close an opening as a light passing port, thereby the light flux incident on the image sensing device 161 is controlled. Further, the release operation unit 201 has a two-stage switch structure where a switch SW1 is turned ON upon half stroke of the release operation unit, and a switch SW2 is turned ON upon full stroke of the release operation unit via the half stroke.
The image sensing device 161 generates electric charge corresponding to the amount of incident light and outputs an image signal corresponding to the charge. The image sensing device 161 comprises, e.g., a semiconductor image sensing device such as a MOS or CCD, however, is not limited to such device.
The light flux (image sensing light) incident via the zoom lens 150 is light-amount limited with the aperture 151, passed through the focus lens 152 and the shutter 153, forms an image on the image sensing device 161, and converted to electric charge and accumulated. The electric charge accumulated in the image sensing device 161 (image signal) is read at predetermined timing, then converted to a digital signal by e.g. an A/D converter (not shown), and inputted into a camera signal processor 162. The camera signal processor 162 performs predetermined signal processing on the input digital image signal to e.g. form luminance signal and color signals, thus forms a color image signal.
The image sensing controller 200, comprising e.g. a microcomputer, performs automatic focus (AF) control, automatic exposure (AE) control, zoom lens control, shutter control and the like, and inputs operation signals from the release operation unit 201, the zoom operation unit 202 and a blur correction operation unit 203. When operation signals have been inputted from the release operation unit 201 and the zoom operation unit 202, the image sensing controller 200 sends control signals based on the input operation signals to the shutter controller 223 and the zoom controller 220 in accordance with an image sensing status of the camera, to set image sensing conditions, such that image sensing is performed.
Further, the image sensing controller 200 obtains an average level of luminance signal obtained from the camera signal processor 162, i.e., subject luminance or the like. The image sensing controller 200 performs predetermined arithmetic processing using the obtained subject luminance, generates an aperture control signal, and sends the signal to the aperture controller 221. The aperture controller 221 converts the aperture control signal into an electrical driving signal to drive the aperture driving motor 211, and opens/closes the plural blades of the aperture 151 by driving the aperture driving motor 211, thereby changes the opening area (aperture diameter) as a light passing port. In this manner, the opening diameter of the aperture 151 is changed until the average level of luminance signal finally becomes equal to a predetermined reference value and stopped, thereby AE control is realized.
On the other hand, the image sensing controller 200 performs predetermined arithmetic processing based on a frequency component or the like included in the luminance signal obtained from the camera signal processor 162, and generates a focus control signal. The focus control signal is sent to the focus controller 222, and converted to an electrical driving signal to drive the focus driving motor 212. The focus driving motor 212 moves the focus lens 152 in an optical axis direction in correspondence with the driving signal. In this manner, focusing operation is performed by moving the focus lens 152 to a position where the predetermined frequency component included in the luminance signal finally has a maximum value (focused position), and stopping the lens 152, thereby AF control is realized.
Numeral 165 denotes a signal switching unit which is switched in correspondence with an operation of the blur correction operation unit 203. When the blur correction operation unit 203 is turned ON, the signal switching unit 165 is connected to the side of an image memory 171, while when the blur correction operation unit 203 is turned OFF, the signal switching unit 165 is connected to the side of an image processor 175. Numeral 172 denotes a shift detector; 173, a coordinate converter; and 174, an image combiner. These circuits are used in blur correction. Numeral 176 denotes a recorder; and 180, a monitor.
Next, the operation of the image sensing apparatus having the above construction will be described with reference to a flowchart of FIG. 11.
At step S101, the operation is started. For example, the operation is repeatedly started at predetermined timing such as a vertical synchronizing period of a moving image.
When the operation starts, it is determined at step S102 whether or not the switch SW1 is turned ON by the user's half stroke of the release operation unit 201. If it is determined that the switch SW1 is turned ON, the process proceeds to step S103, while if the half stroke operation is not performed, the current process ends.
At step S103, as described above, the image sensing controller 200 outputs an aperture control signal based on an image signal obtained from the camera signal processor 162, to control the aperture 151 to have an appropriate opening area via the aperture controller 221 and the aperture driving motor 211, thereby AE control is performed.
When the AE control has been completed, the process proceeds to step S104, at which, as described above, the image sensing controller 200 outputs a focus control signal based on the image signal obtained from the camera signal processor 162, to move the focus lens 152 to a focusing position via the focus controller 222 and the focus driving motor 212, thereby AF control is performed.
Next, at step S105, it is determined whether or not the user has turned the blur correction operation unit 203 ON. If it is determined that the blur correction operation unit 203 is ON, the process proceeds to step S106, at which a blur correction operation is started, while if it is determined that the blur correction operation unit 203 is OFF, proceeds to step S108.
As an example where the blur correction operation unit 203 is ON, the subject of image sensing is dark and sufficient exposure cannot be performed within a short period. The timing of closing the shutter 153 (exposure time) and the aperture diameter of the aperture 151 are set based on a photometry value obtained by the AE operation at step S103. Generally, in an image sensing condition where a blur correction system is used, as the subject is dark, the aperture is fully opened and the exposure time is long. In this case, as the influence of camera shake of the image sensing apparatus, i.e., the shake of image on an image sensing surface cannot be negligible, the blur correction operation unit 203 is turned ON and the following operation is performed.
First, at step S106, the number of images to be taken and respective exposure time are determined from the image sensing conditions such as brightness of the subject obtained at step S103. Note that the image sensing conditions are:    Brightness of the subject    Focal length of image sensing optical system    Brightness of image sensing optical system (aperture value)    Sensitivity of image sensing device
For example, assume that the sensitivity of the image sensing device 161 is ISO 200, the brightness of the subject is measured (photometry) and based on the result of the photometry, the aperture 151 is set to f2.8 for appropriate exposure, and the opening timing of the shutter 153, i.e., the exposure time is set to ⅛ sec. If the focal length of the image sensing optical system is 30 mm for a 35 mm film, there is a possibility of image blur due to camera shake in image sensing with the exposure time of ⅛ sec. To prevent the effect of camera shake, the exposure time is set to 1/32 sec and the number of times to perform image sensing operation is set to 4, thereby a total exposure time of ⅛ sec can be obtained.
As another example, if the focal length is 300 mm, the exposure time is set to 1/320 sec and the number of times to perform image sensing operation is set to 40 ( 1/320 sec×40 times=⅛ sec).
In this case, as each image obtained by short exposure time becomes an underexposure image, however, the influence of camera shake is reduced.
In this manner, when the image sensing operation is repeated a plurality of number of times, the exposure time in each image sensing operation is set in correspondence with the image sensing conditions, and further, the number of images to be taken (how many times image sensing operation is to be performed) is set in correspondence with the image sensing conditions.
At step S107, when the switch SW2 becomes ON by the user's full stroke of the release operation unit 201, the process proceeds to step S112, while if the full stroke of the release operation unit 201 is not performed, the process ends.
At step S112, the shutter 153 is released to achieve the exposure time obtained at step S106, and electric charge is read from the image sensing device 161. The read electric charge is A/D converted to a digital signal, then is subjected to predetermined signal processing by the camera signal processor 162. The signal-processed image is recorded via the signal switching unit 165 into the image memory 171. At step S113, it is determined whether or not the number of images obtained by image sensing is equal to the number of images set at step S106. If the necessary number of images have been obtained, the process proceeds to step S114, while if the necessary number of images have not been obtained, the process returned to step S112 to repeat the processing from the image sensing to the recording in the image memory 171.
Even though the influence of camera shake is not caused in each of the images obtained by the plurality of image sensing operation, the composition among the images may be slightly shifted due to camera shake during the consecutive image sensing operation. Accordingly, if these images are simply combined without any processing, the combined image becomes a blurred image due to the shift of composition among the images. Therefore, the shift among the images must be corrected.
For this purpose, at step S114, successive two images are read out of the images stored in the image memory 171, and the shift detector 172 extracts a characteristic feature image portion (feature point), and obtains the coordinates of the feature point in the image. More specifically, an image and its subsequent image are compared with each other, then a feature point is extracted and its coordinates are obtained.
Next, at step S115, the coordinates of each image stored in the image memory 171 are converted by the coordinate converter 173. More specifically, each image is shifted by a difference in coordinate values such that the feature point obtained at step S114 has the same coordinate values in the respective images.
The feature point extraction, the coordinate position calculation, and the coordinate conversion will be described in detail later.
At step S116, it is determined whether or not the coordinate conversion has been completed for all the images stored in the image memory 171. The processing at steps S114 and S115 is repeated until all the images have been processed, and when all the images have been processed, the process proceeds to step S117.
At step S117, all the images, of which the coordinates of the feature point have become the same by coordinate conversion at step S115, are combined by the image combiner 174. The image combining may be performed by, e.g., adding image sensing signals of corresponding coordinates in the respective images. In this manner, as the plurality of images are combined to one image, underexposure state of each image can be improved.
The combined image is considered to have an area, where all the images are not superposed due to composition shift, in a peripheral portion of the combined image. Accordingly, at step S117, at the same time of the image combining, the area where all the images are not superposed is cut and a rectangular image is obtained. This is because as a result of combining processing, in such area where all the images are not superposed, the amount of added image signals is insufficient and is dark. The above cutting processing will be described later.
Next, at step S118, the image processor 175 performs predetermined image processing such as gamma correction and compression on the image signal of the obtained combined image.
At step S119, the processed combined image data is recorded in the recorder 176 such as a hard disk recorder or a semiconductor memory. Further, at step S120, the image data recorded at step S119 is displayed on the monitor 180 such as a liquid crystal monitor.
Note that the method of combining images obtained by image sensing while correcting shifts of the images to obtain an image (combined image) with an expanded dynamic range has been conventionally disclosed (for example, Japanese Patent NO. 3110797).
Next, the processing in a case where it is determined at step S105 that the blur correction operation unit 203 is OFF will be described.
At step S108, the exposure time to be used in image sensing operation is determined from the image sensing conditions such as the brightness of the subject or the like obtained at step S103. The way of obtaining the exposure time is similar to that performed at step S106, however, as the blur correction processing is not performed here, the exposure time is obtained on the premise that image sensing is performed once.
At step S109, when the switch SW2 is turned ON by the user's full stroke of the release operation unit 201, the process proceeds to step S111, while if the full stroke of the release operation unit 201 has not been performed, the process ends.
At step S111, the shutter 153 is released to achieve the exposure time determined at step S108, and an image is obtained. The image is subjected to predetermined signal processing by the camera signal processor 162, then inputted via the signal switching unit 165 into the image processor 175, and the above-described processing such as gamma correction and compression are performed at step S118. As the subsequent operations are the same as those described above, the explanations thereof will be omitted. Note that the image obtained at step S111 may be temporarily stored in an internal memory (not shown) then sent to the image processor 175. Further, the image memory 171 may be employed as this internal memory.
Next, the feature point extraction, the coordinate calculation and the coordinate conversion performed by the shift detector 172 at step S114 will be described.
As shown in FIG. 12, for example, a picture where a person 411a stands against the background of a building 412a and a tree 413a in a frame 401a is considered. When the same subject is sensed a plurality of times, a frame 401b with a composition shifted from that of the frame 401a may be obtained due to camera shake. Note that the frame with shifted composition is denoted by 401b for convenience of explanation, however, an actually image-sensed area is the frame 401a. 
The shift detector 172 extracts an edge of a window 421a, which is a high brightness point in the building 412a in the image, as a feature point 422a, by edge detection, then compares the feature point 422a with a feature point 422b in the image frame 401b, and corrects the difference therebetween (coordinate conversion).
In FIG. 12, as indicated with an arrow 423, the frame 401b is coordinate-converted such that the feature point 422b in the frame 401b is superposed on the feature point 422a in the frame 401a. 
In this example, the number of feature points is one for convenience of explanation, however, actually, plural feature points exist within one image. It may be arranged such that based on these information, the amount of coordinate movement is calculated by averaging shift amounts of the feature points and the coordinate conversion is performed.
Further, in the above description, the coordinate conversion is performed between two frames, however, the images are obtained in correspondence with the number of images to be taken determined at step S106. Accordingly, regarding more than two frames, shift correction of all the images can be performed by repeating the coordinate conversion as described above.
The feature point extraction, the coordinate position calculation and the coordinate conversion are performed as described above.
Finally, the cutting processing of an area where all the images are not superposed due to composition shift, performed at step S117, will be described.
FIGS. 13A to 13C show a case where two images with a composition shift as shown in FIG. 12 are combined.
If coordinate conversion is performed on the two images 402a and 402b as shown in FIGS. 13A and 13B and image combining is performed, an image 403 as shown in FIG. 13C is generated. However, the image 403 has an area 404 where the images are not superposed.
As described above, in an image portion where all the images are not superposed, as the level of added image signals is insufficient and the combined image is dark, only an area where all the images are superposed is handled as a combined image, and the area 404 is removed from the image.
However, in the above-described blur correction system using electrical coordinate conversion, the view angle of the combined image becomes smaller than that of images upon image sensing because of the combining method, and the user's intended view angle cannot be obtained.
This problem occurs since an area where the images are not superposed is cut after the coordinate conversion by the image processor 175.