1. Field of the Invention
The present invention relates to image processing devices and image processing methods. More particularly, the present invention relates to image processing devices and image processing methods allowing correction of image distortion produced by relative misalignment of an original and a camera at the time of image pick-up.
2. Description of the Related Art
Recently, digital cameras have become quite popular. The digital cameras enable the image pick-up and storage in the form of digital data of originals on which photographs or characters are printed. In digital cameras, usually the picked-up images are first subjected to an image processing for removal of noise from the image and then to a compression processing for reducing the data amount. A different image processing and the compression processing are applied for each picked-up image.
An original paper sheet usually includes a character area, a line-drawn area, a photograph area and so on. Here, the character area represents a rectangular area including only a character in black. The line-drawn area represents a rectangular area mainly constituted of a monochrome area and an edge area, such as a bar graph, an image daubed with one color or the like. The photograph area represents a rectangular area mostly consisted of an image depicted by a variation of tones. The photograph area includes pictures, illustrations, textures, for example.
In the image processing, for example, the correction process or processing process are usually performed for the photograph areas. A focus of the processes is on the tone rather than on the resolution. Conversely, for the character areas, the focus of the processes is usually on the resolution rather than on the tone. With regard to the compression processing, use of compression processing defined by JPEG (Joint Photographic Coding Experts Group) or PNG (Portable Network Graphics) utilizing the concentration of frequency is effective for the photograph areas in view of image quality and compression ratio because high frequency components are not important for the photograph areas. On the other hand, the processing such as predictive coding and dictionary type coding utilizing the concentration of tones is effective for the character areas because pixel values of the character areas are mostly black or white. Hence, the selective application of optimal image processing and compression processing for each area is desirable in view of image quality and compression ratio when areas of different types are included in one image.
Therefore, an area determination process for extracting a rectangular area is conventionally performed. A picked-up image is divided into a plurality of blocks and an attribute indicating which area type the block belongs to is found for each block and then the blocks with the same attribute are enclosed by horizontal lines and vertical lines. Next, the area determination process will be described. FIG. 37 is an illustration of an image obtained by image pick-up of an original by a digital camera. An image 240 includes photograph areas 242 and 246 including an image of photographs, a line-drawn area 244 including an image of a graph and a character area 248 including images of characters. FIG. 38 is an illustration of an image after the area determination processing of the image shown in FIG. 37. With reference to FIGS. 37 and 38, photograph areas 242 and 246 in image 240 are extracted as photograph areas 252 and 256 in an image 250, respectively, through the area determination process. In addition, line-drawn area 244 in image 240 is extracted as a line-drawn area 254 in image 250. Similarly, character area 248 in image 240 is extracted as a character area 258 in image 250.
To extract an area depending on each attribute in the area determination processing, a rectangle enclosed by two vertical lines and two horizontal lines are used as a unit of extraction. This is because most of the conventional compression scheme according to JPEG, PNG or the like are performed based on the rectangular unit, and areas included in an original in which photographs and characters are shown are mostly rectangular.
Now, when an image of an original representing information such as characters and photographs is to be picked up by a digital camera, for example, the picked-up image becomes distorted due to a swing of a lens of the digital camera. Even when the actual shape of the original is rectangular, the image picked up by the digital camera will be in a form of a trapezoid.
As a technique for correcting the distortion of an image attributable to a swing at the image pick-up in an image obtained by the digital camera, U.S. Pat. No. 5,528,194 is known, for example. In U.S. Pat. No. 5,528,194, a technique is described for correcting a swing by deforming a trapezoidal object, which is originally a rectangle, in an image, into a rectangle.
In addition, a technique for correcting a distortion of an image projected on to a screen according to an image projecting device projecting an image on a screen is disclosed in Japanese Patent Laying-Open No. 10-200836. The image projecting device described in Japanese Patent Laying-Open No. 10-200836 projects a reference pattern on a screen, measures a distortion of the projected reference pattern, finds a relation between the location of the screen and the projecting device, and corrects the distortion of an image caused by the swing based on this relation.
(1) First Problem
When a digital camera picks up an image of an original, the camera sometimes becomes tilted with respect to the original during the pick-up. The image of the original thus picked up, therefore, is rotationally shifted. When the area determination process described above is performed on such image obtained through the pick-up, areas cannot be extracted appropriately according to the attribute. This problem will be described in detail below.
FIG. 39 shows an image obtained through the image pick-up of an original by a digital camera and the original shown in the image is rotationally shifted. An image 260 shown is an image obtained by rotating image 240 shown in FIG. 37 by a given angle. Image 260 includes photograph areas 262 and 266 including images of photographs, a line-drawn area 264 including a image of a graph and a character area 268 including images of characters. Each of areas 262, 264, 266 and 268 are tilted by the same angle as image 260. FIG. 40 shows an image obtained by performing the area determination processing on the image of FIG. 39. With reference to FIGS. 39 and 40, photograph areas 262 and 266 in image 260 are extracted as photograph areas 272 and 276 in an image 270, respectively. Line-drawn area 264 in image 260 is extracted as a line-drawn area 274 in image 270. Similarly, character area 268 in image 260 is extracted as a character area 278 in image 270. Here, comparing character area 278 in image 270 with character area 268 in image 260, it is found that character area 278 includes parts of photograph areas 262 and 266 in addition to character area 268. Similarly, areas 272, 274 and 276 in image 270 include parts that should be included in other areas.
When an area extracted through the area determination processing includes a part which should be included in a different area, an optimal image processing or compression processing cannot be performed, resulting in the degradation of image quality, compression ratio and so on.
To solve this problem, it is possible to detect an edge of the original in the image obtained through image pick-up of the original, and to determine the inclination of the original based on the inclination of the detected edge of the original. The image obtained through image pick-up of the original by the digital camera, however, does not necessarily include an edge of the original. Even if the edge of the original is included in the obtained image, when the image of the original is picked up together with the image of a desk, for example, on which the original resides, the pattern on the desk may be shown in the background and makes it difficult to distinguish the edge of the original from the pattern of the desk.
FIG. 41 shows an image, which is obtained through the image pick-up of an original by a digital camera and does not include an edge of the original. As an image 280 does not include the edge of the original, the rotational shift of the original in the image cannot be found through the detection of inclination of the edge of the original. FIG. 42 shows an image, which is obtained through the image pick-up of an original by a digital camera and includes an image of a desk as a background of the image of the original. An image 285 is shown as a picked-up image of an original 290 and a desk 292 on which the original resides. Desk 292 has a pattern 296 thereon. Though the determination of an edge 294 of desk 292, pattern 296 of desk 292 thereon and an edge 298 of original 290 as edges is possible through the detection of edges from image 285, it is difficult to detect which corresponds to the edge of original 290.
(2) Second Problem
According to the technique described in U.S. Pat. No. 5,528,194, an entire object (original) must be included in an input image. However, depending on the zoom magnification and the position of pick-up, the entire object may not be included in the picked-up image. Specifically, when the image of the original is picked up by a digital camera, if the zoom magnification is high or the distance between the digital camera and the original is short, it is impossible to place the original as a whole into the scope of the camera allowing the pick up.
FIG. 43 shows an example of an original that is an object of an image pick-up by a digital camera. FIG. 44 shows an example of an image obtainable by the image pick-up of the original shown in FIG. 43 by the digital camera and does not include an edge of the original. FIG. 45 shows an example of an image obtainable by the image pick-up of the original shown in FIG. 43 by the digital camera and includes a portion around an edge of the original. As an overall outline of the original is not included in the image in FIGS. 44 and 45, the identification of the shape of the original is impossible.
Alternatively, a rectangular area included in the original can be chosen as an object to be processed. The rectangular area, however, is not always included in the original and even if the rectangular area is included in the original, it is not always included in the image as described above. Therefore, when the object is not entirely included in the image, an appropriate correction cannot be performed according to U.S. Pat. No. 5,528,194.
Additionally, though in the technique described in Japanese Patent Laying-Open 10-200836, the reference pattern is projected onto the screen, it is not easy to measure the distortion of the reference pattern projected on the original. This is because that the condition of image pick-up is different from time to time and the image pick-up is not always performed under the sufficient light and that it is difficult to distinguish a design pattern on the original from the reference pattern as the design pattern is shown on the original in various forms such as characters, photograph, and so on.