People love to observe pictures and images displayed all around them. The bigger, the brighter, the higher resolution, and the more inexpensive these displays are the better. The future promises digital cinema and immersive virtual reality systems that provide a realistic visual experience. While the computational power necessary to make these systems a reality is available, high-quality large-scale displays are not.
Typically, computer-generated images are stored as bitmaps in memory. A picture element, or pixel, is represented by three bytes in the bitmap representing color intensities for the red, green and blue values of the pixel. To display the image, the bitmap is stored in a special, high-speed memory called a frame buffer which is accessed by the display hardware. The display hardware may then, from the image stored in the frame buffer, create a composite analog video signal, or separate red, green and blue (RGB) video signals, or whatever type of signal is appropriate for the display.
Current display technology is largely based on the cathode ray tube, the last remaining hot cathode vacuum electronic element in modem computer systems. Resolutions on displays built using this technology are limited by the required precision analog electronics and the achievable bandwidths of high energy video circuitry to 1200xc3x971600 pixels for off the shelf displays, and to 4000xc3x974000 pixels for research prototypes.
Liquid crystal displays are gradually replacing CRTs in portable applications with dramatic improvements in power, but the resolution remains limited to the 1600xc3x971200 pixel range for even high end prototypes.
High performance projection displays exhibit similar resolution limitations, partly due to the large fill factor of the pixel elements required for high optical efficiency. Even the micro-mirror technologies based on micromechanical deflection systems appear (perhaps for market reasons) to be limited to relatively low resolution.
Various aspects of a projection system, such as the optics, defects in the projection screen, the projection angle relative to the screen, etc. can introduce various types of distortion into a displayed image. These include pincushion and barrel distortion which appear as the bowing inward and outward respectively of square objects. Keystoning and trapezoidal distortion results when the projection angle is not perpendicular to the screen angle. Lateral and rotational displacement of the image is directly due to similar displacement of the projector.
Throughout the history of display design, the focus has been on constructing ever larger, higher brightness single display elements. The optical quality, alignment, precision, and perfection in manufacturing these displays has progressed enormously over the past 25 years, but we are still faced with the essential limitation of low resolution of the display array.
A key technological trend of the past 20 years has been the dramatic reduction in the price of computation. The present invention exploits this cost reduction by replacing high precision optical and mechanical assembly with increased computation. As the price of computation continues to drop, this tradeoff increasingly favors computation over precision assembly.
The problems that need to be overcome to make a scalable technology for seamless large-scale displays are elucidated by discussing the prior work in this field. Three well-known technologies are single projector, discrete tiling, and edge-blended projector arrays.
Single projector technology for large-scale displays uses a single optical system and display source to project a large-scale display. While this approach provides a seamless picture, it is not physically scalable. The resolution of such a system is limited to the resolutions of projectors that are available. The resolutions available are determined by the media formats commonly available, for example, NTSC through HDTV from the television industry, and VGA through XGA from the notebook computer industry. To obtain a larger projected image, the distance of the projected image from the projector is increased. The resulting image is larger, but the resolution is constant and brightness decreases inversely with the increase in area projected. Currently there are three types of systems in common use: cathode ray tube based systems, light-valve based (DMD, LCD, ILA) systems, and laser projection systems, discussed below.
Cathode ray tube technology is well understood. Brightness and contrast ratios for these systems are reasonable, and the resolutions available are sufficient for a 60-inch rear-projection screen. The main disadvantages are that building a larger system requires custom built cathode ray tubes to create the appropriate resolution and brightness. In addition, such systems are subject to phosphor burn-in and drift. The best systems available on the market for wide distant viewing require that the displayed images be dynamic to avoid phosphor burn-in, whereby the phosphor surface is physically damaged by other ions where the electrons are supposed to impinge.
Furthermore, color systems using this technology have three separate guns which produce the additive red, green and blue. These guns must be manually converged; however, the convergence typically drifts out of alignment after a month of use. In summary, these disadvantages present problems with scalability of resolution and brightness, and recurrent maintenance problems for maintaining convergence.
Light-valve based systems modulate light to represent each pixel. Such a system uses a light-valve to reflect or transmit light from a light source, based on the intensity of the pixel represented by a particular valve. The light valves are typically liquid crystal displays (LCD), or digital micro-mirror device arrays (DMD), or image light amplifiers (ILA). Each of these systems suffers commercially in the consumer market because costs are higher than pure CRT-based systems. An advantage of light-valves over CRT systems is that they do not suffer from the analog drift and can be made very bright. In terms of scaling these technologies up for resolution and brightness for larger images, there are still fundamental physical concerns. Custom designing a new light-valve for every situation is not possible. Additionally, since only a single light source is typically involved, one would have to search for other solutions when the limits of single bulb technology are hit, though that is not the current constraint.
Laser projection systems utilize xe2x80x9ca whitexe2x80x9d laser (composed of three separate lasers) and a mechanical system much like those inside laser printers, to write raster lines to form an image. There is no theoretical reason why the same resolutions available for laser printing are not possible for display purposes. Thus, resolution scales quite nicely.
The main disadvantage is the expense of making a laser with the necessary power requirements. For example, one system built by Samsung uses a laser with an output power of 4 watts to achieve the brightness necessary for a particular large display. That power output is split up among three separate lasers. The three lasers are powered to take the empirical difference between a watt and lumen for each frequency into account. Such lasers can be expensive and quite dangerous. For example, the system could malfunction and the laser light remain fixed in space, possibly damaging the display surface or anything else in the way.
This technology does not scale well. To quadruple the size of the image, the laser power input must be quadrupled. A system with 100 times the image area needs 100 times the power from a single set of lasers.
It is clear that scalability in resolution and power requirements are major obstacles that need to be overcome to make single projectors good for projecting images larger than the currently available systems provide. Scalability in resolution is a problem for CRT and light-valve based solutions. Scalability in light generation form obstacles for all three of the systems discussed. Discrete tiling is a technology that addresses these scalability problems much better.
Rather than relying on a single projector, discrete tiling juxtaposes single projection systems to create display systems that scale linearly in both resolution and brightness, but are marred by some visible artifacts. A screen four times larger with four times the resolution and the same brightness, requires four times as many projectors. By using existing projection display technology, one gets all the advantages and disadvantages of each technology coupled with two significant visible drawbacks.
The first drawback is a visible mullion, the black area between each projector""s projected image, which is typically between 8 and 50 mm. Despite the small width (8 mm) of the mullion, it is noticeable. A second drawback is that each of the projectors must be calibrated to uniformly display each color. Each particular projection technology has its own set of problems that must be addressed to address this uniformity problem. For CRT-based displays, all of the projectors must first be converged. Then, each image must be squared off so that it can be squarely joined with adjacent images across the visible boundary. Finally, color variances across the monitors must be adjusted. Due to analog drift, calibration must be performed monthly. A 20xc3x975 CRT-based system such as is used by the NASDAQ stock market site, typically takes sixteen man-days to fully calibrate.
For light-valve systems, a major problem is that light sources degrade differently across projectors over time. Thus, any corrections to the system must be redone at a later date depending on the amount of usage. If any bulb degrades significantly differently from the others all the bulbs in the system need to be replaced to ensure uniformity.
While the above approach provides large scale systems that scale in resolution and brightness, removing the seam is preferable. Today, small scale seamless systems are available. Commercially available manually calibrated edge-blended linear projector arrays are considered state of the art in seamless large-scale display systems.
In such systems, projectors are fed redundant picture information where the projections overlap in space and positioned so that the redundant pixels overlap exactly in space. The redundantly projected pixels are partially projected by each projector in the overlapping region so the projectors"" contributions to a particular pixel""s intensity sum to the pixel""s overall intensity. Otherwise, the resultant image displayed across the projected area will not appear seamless.
The advantage of this approach is that large-scale displays are available that are high resolution, bright and seamless. The disadvantage is that the approach does not scale beyond a few projectors arranged linearly because of the amount of manual calibration necessary. Several issues arise in manually calibrating these systems and are discussed below.
The first issue is with registration accuracy in the overlap regions of multiple projectors, particularly, getting the projected regions to overlap with highly accurate pixel registration. This requires manual geometric adjustments that can be done positionally, optically and electronically to completely overlap the adjacent projectors, removing pincushion, trapezoid, keystone and other optical distortions.
Even after the edge-match and geometric correction, the overlap region is not usually rectangular. This must be taken into account when manually adjusting both the redundant information fed to each projector and the timing for each line of information.
Another problem is color matching between adjacent projectors. If the projectors"" colors do not match and a patch of uniform color is supposed to be displayed across a seam, the visible mismatch will make the seams apparent. Thus, adjacent projectors must be manually adjusted to match across all colors.
Yet another problem is the intensity variation across boundaries. Smoothing function are employed on the data input to each projector to gradually transition the burden of displaying a pixel in the overlap regions between projectors. Care must be taken so that these smoothing function properly deal with the projector""s xe2x80x9cgammaxe2x80x9d correction for each color. Otherwise, the seams appear either brighter than the image or less bright.
When two projectors overlap, edge-blending functions must be created and applied. Each edge-blending function gradually transitions from 100% down to zero. Each projector""s edge-blending function is multiplied with pixel values across each input raster line. The edge-blended functions sum to one. This requires that raster lines of the projectors be properly lined up in space. If everything is aligned correctly, the overlap region will look as though it were generated by one projector. These smoothing functions must be manually calibrated for each horizontal input line for each projector.
Still another problem is the brightness uniformity across the entire viewing area. Without brightness uniformity across the system, the edges of each projector""s projected image become accented and conspicuous, possibly revealing the seams. Manual calibration that takes this into account normally occurs in smoothing functions. Alternatively, the problem has been optically reduced by increasing the f-number of the lens. In single projector systems there is a brightness gradient from the center of the image to the edge typically caused by lens roll-off. The brightness ratio from center to edge can be as high as 1.66 with some projectors. In rear projection, this aspect coupled with limited projection screen diffusion causes a hotspot effect, the apparent location of which depends on the position of the viewer and the angle of view. In any case, the end result is that the field of view for which the system appears seamless may be limited.
Another problem is the exit pupil size. The exit pupil is the envelope from which quality edge-blending can be viewed. The size of this region is determined by how the projected light rays are absorbed, reflected and diffused by the screen surface. If the image is viewed outside this envelope, a distinct brightness mismatch becomes apparent across projection boundaries. This is a result of the screen surface reflecting or transmitting perceivably unequal amounts of light from juxtaposed projectors toward the viewer.
Screen gain quantifies the dispersion of transmitted or reflected light for either a front projection or rear projection screen. A screen with a gain of 1.0 (unity gain), known as a lambertian screen, is one which disperses light uniformly without regard to its incident angle. The higher the screen gain, the more the dispersion of light is affected by the angle of incidence of the source light. On such screens, one can often observe xe2x80x9chot spotsxe2x80x9d of high brightness in multiple projector setups that move as the viewpoint changes.
Finally, the degree of manual calibration needed to set up one of these systems becomes ominous when one considers the corrections required for each projector, which is why this approach is not scalable. Creating a two-dimensional array clearly involves even more manual calibration, so much so that such commercial systems are not readily available.
P. Lyon, xe2x80x9cEdge-Blending Multiple Projection Displays On a Dome Surface To Form Continuous Wide Angle Fields-of-Viewxe2x80x9d, Evans and Sutherland Computer Corp., pp. 203-207, Proceedings, Nov. 19-21, 1985, 7th Interservice Industry Training Equipment Conference, suggests changing the shape of the projectors"" rasters. This, of course, requires additional electronics and precludes the use of existing displays without modification.
However, major technological advances have been made recently in fields ranging from optics to civil engineering, by coupling computation with feedback. As the price of computation continues to decrease, we can enjoy the fruit yielded by technologies that utilize this approach on new problems that require more computational power than available today. The computation-coupled-with-feedback approach can be applied to large-scale displays based on overlapping projection, using commonly available projection hardware, to create a scalable technology which provides the necessary precision and automatic calibration that traditional optical, mechanical and electrical approaches have not been able to provide.
The present invention uses a low-cost technology based on computation coupled-with-feedback to make large-scale displays that provide a realistic visual experience. Automating the calibration phase enables a seamless large-scale display technology which is linearly scalable in resolution and brightness.
Low cost computation coupled with feedback enables the correction of imperfections in the alignment, optical system and fabrication of such very high-resolution display devices as a seamless video wall. More particularly, digital photogrammetry and digital image warping are combined to couple computation with video feedback and projection to provide an entirely new technology that is scalable, bright, high resolution, portable and self-calibrating, and which can create a single seamless image across aggregated overlapping projections. Furthermore this approach enables the creation of display systems which cannot be created by conventional mechanical and optical precision engineering means.
The present invention combines high performance computing with feedback to enable the correction of imperfections in the alignment, optical system and fabrication of very high-resolution display devices. The key idea relies on the measurement of relative alignment, rotation, optical distortion, and intensity gradients of an aggregated set of image display devices using a precision low cost reference. Use of a reference makes it possible to construct a locally correct map relating the coordinate systems of the aggregate display to the coordinate systems of the individual projectors.
The techniques form a new technology for scalable, bright, seamless, high-resolution large-scale self-calibrating displays such as seamless video walls. The technology aggregates low-cost component displays and allows one to linearly scale up resolution or size while maintaining a single level of brightness. None of the three well-known technologies for creating large-scale displays provide results that are as visually satisfactory, portable or scalable. The manual calibration and precision required by those technologies is too great to be scalable. The automatic calibration approach of the present invention is far superior and enables scalability. Low cost computation coupled with feedback is leveraged to provide the necessary precision, and thus correct the distortions digitally.
In accordance with the present invention, a method of displaying images comprises deriving a display map by selectively driving the display while sensing the display output. The image may be generated by, for example, a VCR, laser disc, computer or a combination of these devices. A stored pixel correction function based on the display map is applied to pixel data corresponding to the images to be displayed, and the display is driven from the corrected or modified pixel data. Preferably, the stored pixel correction function comprises an anti-aliasing filter. One or more processors may be used to derive and apply the pixel correction function.
Application of the pixel correction function can be used to correct for many types of distortion, including, but not limited to: imperfections across the display, misalignment of plural projections in a common region; intensity variations across the display; keystone distortion; trapezoidal distortion; pin cushion distortion; barrel distortion; chromatic aberration; color mismatch; and lateral and rotational displacement. In particular, the pixel correction function performs smooth warping of an image.
In a preferred embodiment, the pixel data is stored in a frame buffer, and the pixel correction function is applied to the pixel data between the frame buffer and the display.
In another preferred embodiment, the pixel correction function is applied first, and corrected pixel data is stored in the frame buffer. The display is then driven from the corrected pixel data in the frame buffer.
In yet another preferred embodiment, the display comprises a plurality of projectors. The projectors may be driven from a single frame buffer, or alternatively, each projector may be driven from a separate frame buffer with which it is associated. Furthermore, a separate frame buffer may be provided for each color, e.g., red, green and blue, for each projection region. The pixel correction function corrects for misalignment of projected overlapping pixel arrays, and blends the overlapping projection regions.
In yet another preferred embodiment, the projected images from plural projectors completely overlap, and the projectors have a small fill factor, resulting in a super-high resolution display.
The display output is sensed by an optical sensor, which may sense visible light, or non-visible light such as infrared or ultraviolet light. Preferably, the optical sensor comprises at least one camera, such as a CCD camera. Alternatively, the optical sensor may comprise a pair of orthogonal linear sensor arrays. The optical sensor is calibrated by positioning a calibration test chart at a projection surface. A mapping of projection surface positions or landmarks, to pixels in sensor space is then created by viewing the test chart with the optical sensor. The test chart is positioned, for example, by placing a physical test chart on the projection surface, or alternatively, by projecting the test chart onto the projection surface.
In various embodiments, the projection surface may be flat, spherical, otherwise curved, or even irregular.