The invention relates generally to range finder type image sensors, especially range finder image sensors that are implementable on a single integrated circuit using CMOS fabrication, and more particularly to reducing power consumption of systems utilizing such sensors.
Electronic circuits that provide a measure of distance from the circuit to an object are known in the art, and may be exemplified by system 10 FIG. 1. In the generalized system of FIG. 1, imaging circuitry within system 10 is used to approximate the distance (e.g., Z1, Z2, Z3) to an object 20, the top portion of which is shown more distant from system 10 than is the bottom portion. Typically system 10 will include a light source 30 whose light output is focused by a lens 40 and directed toward the object to be imaged, here object 20. Other prior art systems do not provide an active light source 30 and instead rely upon and indeed require ambient light reflected by the object of interest.
Various fractions of the light from source 30 may be reflected by surface portions of object 20, and is focused by a lens 50. This return light falls upon various detector devices 60, e.g., photodiodes or the like, in an array on an integrated circuit (IC) 70. Devices 60 produce a rendering of the luminosity of an object (e.g., 10) in the scene from which distance data is to be inferred. In some applications devices 60 might be charge coupled devices (CCDs) or even arrays of CMOS devices.
CCDs typically are configured in a so-called bucket-brigade whereby light-detected charge by a first CCD is serial-coupled to an adjacent CCD, whose output in turn is coupled to a third CCD, and so on. This bucket-brigade configuration generally precludes fabricating processing circuitry on the same IC containing the CCD array. Further, CCDs provide a serial readout as opposed to a random readout. For example, if a CCD range finder system were used in a digital zoom lens application, even though most of the relevant data would be provided by a few of the CCDs in the array, it would nonetheless be necessary to readout the entire array to gain access to the relevant data, a time consuming process. In still and some motion photography applications, CCD-based systems might still find utility.
As noted, the upper portion of object 20 is intentionally shown more distant that the lower portion, which is to say distance Z3 greater than Z2 greater than Z1. In a range finder autofocus camera environment, one might try to have devices 60 approximate average distance from the camera (e.g., from Z=0) to object 10 by examining relative luminosity data obtained from the object. In some applications, e.g., range finding binoculars, the field of view is sufficiently small such that all objects in focus will be at substantially the same distance. But in general, luminosity-based systems do not work well. For example, in FIG. 1, the upper portion of object 20 is shown darker than the lower portion, and presumably is more distant than the lower portion. But in the real world, the more distant portion of an object could instead be shinier or brighter (e.g., reflect more optical energy) than a closer but darker portion of an object. In a complicated scene, it can be very difficult to approximate the focal distance to an object or subject standing against a background using change in luminosity to distinguish the subject from the background. In such various applications, circuits 80, 90, 100 within system 10 in FIG. 1 would assist in this signal processing. As noted, if IC 70 includes CCDs 60, other processing circuitry such as 80, 90, 100 are formed off-chip.
Unfortunately, reflected luminosity data does not provide a truly accurate rendering of distance because the reflectivity of the object is unknown. Thus, a distant object surface with a shiny surface may reflect as much light (perhaps more) than a closer object surface with a dull finish.
Other focusing systems are known in the art. Infrared (IR) autofocus systems for use in cameras or binoculars produce a single distance value that is an average or a minimum distance to all targets within the field of view. Other camera autofocus systems often require mechanical focusing of the lens onto the subject to determine distance. At best these prior art focus systems can focus a lens onto a single object in a field of view, but cannot simultaneously measure distance for all objects in the field of view.
In general, a reproduction or approximation of original luminosity values in a scene permits the human visual system to understand what objects were present in the scene and to estimate their relative locations stereoscopically. For non-stereoscopic images such as those rendered on an ordinary television screen, the human brain assesses apparent size, distance and shape of objects using past experience. Specialized computer programs can approximate object distance under special conditions.
Stereoscopic images allow a human observer to more accurately judge the distance of an object. However it is challenging for a computer program to judge object distance from a stereoscopic image. Errors are often present, and the required signal processing requires specialized hardware and computation. Stereoscopic images are at best an indirect way to produce a three-dimensional image suitable for direct computer use.
Many applications require directly obtaining a three-dimensional rendering of a scene. But in practice it is difficult to accurately extract distance and velocity data along a viewing axis from luminosity measurements. Nonetheless many applications require accurate distance and velocity tracking, for example an assembly line welding robot that must determine the precise distance and speed of the object to be welded. The necessary distance measurements may be erroneous due to varying lighting conditions and other shortcomings noted above. Such applications would benefit from a system that could directly capture three-dimensional imagery.
Although specialized three dimensional imaging systems exist in the nuclear magnetic resonance and scanning laser tomography fields, such systems require substantial equipment expenditures. Further, these systems are obtrusive, and are dedicated to specific tasks, e.g., imaging internal body organs.
In other applications, scanning laser range finding systems raster scan an image by using mirrors to deflect a laser beam in the x-axis and perhaps the y-axis plane. The angle of defection of each mirror is used to determine the coordinate of an image pixel being sampled. Such systems require precision detection of the angle of each mirror to determine which pixel is currently being sampled. Understandably having to provide precision moving mechanical parts add bulk, complexity, and cost to such range finding system. Further, because these systems sample each pixel sequentially, the number of complete image frames that can be sampled per unit time is limited. It is understood that the term xe2x80x9cpixelxe2x80x9d can refer to an output result produced from one or more detectors in an array of detectors.
In summation, there is a need for a method and system that can produce direct three-dimensional imaging, preferably using circuitry that can be fabricated on a single IC using CMOS fabrication techniques, and requiring few discrete components and no moving components. Optionally, the system should be able to output data from the detectors in a non-sequential or random fashion. Very preferably, such system should require relatively low peak light emitting power such that inexpensive light emitters may be employed, yet the system should provide good sensitivity.
The present invention provides such a method.
The present invention provides a system that measures distance and velocity data in real time using time-of-flight (TOF) data rather than relying upon luminosity data. The system is CMOS-compatible and provides such three-dimensional imaging without requiring moving parts. The system may be fabricated on a single IC containing both a two-dimensional array of CMOS-compatible pixel detectors that sense photon light energy, and associated processing circuitry.
In applicant""s U.S. Pat. No. 6,323,942 B1 (2001) entitled xe2x80x9cCMOS-Compatible Three-Dimensional Image Sensor ICxe2x80x9d, a microprocessor on a CMOS-compatible IC continuously triggered a preferably LED or laser light source whose light output pulses were at least partially reflected by points on the surface of the object to be imaged For good image resolution, e.g., a cm or so, a large but brief pulse of optical energy was required, for example, a peak pulse energy of perhaps 10 W, a pulse width of about 15 ns, and a repetition rate of about 3 KHZ. While average energy in applicant""s earlier system was only about 1 mW, the desired 10 W peak power essentially dictated the use of relatively expensive laser diodes as a preferred energy light source. Each pixel detector in the detector array had associated electronics to measure time-of-flight from transmission of an optical energy pulse to detection of a return signal. In that invention, the transmission of high peak power narrow energy pulses required the use of high bandwidth pixel detector amplifiers.
Applicants"" referenced co-pending parent application disclosed a system that transmitted high frequency component periodic signals having low average power and low peak power, e.g., tens of mW rather than watts. For ease of analysis, optical energy periodic signals such as an ideal sinusoid waveform, e.g., cos(xcfx89xc2x7t), were assumed, and will be assumed herein. Emitting such low peak power high frequency component periodic signals permitted use of inexpensive light sources and simpler, narrower bandwidth pixel detectors. Bandwidths could be on the order of a few hundred KHz with an operating (emitted energy modulation) frequency of about 200 MHz. Good resolution accuracy was still obtainable using a low peak power optical emitter in that the effective duty cycle is greater than the output from a narrow-pulsed optical emitter of higher peak power.
In such system and in the present invention, assume that the energy emitted from the optical source is approximately S1=Kxc2x7cos(xcfx89xc2x7t) where K is an amplitude coefficient, xcfx89=2Πf, and frequency f is perhaps 200 MHz. Assume further that distance z separates the optical energy emitter from the target object. For ease of mathematical representation, K=1 will be assumed although coefficients less than or greater than one may be used. The term xe2x80x9capproximatelyxe2x80x9d is used in recognition that perfect sinusoid waveforms can be difficult to generate. Due to the time-of-flight required for the energy to traverse distance z, there will be a phase shift xcfx86 between the transmitted energy and the energy detected by a photo detector in the array, S2=Axc2x7cos(xcfx89xc2x7t+xcfx86). Coefficient A represents brightness of the detected reflected signal and may be measured separately using the same return signal that is received by the pixel detector.
The phase shift xcfx86 due to time-of-flight is:
xcfx86=2xc2x7xcfx89xc2x7z/C=2xc2x7(2xc2x7Πxc2x7f)xc2x7z/C
where C is the speed of light 300,000 Km/sec. Thus, distance z from energy emitter (and from detector array) is given by:
z=xcfx86xc2x7C/2xc2x7xcfx89=xcfx86xc2x7C/{2xc2x7(2xc2x7Πxc2x7f)}
Distance z is known modulo 2ΠC/(2xc2x7xcfx89)=C/(2xc2x7f). If desired, several different modulation frequencies of optically emitted energy may be used, e.g., f1, f2, f3 . . . , to determine z modulo C/(2xc2x7f1), C/(2xc2x7f2), C/(2xc2x7f3). The use of multiple different modulation frequencies advantageously can reduce aliasing. If f1, f2, f3 are integers, aliasing is reduced to the least common multiplier of f1, f2, f3, denoted LCM(f1, f2, f3). If f1, f2, f3 are not integers, they preferably are modeled as fractions expressible as a1/D, a2/D, and a3/D, where i in ai is an integer, and D=(GCD) represents the greatest common divisor of a1, a2, a3. From the above, distance z may be determined modulo LCM(a1, a2, a3)/D. This same analytical approach is also practiced with the various embodiments of the present invention, described later herein.
Phase xcfx86 and distance z were determined by mixing (or homodyning) the signal detected by each pixel detector S2=Axc2x7cos(xcfx89xc2x7t+xcfx86) with the signal driving the optical energy emitter S1=cos(xcfx89xc2x7t). The mixing product S1xc2x7S2 is 0.5xc2x7Axc2x7{cos(2xc2x7xcfx89xc2x7t+xcfx86)+cos(xcfx86)} and will have a time average value of 0.5xc2x7Axc2x7cos(xcfx86). If desired, the amplitude or brightness A of the detected return signal may be measured separately from each pixel detector output.
To implement homodyne determination of phase xcfx86 and distance z, each pixel detector in the detector array had its own dedicated electronics that included a low noise amplifier to amplify the signal detected by the associated pixel detector, a variable phase delay unit, a mixer, a lowpass filter, and an integrator. The mixer mixed the output of low noise amplifier with a variable phase delay version of the transmitted sinusoidal signal. The mixer output was lowpass filtered, integrated and fedback to control phase shift of the variable phase delay unit. In the equilibrium state, the output of each integrator will be the phase "psgr"(where "psgr"=xcfx86xc2x1Π/2) associated with the TOF or distance z between the associated pixel detector and a point a distance z away on the target object. The analog phase information is readily digitized, and an on-chip microprocessor can then calculate z-values from each pixel detector to an associated point on the target object. The microprocessor further can calculate dz/dt (and/or dx/dt, dy/dt) and other information if desired.
However applicants"" referenced parent co-pending provisional application substantially enhances detection sensitivity for such systems in which low peak power high frequency component periodic signals were used, and in which phase delay is used to determine TOF, dz/dt (and/or dx/dt, dy/dt, and other information. More specifically, an improved mixer is described, in which mixing results from modulating quantum efficiency (QE) of the photodiodes in the detector array, for example through use of a MOS transistor gate or altering reverse bias of the photodiodes. Such mixing offers many advantages including improved high frequency sensitivity, improved detection signal/noise, smaller form factor, lower power consumption, and less cost to fabricate.
Several embodiments of QE modulation are described in the present invention. Conceptually the embodiments may be grouped into two general categories. One category involves variable phase delay approaches (not unlike those described in applicants"" co-pending application Ser. No. 09/876,373) but in which dedicated electronic mixers (e.g., Gilbert cells) are replaced by QE modulation. A second category involves mixing with fixed phase delays using QE modulation, and implements a variety of spatial and temporal multiplexing approaches. Advantageously, both methods can modulate QE of MOS-implemented photodiodes by changing photodiode reverse bias, or by providing MOS-implemented photodiodes with a photogate, and then changing the gate voltage. Single-ended or double-ended differential signal processing may be employed with both methods. Differential QE modulation advantageously allows faster QE modulation, and provides a differential output that substantially removes common mode effects due to ambient light and photodiode dark current. In general, both categories of methods advantageously accumulate photodetector signal charge on a photodiode capacitor. If desired, accumulated charge may be examined periodically when QE modulation is stopped. Such signal accumulation approaches are preferred over methods that seek to directly measure a high frequency small magnitude photocurrent.
Using variable phase delay (category one), photocurrent from each QE-modulated pixel photodiode (or photogate photodiode) is coupled as input to an associated relatively high input impedance amplifier that need not exhibit high bandwidth, high frequency response, or high closed-loop gain. The amplifier output feeds directly to a low pass filter (LPF) whose output drives an integrator. The integrator output is coupled as to control phase of the variable phase delay (VPD) that controls QE modulation signals that drive the photodetector diodes. The VPD is also driven by a signal from the periodic signal generator that controls the optical energy emitter. There may or may not be a DC offset associated with the output signal from the pixel photodiode detectors and with the homodyne drive signal. Assuming no offsets, at steady-state the LPF output will be zero. Assuming appropriate DC offsets, at steady-state the LPF output will be a minima or a maxima. This method may be implemented single-ended, or preferably double-ended using a complementary approach in which positive and negative signals are derived from photodiodes that are QE modulated out of phase.
Using fixed phase delay (category two) fixed homodyne signals are used to QE modulate each photodetector. In category two, different groups or banks of photodiode detectors may be defined in a non-localized manner within the array. For example, a first bank of photodiode detectors may be QE modulated with fixed 0xc2x0 phase shift, a second bank may be QE modulated with fixed 90xc2x0 phase, shift, a third bank with fixed 180xc2x0 phase shift, and a fourth bank with fixed 270xc2x0 phase shift. Within each pixel, there may be photodiode detectors that correspond to every one of the four banks. Phase information and target object brightness information can be determined by examining output values for each bank within a pixel. This fixed delay approach simplifies the electronic circuitry associated with each pixel, reduces power consumption, can reduce IC chip area requirement, and enables a range of techniques for temporal and spatial multiplexing.
In the various embodiments of the present invention, on-chip measurement information may be output in random rather than sequential order, and on-chip signal processing for object tracking and other information requiring a three-dimensional image can be readily accomplished. The overall system is small, robust, requires relatively few off-chip discrete components, and exhibits improved detection signal characteristics. On-chip circuitry can use such TOF data to readily simultaneously measure distance and velocity of all points on an object or all objects in a scene.
Other features and advantages of the invention will appear from the following description in which the preferred embodiments have been set forth in detail, in conjunction with their accompanying drawings.