The field includes automated microscopy. More particularly, the field includes automatically focusing a microscope using a reflection from a reference surface positioned with respect to an objective of the microscope and information acquired from magnified images obtained by the microscope. In particular, the field includes a multifunction autofocus mechanism for automated microscopy that combines reflective positioning with use of the image characteristics obtained from a sequence of focal planes.
Automated microscopy refers to a microscope system including a microscope equipped with one or more motor-actuated components and a programmable control mechanization that governs the functions of the system and operates one or more motor-actuators to set and/or change operational conditions of the microscope. Such conditions include, for example, focus, illumination, and specimen position. The microscope includes an objective and a stage. In particular, but without limitation, the control mechanization operates to automatically focus the microscope and to scan a specimen via the microscope by generating control signals that cause relative movement between the objective and the stage. In some aspects, the stage is motorized and the control mechanization, using autofocus and positioning algorithms, processes images to generate control signals operative to cause the motorized stage to be moved and/or positioned in three dimensions with reference to the objective. See, for example, http://www.microscopvu.com/articles/livecellimaging/automaticmicroscope.html.
Image-based autofocus of a microscope uses a value obtained by transformation of a magnified image to indicate a degree of focus or sharpness. For example, but without limitation, presume an automated microscope stage supports a sample plate with one or more samples on a surface thereof. (The term “sample plate” is used herein to denote a microscope component such as a microscope slide, a multiwall plate, a microtiter plate, or equivalent). The stage is operated to place the sample plate at an X-Y location where the stage is moved vertically through a succession of vertical (Z) positions. At a first Z position, a magnified image is obtained from the sample, and the magnified image is transformed to obtain a first value of an image characteristic such as focus. A second value is obtained by the same transformation of another magnified image obtained at a second Z position. More values are obtained at additional vertical positions. The values are compared and an automated stage controller is operated to move the stage to a vertical position where the values indicate the best degree of focus is obtained. In automated scanning microscopy, best focus is maintained in this manner as the sample plate is scanned by systematically moving the stage in an X-Y scan pattern. See, for example, U.S. Pat. Nos. 5,548,661; 5,790,710; 5,790,710; 5,995,143; 6,640,014; and 6,839,469. Other mechanisms can be utilized to change focus to different focal planes of a specimen, including movement of the objective relative to the stage. If a camera is used for image acquisition, the camera can be moved relative to the objective, and/or optical elements between the camera and objective can be moved.
Reflective positioning autofocus does not use a magnified image obtained from a sample to determine best focus; rather, it uses a light signal that is generated by the microscope's optical system, projected through the objective toward the sample plate, and reflected off of a surface of the sample plate, back through the objective. A desired focus location for the sample is obtained by a value entered by a user that vertically offsets the focal plane from the Z position where the reflected light signal is in focus. See, for example, U.S. Pat. No. 7,071,451; see also the Nikon® perfect focus system description at http://www.microscopvu.com/tutorials/flash/focusdrift/perfectfocus/index.html.
Automated digital imaging with a high resolution microscope requires autofocus that performs with submicron accuracy and precision. Both image-based autofocus and reflective positioning exhibit shortcomings that compromise performance in virtually all modes of high resolution whole-slide and whole-microtiter plate automated imaging. Image-based autofocus does find a best focus location in a three-dimensional specimen, but it is slow. Image-based autofocus finds the best focus as indicated, for example, by a best resolution of the sample itself; but it can be fooled by empty fields of view and artifacts (e.g., clumps of cells or debris). It has been found that the speed of image-based autofocus can be increased by taking advantage of chromatic aberrations to increase the number of planes per image at which sharpness/resolution measurements are made simultaneously, thereby also reducing the complexity of the multi-planar imaging optics. See, in this regard, US 2010/0172020, published Jul. 8, 2010, for “Automated Scanning Cytometry using Chromatic Aberration for Multiplanar Image Acquisition”. Reflective positioning is designed to speed up total internal reflective (TIRF) microscopy where the goal is to find a single image plane—the surface at which the image is created. However, it cannot find the best focal planes of cells and tissue sections that inevitably reside at different distances from the reflective surface. Thus, although reflective positioning is fast, it requires an estimate of the axial offset from the reflecting surface that can exhibit substantial variability during a scan, thereby often deviating from best focus.
The combination of automated microscopy with image cytometry has become an essential biological research tool that is incorporated into a wide variety of applications. Such applications include: high content screening (HCS) instruments use in compound (drug discovery and toxicity) and genomic (RNAi and cDNA) automated screening, where libraries of hundreds to tens of thousands can be screened by individual investigators and whole genomes and compound libraries of hundreds of thousands can be screened by large laboratories; high throughput microscopy (HTM) where screening per se is not necessarily the experimental goal; and image (or scanning) cytometers where the emphasis is on automated spatially dependent measurements of large numbers of cells as a superset of the measurements acquired by flow cytometry instruments. The latter application tends to be used more in analysis of tissue sections and the former more often for cell monolayers, but the principles are analogous. In the last decade, HCS/automated image cytometry has grown rapidly not only for large screening efforts but also for routine use in individual labs enabling automated parallel analysis of from one to a few 96-, 384-well microtiter plates where months or years of experiments can be distilled into a few weeks (aka, HC analysis or HCA). Advances in automatic acquisition, measurement, comparison and pattern classification tools for cellular images continue to add sophisticated new automation for mining statistically relevant quantitative data from the inherently heterogeneous cell populations.
Obtaining a sharply focused image automatically is a key requirement for automated microscopy. It is therefore desirable to provide autofocus systems and methods that improve the speed and image sharpness of an automated microscopy system while still providing a best focus for each magnified image acquired.
Measurement of Image-Based Autofocus Accuracy
Best focus of a microscope is the Z (axial) position with the best resolution of an image. Autofocus error is, in turn, defined by an axial distance from best focus and a degree of defocus is reported as a percent loss in resolution.
To find best focus as a standard against which to compare autofocus methods, measurements of the slope of the change in brightness of the images of a “knife edge” were made on through-focus stacks of reflection images centered on the sharp edge between 100% transmitting and 100% reflective regions on a microscope resolution target. The derivatives of 35 lines perpendicular to the knife edge and near the center of the through-focus stacks of the knife edge were fit with Gaussians. The resolution was computed as 2.77*(mean of standard deviations (“SDs”) of the 35 Gaussian fits of each optical section), which, in some respects, corresponds to translating this experimental knife-edge resolution measurement to the theoretical Rayleigh resolution criterion. The results for two different 20× objectives (0.75 NA 20× Nikon® Plan Apo VC and Nikon® 0.5 NA 20× Plan Fluor) are plotted in FIG. 1.
In FIG. 1, the left-hand graph shows resolution in μm as a function of defocus distance in micrometers (μm) with diamonds marking the best focus positions via an image-based autofocus system and method to be described. The right-hand graph shows the same data as a percent change in resolution (100*(Resolution—Best Resolution)/(Best Resolution)) plotted as a function of defocus distance. The knife edge measurements and the autofocus measurement were performed on the same stack of images. Table 1 summarizes best resolutions, autofocus errors, and corresponding resolution losses per FIG. 1.
TABLE 1Knife-Edge Best Resolutions and Autofocus ErrorsResolutionAutofocusAutofocus Loss(μm)Errorof Resolution0.75 NA1.0X Relay Lens0.71 μm0.250 μm0.020 μm (2.8%)20X:1.5X Relay Lens0.51 μm0.100 μm0.001 μm (0.2%)0.50 NA1.0X Relay Lens0.78 μm0.050 μm0.030 μm (3.8%)20X:1.5X Relay Lens0.69 μm0.650 μm0.040 μm (5.8%)
FIG. 1 and Table 1 show that:                1. Through-focus resolution measurements can objectively identify best focus,        2. Image-based autofocus is accurate (resolution losses of 0.2-6% are likely within resolution measurement noise—see also Table 2); it is a resolution measure that is designed to be insensitive to contrast reversals (See Bravo-Zanoguera, M. E., Laris, C. A., Nguyen, L. K., Oliva, M. & Price, J. H. Dynamic autofocus for continuous-scanning time-delay-and-integration image acquisition in automated microscopy. Journal of Biomedical Optics 12, 34011/34011-34016 (2007)),        3. Decreased sampling (1.0× instead of 1.5× tube lens) resulted in decreased peak resolution without always decreasing the sensitivity of resolution to defocus far from focus,        4. Lower NA objectives have both lower maximum resolution and greater depth of field than objectives with higher NA.        
This approach provides a way to grade the severity of reflective positioning and autofocus errors. A standard criterion to define “out of focus” for an image detector is the depth of field obtained per Equation (1):
                              d          tot                =                                                            λ                0                            ⁢              n                                      NA              2                                +                                    n              MNA                        ⁢            e                                              (        1        )            where λ0 is the wavelength of light, n is the index of refraction of the immersion medium, NA is the numerical aperture, M is the magnification, and e is smallest distance that can be resolved by the detector (From Inoue, S. & Spring K. R., Video Microscopy, The Fundamentals, (Plenum, N.Y., 1997). The depth of field and corresponding focus error data are summarized in Table 2.
TABLE 2Depth of Field Out of Focus Distances20X 0.75 NA20X 0.5 NAObjectiveObjective1.0X1.5X1.0X1.5XTube LensTube LensTube LensTube LensDepth of Field (μm)a1.23 μm1.04 μm2.66 μm2.43 μmResolution Loss (%)b21%23%29%34%Distances from Focus−0.6,−0.44,−1.38,−1.4,0.63 μm0.6 μm1.28 μm1.03 μmaSee Eq. (1).bDetermined from FIG. 1 data.
For comparison with detection of defocus by eye, see the enlarged through-focus sequences of DAPI-labeled NIH-3T3 cells in FIG. 2. The sequences were obtained using a Nikon® Ti-E microscope with a Nikon® 0.75 NA 20× Plan Apo VC objective and 1.0× and 1.5× tube lenses. The images were captured on a Hamamatsu® ORCA ER CCD (6.45×6.45 μm2 pixels) sampled at 0.323×0.323 μm2/pixel and 0.215×0.215 μm2/pixel, respectively. Defocus of +1.0 μm appears less blurry than −1.0 μm (compare with nuclear edges at 0.0-μm defocus); defocus is asymmetric about best focus consistent with optical theory. Note that in both sequences blur is visible 1.0 μm from best focus, which corresponds to about a 50% loss in resolution.
Both image-based autofocus and reflective positioning exhibit limitations that compromise performance for some fields of view encountered in both whole-slide and whole-microtiter plate scanning. One solution, as in the PerkinElmer Opera® imaging reader, is to remove some of the out-of-focus information with confocal optics; but this is an expensive approach that may also be less optimal than imaging at best focus. Reflective positioning can take <0.1 s and autofocus can take as long as 1-2 s using previous technology. Reflective positioning locates a surface and collects the image at a preset user-defined offset from the surface.
Autofocus is an important component of microscope automation: if autofocus is poor, the primary output—the image, as well as the data derived from it—are compromised. As opposed to macroscopic digital imaging (photography), where depths of field even with close-up low F/# lenses are measured in millimeters, the microscopic depth of field with a 20× 0.75 NA objective and 0.52 NA condenser is 1.23 μm with a 1.0× tube lens (see Table 2). This makes microscopy autofocus substantially more challenging than autofocus in macroscopic photography. At about NA 0.5 and higher, depths of field are often comparable to the thicknesses of commonly cultured, tightly adherent cells (rounded, less adherent cells, such as T-lymphocyte cell lines, exceed the depth of field at even lower NAs). Many reflective systems sense only the first surface, and most specimen holders (coverslips, slides, and membrane-, glass-, and plastic-bottom microtiter plates) vary in thickness. Reflective system characteristics include: absolute (open loop) displacement sensors, error-minimizing (closed loop) positioning, and through the lens (TTL) and non-TTL (or beside the lens, BTL) configurations. By observing the digital images produced by reflective positioning HCS instruments as they scan, out-of-focus images are visible to the eye. Image-based autofocus optimizes the quality, sharpness, and or resolution of the image itself depending on the algorithm. But autofocus can also be compromised by a poor specimen (e.g., thick piles of dead cells or debris and lint or fibers occupying large axial ranges), out-of-focus contrast-generating artifacts (e.g., cells, debris, scratches, condensation, dried water droplets and lint on the other surfaces of the specimen holder—coverglass, slide or plate bottom), and by mismatches in image sampling that alter the system modulation transfer function and violate the assumptions about the native resolution of the image upon which the sharpness measurements rely. There are many other things that can compromise image quality, including not properly aligning the condenser and lamp. Relative merits of the two approaches are summarized in Table 3.
TABLE 3Summary of Autofocus vs. Reflective Positioning TradeoffsAutofocus - Focuses on ImageReflective Positioning to a SurfaceDirectly optimizes image quality“Guesses” specimen location based on user-defined offset from a specimen holder surfaceAdvantages include:Advantages include:Focuses properly even if the absoluteRequires less calibrationspecimen position changesLikely to work even if microscope notFinds best “average” focus position inalignedthicker samplesAlways positions in the same location,Discerns focus with better precision thaneven if the sample very thick or if thereby eyeis debrisBest for total internal reflectionmicroscopy (TIRF)Poor focus possible with:Poor focus possible with:Thick debrisNA ≥0.5 where depths of field approachOut-of-focus artifacts on other surfacesspecimen thicknessesincluding dirty opticsSubstrates with varying thickness wherePoorly aligned microscope componentsthe first surface is sensed and theMismatches in optical sampling (cameraspecimen is on the second surfaceresolution vs. optical magnification, NA,Coatings on plates, slides andzoom, etc.)coverslips - e.g., Matrigel for primaryDim illumination, or too brightcells - that vary in thicknessillumination (above camera saturation)Offset not recalibrated for changes inpreparation - e.g., altering adherence ofcells with coatingsOther factors:Other factors:Slow: ~0.25 s (IC 100, adjacent images)Fast (10 s of ms)Slower when next field is far out of focusSpeed minimally dependent on re-focus(e.g., from well to well in irregulardistance (e.g., with piezo focus, add ~10 ms/microtiter plates)50 μm)
We observed that with cell monolayers cultured on a coverslip, reflective positioning can provide reasonably focused images, whereas with cells cultured on the slide or tissue sections mounted on the slide, the degree of defocus appeared greater.
Image-based autofocus, on the other hand, finds the sharpest image directly from a stack of images. For example, autofocus functions based on image resolution find the axial position where the most cells are in focus if the tissue is thicker than the depth of field. (Bravo-Zanoguera, M. E., Laris, C. A., Nguyen, L. K., Oliva, M. & Price, J. H. Dynamic autofocus for continuous-scanning time-delay-and-integration image acquisition in automated microscopy. Journal of Biomedical Optics 12, 34011/34011-34016 (2007)) However, if the field is blank or if there is debris out of the normal plane of focus, autofocus can lose track of focus.
To understand these errors systematically, we used both image-based autofocus and reflective positioning to scan three specimens:
1) NIH 3T3 cells cultured on a coverslip,
2) A histological section of human prostate cancer, and
3) Adipocytes differentiated from 3T3-L1 cells cultured in a 96-well microtiter plate
FIG. 3 illustrates differences between image-based autofocus and a commercially-available reflective positioning system (a Nikon® Perfect Focus System) on DAPI labeled NIH 3T3 cells cultured on a coverslip using a Ti-E with image-based autofocus added. A 4×1 cm2 area comprising 157×67=10,519 fields (images) was scanned with reflective positioning and image-based autofocus also calculated by sampling 20 focal planes in a range of 10 μm centered at the reflective positioning offset (20× 0.75 NA VC). The histogram in FIG. 3 shows that for NIH 3T3 cells grown on a coverslip, the range of reflective positioning errors relative to image-based autofocus is about ±4 μm with most errors <±2 μm. The two example fields exhibiting reflective positioning focus error in FIG. 3 also show how image-based autofocus succeeds in focusing on most of the nuclei even where cells are distributed axially beyond the depth of field (1.23 μm for the 20× 0.75 NA objective used for FIG. 1, as shown in Table 2).
A test of the differences between image-based autofocus and reflective positioning on a prostate cancer tissue section using the same experimental setup are shown in FIG. 4. This figure shows differences between image-based autofocus and reflective positioning (using a Nikon® microscope equipped with the Perfect Focus System that performs reflective positioning) on a 5-μm thick section cut from formalin-fixed paraffin-embedded (FFPE) prostate cancer tissue and labeled with DAPI (blue) and for AMACR (green) and ACTA2 (red). A 2.2×1.0 cm2 area comprising 94×64=6,016 fields was scanned with reflective positioning and image-based autofocus calculated by sampling 20 focal planes in a range of 10 μm centered at the reflective positioning offset (using the 20× 0.75 NA VC objective). The error range shown in the histogram in FIG. 4 is larger at about ±7.5 μm than for cells cultured on the coverslip (FIG. 3). The 3D plot of focus errors exhibits tilt likely because the coverslip is not parallel to the slide, which contributes to the larger range of errors. Conversely, focus tracking with image-based autofocus is compromised at the left and right corners of the scan region in the 3D plot of focus differences because those areas of the slide contained no tissue. Out of focus empty fields aren't relevant, but focus tracking at the next field of view containing tissue can be compromised if the system wanders too far from best focus. The present disclosure provides a way to correct for this by using reflective positioning in conjunction with image-based autofocus.
FIG. 5 shows reflective positioning focus errors on a Nikon® perfect focus system with a Nikon® 0.75 NA 20× Plan Apo VC plotted (left) for a 96-well microtiter positive control plate for a lipid-reducing assay (for screening for compounds that inhibit lipid formation as might be utilized to screen to discover new weight loss drugs) in adipocytes differentiated from 3T3-L1 cells (top). Image-based autofocus was performed on the DAPI channel in a search range of 10 μm in 0.5 μm steps and 16 fields/well were scanned. The reflective positioning error envelope in the histogram in FIG. 5 is about −3 to +5 μm (see the histogram, FIG. 5, middle-right). The middle-left color temperature plot best demonstrates the unexpected trend of changing reflective positioning errors across the plate (likely due to changing bottom thickness). This trend is seen as a tilt in the error from row 1 to row 8 (see the heat map and 3D plot in FIG. 5, bottom), consistent with a change in thickness of ˜2 μm across the coverglass bottom of the plate. The middle-right histogram of the error distribution shows that a substantial number of the images are >1.0 μm, or more, out of focus, i.e., visibly blurred. The lower 3D surface plot best demonstrates the local variations in reflective positioning errors. The positive control plate for inhibition of lipid formation is created by beta-adrenergic stimulation of the cells with isoproterenol, which results in hormone sensitive lipase (HSL) activation (quantified via increased brightness of anti-HSL fluorescent-labeled antibody, (anti-HSL) in rings around the lipid droplets) followed by a reduction of lipid (quantified by cellular brightness of a fluorescent lipid label). The image colors are blue (DAPI/Nuclei), green (lipid droplets), and red (anti-HSL). The HSL rings around the lipid droplets exhibit the kind of fine detail (rings are especially visible in the anti-HSL image) expected to be most sensitive to blurring caused by focus errors.
For all three specimens (FIGS. 3-5)—cells cultured on a coverslip, a tissue section on a slide and cells cultured on a coverslip-bottom microtiter plate—the error distributions indicate that many images were out of focus using reflective positioning with an offset. In Table 4, the percentages of images out of focus are collated using both the depth of field criterion and an arbitrary loss in resolution of 50% (compare FIGS. 1 and 2).
TABLE 4Percentages of Images Out of Focus>Depth of Field>50%(21% ResolutionResolutionLoss)LossNIH-3T3 Cells on Coverslip40%21%Prostate Cancer Tissue Section80%71%Adipocytes on Coverslip-Bottom68%55%96-well Plate
NIH-3T3 cells are adherent and flat, and are thus expected to provide a straightforward test for reflective positioning with an offset. Nevertheless, over 20% of the images are out of focus as judged by a 50% loss in resolution and 40% are out of focus as judged by the depth of field criterion.
In the examples of FIGS. 3-5, image-based autofocus reliably delivered the best focused images in “normal” fields of view when a difference could be discerned by eye. “Normal” fields of view are those with tissue that have no out-of-focus debris. We made three observations based on these experiments (FIGS. 3-5). First, image-based autofocus delivers better-focused images than reflective positioning on “normal” fields of view that have tissue present and are not compromised by debris out of the focal plane of the target tissue. Second, with reflective positioning (+ offset), 21-80% of the images were out of focus. Finally, reflective positioning kept track of the focal plane, even without any tissue present, keeping the cells within the 10-μm search range of image-based autofocus for a large proportion of the thousands of fields of view.