Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy.pdf
Article https://doi.org/10.1038/s41467-024-48765-5 Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy Jie Zhang1,2 , Jonathan Newman1,2, Zeguan Wang2,3, Yong Qian2,3, Pedro Feliciano-Ramos1,2, Wei Guo1,2, Takato Honda 1,2, Zhe Sage Chen 4, Changyang Linghu 5, Ralph Etienne-Cummings6, Eric Fossum7, Edward Boyden 2,3 & Matthew Wilson 1,2 High-speed wide-ﬁeld ﬂuorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio at high frame rates, limiting their ability to detect faint ﬂuorescent events. Here, we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high signal-to-noise ratio. In high-speed voltage imaging experi- ments, our image sensor signiﬁcantly increases the output signal-to-noise ratio compared to a low-noise scientiﬁc CMOS camera (~2–3 folds). This signal-to- noise ratio gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientiﬁc CMOS cameras. Our camera with ﬂexible pixel exposure conﬁgurations offers versatile sampling strategies to improve signal quality in various experimental conditions. Wide-ﬁeld ﬂuorescence microscopy enables direct observation of physiological processes, such as cell signaling and local chemical concentrations. In neuroscience, this method has revolutionized the study of the neural basis of behavior by capturing the dynamics of thousands of cells in behaving animals1. However, challenges still remain in using functional ﬂuorescence microscopy to sample fast neural dynamics2–6, such as the membrane potential that reﬂects neural activity occurring at millisecond timescales. High sampling speeds (1 kHz) of ﬂuorescence indicators increase pixel readout noise and limit the number of ﬂourescence photons integrated during each sampling period. These factors lead to a low signal-to- noise ratio (SNR) for detecting ﬂuorescence activities associated with spiking events and subthreshold activity. Imaging setups can use high excitation power to maintain the SNR, but suffer from many caveats, such as photobleaching, heat, and cytotoxicity. Currently, experiments at kHz camera speed were conducted only in head-ﬁxed animals using high-performance but bulky micro- scopes and were limited to a few minutes in duration5,6, which was insufﬁcient to observe many behavioral states and their associated neural mechanisms. While many efforts have focused on improving the imaging sys- tems’ optics7,8 and ﬁnding denoising algorithmic solutions9,10, the key trade-offs of speed and SNR are also fundamentally linked to the image sensor. In this work, we examine and address the limitations of SNR at high speed by introducing a pixel design and sampling method. Advancements in sensor technology will complement ongoing inno- vations in optics and denoising algorithms. Together, they will enhance our ﬂuorescence imaging technology to enable the tracking of neural activity at millisecond resolution across a large number of neurons, over long experimental durations. Received: 17 June 2023 Accepted: 30 April 2024 Check for updates 1Picower Institute for Learning and Memory, MIT, Cambridge, MA, USA. 2Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA. 3McGovern Institute for Brain Research, MIT, Cambridge, MA, USA. 4Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA. 5Department of Cell and Developmental Biology, University of Michigan, Ann Arbor, MI, USA. 6Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA. 7Thayer School of Engineering, Dartmouth College, Hanover, NH, USA. e-mail: jzhang41@mit.edu Nature Communications|        (2024) 15:4480  1 1234567890():,; 1234567890():,; Results A fundamental trade-off exists between a pixel’s sampling speed and SNR. Fast sampling speeds lead to high readout noise, shortened exposure time, and fewer collected photons, inevitably lowering the SNR. The CMOS image sensor, which uniformly exposes and samples an array of pixels, is subject to the same SNR and speed limitation. High frame rates result in low SNR, and low frame rates with long pixel exposure lead to signal aliasing (Fig. 1a, b). Programmable pixel-wise exposure CMOS image sensor Despite this trade-off, physiological signals, such as the membrane voltage of a cell soma, are often spatially correlated, and the same signal can be redundantly captured by multiple pixels within a region of interest (ROI). We demonstrate a CMOS image sensor with pixel- wise programmable exposures (PE-CMOS) to take advantage of the highly correlated nature of microscopy scenes. The PE-CMOS permits ﬂexible exposure at each pixel. This feature allows versatile pixel conﬁgurations to increase temporal resolution at sampling physiolo- gical signals without sacriﬁcing SNR. In one conﬁguration, the PE- CMOS staggered pixels’ exposure in time to acquire fast-spiking events at multiple phases (Fig. 1c), resolving temporal details ﬁner than the sampling period and exposure time. Importantly, this increase in temporal resolution is achieved without raising the pixel sampling rate or reducing exposure time, therefore avoiding sacriﬁcing the SNR. In another conﬁguration, the PE-CMOS samples the ROI with pixels at different speeds, capturing high-frequency events (spiking activity) and weak signals (subthreshold potentials) that are difﬁcult to acquire simultaneously at a ﬁxed frame rate (Fig. 1c). The ﬂexible pixel-wise exposure conﬁguration is not achievable in conventional CMOS architecture, as all the pixels must have the same exposure, limited by the frame rate, and are sampled concurrently (global shutter) or sequentially in lines (rolling shutter). PE-CMOS circuit architecture The PE-CMOS sensor design enables pixel-wise programmability without compromising pixel sensitivity, which is determined by the percentage of effective photodiode area within each pixel (ﬁll factor). Prior approaches to achieve pixel conﬁgurability required a high number of pixel-level circuits11–13 (~25 transistors), which occupied valuable pixel area and reduced the photodiode ﬁll factor (38% and 45% for pixel pitches of 12.6 and 11.2 μm, respectively12,13). In contrast, the PE-CMOS achieves pixel-wise programmability using only six transistors per pixel, enabling high sensitivity (75% and 52% ﬁll factor for smaller pixel pitch of 10 μm and 6.5 μm), comparable to high- performance CMOS sensors (Fig. 1d, e). For the PE-CMOS with a 10 µm pitch, the measured conversion gain is 110 μV/e-, with a read noise of 2.67 e-, measured at room temperature without active cooling. The quantum efﬁciency (QE) of the pixel is 68% without on-chip micro- lens array. The PE-CMOS chip was fabricated using the 180 nm CMOS image sensor process from XFAB, a commercial foundry service. Two pro- totype designs were produced with pixel arrays containing a 6.5 µm (Chip A) and 10 µm (Chip B) pixel pitch (Supplementary Fig. 1a and Methods). The pixel array is arranged in rows and columns, with readout circuitry at each column (Supplementary Fig. 1b and Meth- ods). Each column readout circuitry comprises a programmable gain ampliﬁer (PGA), correlated-double sampling (CDS) circuits, and an analog-to-digital converter (ADC). The PGA offers a variable gain of 8 – 64 to the pixel output. The ampliﬁed signal is then processed by the CDS circuits to reduce ﬂicker (1/f) noise before being sampled by the 10-bit ADC. The row-wise signals (RST, TX, and SEL) multiplex pixels to the readout circuits at each column. These signals also control pixel reset (RST), charge transfer (TX), and row-wise multiplex (SEL) operations. In a single readout period (0.9 ms in our implementation), each column readout circuit selects pixels of the column that end the exposure and converts their outputs to digital bits that are then sent off-chip. The PE-CMOS pixel design (Fig. 1d) comprises 6 transistors (T1–T6) and one pinned photodiode (PD). T1 to T4 form the standard rolling shutter 4T-pixel design14. During pixel exposure, the PD con- verts incoming photons into electrons. When the PD is ready to be sampled, T3, controlled by a row reset (RST) signal, ﬁrst resets the voltage on the ﬂoating diffusion (FD) node. T4 (controlled by TX) then moves the electrons from the PD to FD, producing a change in voltage. The voltage signal is buffered by T1 and connected to the readout circuits by T2 (controlled by SEL), where the column readout circuitry samples it. We incorporated two additional transistors (T5 and T6) into the PE- CMOS pixel design to enable pixel-level exposure programmability. T5 and T6 operate as switches with input driven by column signals, EX. In PE-CMOS, each set of row signals (TX, RST, SEL) selects K rows of pixels as candidates for readout (K = 8 in Chip A, K = 4 in Chip B). At each column, out of these K pixels, only the pixel with both T5 and T6 acti- vated will end its exposure phase and be sampled by the column cir- cuitry. Meanwhile, the other pixels, with their T5 and T6 remaining off, will continue their exposure. T5 and T6 are controlled by column bus lines EX < 1:K> placed at each column. Pixel-wise exposure control can be achieved by synchronizing the EX signals with row signals (RST, SEL, and TX). An example pixel-wise operation diagram is shown in Sup- plementary Fig. 1d. For a group of K pixels, the pixel ends its exposure (at the time marked by dotted red line in Supplementary Fig. 1d) whenever its corresponding EX signal is high during the readout operation initiated by the signals: RST < N > , TX < N> and SEL < N>. Flexible pixel-wise conﬁguration We can conﬁgure the pixels into custom exposure and sampling pat- terns by controlling the EX-signals at each pixel column, accomplished through a high-speed I/O interface on the PE-CMOS chip. Supple- mentary Fig. 2 illustrates various examples of pixel arrangements, with each color representing a unique temporal conﬁguration. In a tiled spatial arrangement, pixels of different conﬁgurations are uniformly arranged in 2 × 2 windows, enabling uniform sampling of the entire array across four distinct temporal conﬁgurations (Supplementary Fig. 2a, b). Alternatively, arranging pixels of different exposures in a random spatial conﬁguration can maximize the incoherence property desirable in a compressed sensing imaging framework15. Pixel conﬁg- urations can be updated instantaneously via on-chip control. This enables the optimization of local spatiotemporal resolution and SNR in real-time. For example, the pixels can be organized in ROI-dependent spatial patterns (Supplementary Fig. 2a). Since many ﬂuorescence proteins are predominantly expressed on the cell membrane, we can encircle the bright outer contour of the cell ROI with fast pixels with short exposure times and high sampling rates while employing slower pixels in the dimmer regions of the ROI to enhance the SNR. Sampling voltage signals at high temporal resolution with phase-shifted long exposure pixels To demonstrate that the PE-CMOS can sample a high-temporal reso- lution voltage signal with low-speed pixels at different phase-offset, we compared PE-CMOS’s performance with a scientiﬁc CMOS camera (Hamamatsu Orca Flash 4.0 v3 sCMOS). Both sensors were used to simultaneously image (through a 50/50 beamsplitter) the spontaneous activity of cultured mouse hippocampal neurons expressing the ASAP3 genetically encoded voltage indicator (GEVI)3 (Fig. 2). We set the sCMOS camera to have a 1.25 ms exposure duration, which also determines its temporal resolution and sampling rate (Fs) of 800 Hz. In the PE-CMOS camera, we used a longer exposure time of 5 ms to integrate over the full spike width (half-width of the action potential generated by ASAP3 spikes is ~ 6 ms). We shift pixel exposures at phase offsets in multiples of 1.25 ms (Fig. 1c). This sets the PE-CMOS temporal Article https://doi.org/10.1038/s41467-024-48765-5 Nature Communications|        (2024) 15:4480  2 where En 2 RM × L denotes the sampling matrix representing pixel exposure and sampling operation that converts the ﬂuorescent signal vn 2 RL into the pixel outputs, yn 2 RM. Now, we can deﬁne an arbitrary signal, vROI, and rewrite each yn as: yn = En vROI + Enðvn  vROIÞ Note that we do not make the assumption that vROI is the average of the pixels. if we write the difference term φn = En vn vROI   , we can then concatenate the output of all of the N pixels into one vector: y = EvROI + φ ð6Þ where y = y1 .. . yN 2 64 3 75,E = E1 .. . EN 2 64 3 75and φ = φ1 .. . φN 2 64 3 75 and Given Eq. 6, we can ﬁnd vROI using a ridge regression by enforcing a L2 norm penalty: vROI = minvROI 1 2        y  EvROI         2 2 + λridge        vROI         2 2 and further derive its closed-form solution: ^vROI = ETE + λridgeI  1 ETy ð7Þ where ^vROI is the least-squared estimate of vROI, and the hyperpara- meter, λridge, controls the trade-off between the L2-norm of vROI and reconstruction error, denoted by ky  EvROIk2 = kφk2. Solving this regression should push vROI to be close to the average of the pixels ð  vROI = 1 N PN n = 1vnÞ, which by deﬁnition, minimizes the L2 norm of ky  EvROIk2. However, in the presence of noise, setting λridge could have a risk of overﬁtting. To avoid this, in the interpolation method used in our manuscript, we set λridge to be large, which con- servatively underﬁt our approximation for vROI. With a large λridge, the term ETE + λridgeI  1 will approximate a scaled identity matrix, and Eq. 7 becomes ^vROI ≈ETy ð8Þ which is equal to up-sampling and interpolation of the pixel outputs y. We can reinforce our analysis with a simple example in Supple- mentary Fig. 8. Here, the time series of 4 pixels, v1 … v4, contains uncorrelated signal, with vROI representing the average of these sig- nals. We mimic the exposure and phase-shifted sampling of these pixels to get y1 … y4. We do this by convolving phase-shifted version of v1 … v4 with a box function of length 4, followed by 4 x down-sam- pling. We can see that down-sampling aliases the high-frequency part of the signal of y2, especially at the peak of the spike (Supplementary Fig. 8 black arrow). We can apply our interpolation outlined in Eq. 8, which results in ^vROI. We can see that ^vROI is proportional to vROI, but underﬁts the spike at the location pointed by the black arrow in Supplementary Fig. 8. This is because we only sampled y2 at a single phase since y2 is uncorrelated with other pixels. This example shows that even with uncorrelated pixels, inter- polating them yields an approximation of the ROI average. As shown, the selection of large λridge in the ridge regression minimizes the overﬁtting of noise during interpolation. Voltage imaging in dissociated neuron cultures Neuron culture preparation and AAV transduction. All procedures involving animals were performed in accordance with National Insti- tute of Health Guide for Laboratory Animals and approved by the Massachusetts Institute of Technology Committee on Animal. Dis- sociated hippocampal neurons were prepared from postnatal day 0 or 1 Swiss Webster mice (Taconic) without regard to sex following the protocol28. Dissected hippocampal tissue was digested with 50 units of papain (Worthington Biochem) for 6–8 min, and the digestion was stopped with ovomucoid trypsin inhibitor (Worthington Biochem). Cells were plated at a density of 40,000–60,000 per glass coverslip coated with Matrigel (BD Biosciences). Neurons were seeded in 200 µl plating medium containing MEM (Life Technologies), glucose (33 mM, Sigma), transferrin (0.01%, Sigma), Hepes (10 mM, Sigma), Glutagro (2 mM, Corning), Insulin (0.13%, Millipore), B27 supplement (2%, Gibco), and heat inactivated FBS (7.5%, Corning). After cell adhesion, additional plating medium was added. AraC (0.002 mM, Sigma) was added when glia density was 50–70% of conﬂuence. Neurons were grown at 37 °C and 5% CO2 in a humidiﬁed atmosphere. We transduce cultured neurons at 5–7 days in vitro (DIV) by administering ~ 1010 viral particles of AAV9-hSyn-ASAP3 (Janelia Viral Tools) per well (of 24-well plate). Voltage imaging was performed 7–14 days after transduction. Microscope setup (without patch clamp). Cultured hippocampal neurons expressing ASAP3 were imaged on a customized upright ﬂuorescent microscope with a 20 × 1.0NA objective lens (Olympus). The light from a 470 nm LED (Thorlabs) was cleaned with a 469/35 nm band pass ﬁlter (Semrock) for excitation. A 488 nm long pass dichroic mirror and a 496 nm long pass ﬁlter were used for illumination and emission. Using a 50/50 beam splitter (Thorlabs), the sample image was evenly split onto both the Hamamatsu sCMOS camera and our PE- CMOS sensor for side-by-side comparison. Whole cell patch clamp and imaging. Intracelluar recordings were acquired using the multiclamp 700B ampliﬁer and pCLAMP 10.0 soft- ware (Molecular Devices), ﬁltered at 2 kHz and digitized at 10 kHz. Cultured neurons were patched with pipettes ﬁlled with a potassium gluconate-based intracellular solution containing the following (in mm): 135 mM K-gluconate, 0.1 mM CaCl2, 0.6 mM MgCl2, 1 mM EGTA, 10 mM HEPES, 4 mM Mg-ATP, 0.4 mM Na-GTP, and 4 mM NaCl, with dropwise addition of 5 M KOH to adjust the pH to 7.2 and addition of potassium gluconate in increments of 25 mg until the ﬁnal osmolarity reached 290–295 mosmol/kg H2O. The open pipette resistance in these experiments was 3–6 mΩ. The current step protocol consisted of 200-ms-long constant level current steps from + 600 pA–0 pA with gradually decreasing 40 pA steps. During patch clamp, the ASAP3 expressing neuron are simultaneously imaged through a 40 x / 0.6 NA objective lens. Using a 50/50 beam splitter (Thorlabs), the sample image was evenly split onto both the Hamamatsu sCMOS camera and our PE-CMOS sensor for side-by-side comparison. ROI time series extraction. We hand selected the ROI using the max projection image of the PE-CMOS and sCMOS videos. Due to differ- ences in pixel size (PE-CMOS: 10 µm pixel pitch, sCMOS: 6.5 µm), sCMOS ROI contains ~2.3 x more pixels than that of the PE-CMOS ROI. PE-CMOS pixels are arranged in the tile conﬁguration (Supplemen- tary Fig. 2). For the sCMOS, we computed the ROI time series by averaging all the pixels values. We then ﬁltered the ROI time-series using a band- Article https://doi.org/10.1038/s41467-024-48765-5 Nature Communications|        (2024) 15:4480  10 pass ﬁlter to remove photo-bleaching effects and high-frequency content too close to the sampling Nyquist frequency (Filter para- meters: 4th-order IIR, with high-pass cut-off at 0.5 Hz and low-pass cut- off at 360 Hz, 90% of the Nyquist bandwidth, 400 Hz, half of pixel sampling speed). For visualization purpose, we invert the time series as ASAP3 GEVI has negative going action potential waveforms. For the PE-CMOS, we ﬁrst computed sub-ROI time series for pixels with the same TE and Δ. We then ﬁltered each sub-ROI time series using a band-pass ﬁlter to remove photo-bleaching effects and high- frequency content too close to the sampling Nyquist frequency. The ﬁlter parameters used in Fig. 2, 3 are: 4th-order IIR, with high-pass cut- off at 0.5 Hz and low-pass cut-off at 90 Hz, 90% of the pixel Nyquist bandwidth 100 Hz. The same ﬁlter is used in Fig. 4, but with low-pass cut-off changed to 234 Hz, 117 Hz, 58 Hz, and 29 Hz, corresponding to different pixel Nyquist bandwidth 260, 130, 65 and 33 Hz. The ROI time series representing the ROI average are interpolated from sub-ROI time-series using method described earlier. No additional ﬁlters are used for the PE-CMOS ROI time-series. To compare the sCMOS output to that of the PE-CMOS atdifferent Δ in Fig. 2, which compose of ~1/4 pixels of the ROI, we computed four sCMOS sub-ROI time series each using only 1/4 ROI pixels extracted through spatial downsampling. The average of these 4 sub-ROI time series is equal to the ROI time series. Noise estimation. To estimate the noise of the times series, we ﬁnd a section of the signal with at least 2 seconds in duration that does not contain any spikes. We then high-pass ﬁlter this trace to remove slow subthreshold oscillations (Filter parameters: 4th-order IIR, fcutoff = 50 Hz). The root-mean squared (rms) noise is then calculated as the standard deviation of the ﬁltered time-series. This value is referred to as 1 SNR in the ﬁgures and text. Putative spike detection. ROI times-series are ﬁrst high-pass ﬁltered to remove high amplitude oscillations at low frequency (Filter para- meters: 4th IIR ﬁlter, fcutoff = 2 Hz). From the ﬁltered PE-CMOS time series, we detected events with SNR > 5 and record their SNR and time of occurrence. We then record the highest SNR in the sCMOS times series within the corresponding time window. We set a 30 ms window centering at the peak time of PE-CMOS to avoid mismatching the peak SNR between PE-CMOS and sCMOS time series. General imaging applications using PE-CMOS We demonstrated two general imaging examples that beneﬁts from using the PE-CMOS. In the ﬁrst example, the PE-CMOS pixels are con- ﬁgured to have ﬁxed exposure with TE = 40 ms and Δ = 0, 10, 20, 30 ms (Supplementary Fig. 6a) This corresponds to pixel sampling speed of 25 Hz. We imaged the motion of bouncing ball with static background (Supplementary Fig. 6b). Staggered exposure embeds the motion within full resolution frames (Supplementary Fig. 6b). We split each frame into four subframe organized according to pixel phase, which yields a 100 Hz equivalent video at 1/4 spatial resolution. From the high-speed video, we can track the motion of the bouncing ball with temporal precision of 10 ms (Supplementary Fig. 6d), which is not possible using a conventional camera where pixels are uniformly sampled at 25 Hz. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Data availability The raw data generated in this study have been deposited in the Zenodo database under accession code https://zenodo.org/records/ 10826791. The PE-CMOS image sensor prototypes are available upon request. We will provide all the information required to replicate the design. However, the design documents use proprietary circuit models and ﬁles from the commercial foundry that are protected under the NDA. We can share the design with the interested party after they have secured the same NDA with XFAB. To interface with the chip, we used open-source hardware, ﬁrmware and software API from the Open- Ephys ONIX16 project and Bonsai16. Code availability The script for generating the ﬁgures have been deposited in the public repository: https://github.com/jz0229/PE-CMOS. References 1. Kim, T. H. & Schnitzer, M. J. Fluorescence imaging of large-scale neural ensemble dynamics. Cell 185, 9–41 (2022). 2. Abdelfattah, A. S. et al. Bright and photostable chemigenetic indi- cators for extended in vivo voltage imaging. Science (1979) https://doi.org/10.1126/science.aav6416 (2019). 3. Villette, V. et al. Ultrafast two-photon imaging of a high-gain voltage indicator in awake behaving mice. Cell 179, 1590–1608.e23 (2019). 4. Piatkevich, K. D. et al. A robotic multidimensional directed evolution approach applied to ﬂuorescent voltage reporters article. Nat. Chem. Biol. https://doi.org/10.1038/s41589-018-0004-9 (2018). 5. Kannan, M. et al. Dual-polarity voltage imaging of the concurrent dynamics of multiple neuron types. Science (1979) 378, eabm8797 (2022). 6. Fan, L. Z. et al. All-optical physiology resolves a synaptic basis for behavioral timescale plasticity. Cell 186, 543–559.e19 (2023). 7. Kim, S. et al. Optical segmentation-based compressed readout of neuronal voltage dynamics. Preprint at bioRxiv https://doi.org/10. 1101/2023.11.10.566599 (2023). 8. Bowman, A. J., Huang, C., Schnitzer, M. J. & Kasevich, M. A. Wide- ﬁeld ﬂuorescence lifetime imaging of neuron spiking and sub- threshold activity in vivo. Science (1979) 380, 1270–1275 (2023). 9. Eom, M. et al. Statistically unbiased prediction enables accurate denoising of voltage imaging data. Nat. Methods https://doi.org/10. 1038/s41592-023-02005-8 (2023). 10. Platisa, J. et al. High-speed low-light in vivo two-photon voltage imaging of large neuronal populations. Nat. Methods 20, 1095–1103 (2023). 11. Zhang, J., Xiong, T., Tran, T., Chin, S. & Etienne-Cummings, R. Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure. Opt. Express 24, 9013 (2016). 12. Sarhangnejad, N. et al. Dual-tap pipelined-code-memory coded- exposure-pixel CMOS image sensor for multi-exposure single- frame computational imaging. Dig. Tech. Pap. IEEE Int Solid State Circuits Conf. 2019-Febru, 102–104 (2019). 13. Luo, Y. & Mirabbasi, S. A 30-fps 192 × 192 CMOS image sensor with per-frame spatial-temporal coded exposure for compressive focal- stack depth sensing. IEEE J. Solid-State Circuits 57, 1661–1672 (2022). 14. Fossum, E. & Hondongwa, D. B. A review of the pinned photodiode for CCD and CMOS image sensors. IEEE J. Electron Devices Soc. 2, 33–43 (2014). 15. Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T. & Nayar, S. K. Video from a single coded exposure photograph using a learned over-complete dictionary. In Proc. 2011 International Conference on Computer Vision 287–294 (IEEE, 2011). 16. Newman, J. P. et al. A uniﬁed open-source platform for multimodal neural recording and perturbation during naturalistic behavior. Preprint at bioRxiv https://doi.org/10.1101/2023.08.30.554672 (2023). Article https://doi.org/10.1038/s41467-024-48765-5 Nature Communications|        (2024) 15:4480  11 17. Lopes, G. et al. Bonsai: An event-based framework for processing and controlling data streams. Front Neuroinform. https://doi.org/ 10.3389/fninf.2015.00007 (2015). 18. Yang, R., Weber, T. D., Witkowski, E. D., Davison, I. G. & Mertz, J. Neuronal imaging with ultrahigh dynamic range multiphoton microscopy. Sci. Rep. 7, 5817 (2017). 19. Zhang, J. et al. A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography. IEEE Trans. Circuits Syst. I: Regul. Pap. 67, 1803–1814 (2020). 20. Nevian, T., Larkum, M. E., Polsky, A. & Schiller, J. Properties of basal dendrites of layer 5 pyramidal neurons: A direct patch-clamp recording study. Nat. Neurosci. 10, 206–214 (2007). 21. Nguyen, C. et al. Simultaneous voltage and calcium imaging and optogenetic stimulation with high sensitivity and a wide ﬁeld of view. Biomed. Opt. Express 10, 789–806 (2019). 22. Ghosh, K. K. et al. Miniaturized integration of a ﬂuorescence microscope. Nat. Methods 8, 871–878 (2011). 23. Juneau, J. et al. MiniFAST: a sensitive and fast miniaturized micro- scope for in vivo neural recording. Preprint at bioRxiv https://doi. org/10.1101/2020.11.03.367466 (2020). 24. Guo, C. et al. Miniscope-LFOV: a large ﬁeld of view, single cell resolution, miniature microscope for wired and wire-free imaging of neural dynamics in freely behaving animals. Sci. Adv. https://doi. org/10.1101/2021.11.21.469394 (2023). 25. Scherrer, J. R., Lynch, G. F., Zhang, J. J. & Fee, M. S. An optical design enabling lightweight and large ﬁeld-of-view head-mounted microscopes. Nat. Methods https://doi.org/10.1038/s41592-023- 01806-1 (2023). 26. Lee, A. K. & Wilson, M. A. Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 36, 1183–1194 (2002). 27. Newman, J. P. et al. Optogenetic feedback control of neural activity. Elife 4, e07192 (2015). 28. Klapoetke, N. C. et al. Independent optical excitation of distinct neural populations. Nat. Methods 11, 338–346 (2014). Acknowledgements We thank S. Chin and G. Brown for discussion and development of additional sparse devolution algorithms. We thank J. Liu, G.F. Lynch, and A. Khalifa for comments on the manuscript. The chip fabrication was funded by support by the Louis B. Thalheimer Fund for Transla- tional Research, and the NIH (R21 EY028381-01). JZ, ZC and MAW acknowledge additional funding through the NIH (MH118928). ZW acknowledges the Alana Foundation. ESB acknowledges HHMI, Lisa Yang, John Doerr, NIH 1R01MH123977, NIH R01DA029639, NIH R01MH122971, NIH RF1NS113287. Author contributions J.Z., J.P.N., M.A.W. conceptualize the ideas. J.Z., J.P.N. and C.L. con- ducted proof of concept testing. J.Z. designed the image sensor with input from E.F. and R.E.C. J.Z. and JPN implemented the acquisition hardware and ﬁrmware to steam the imaging data. J.Z., J.P.N., Z.W., and Y.Q. conducted the in vitro experiments using cultured neurons. P.F.R. conducted the patch clamp recording experiment with help from T.H. J.Z., W.G. and Z. C. processed the imaging data. All authors contributed to the manuscript. J.Z. wrote the manuscript with input from all authors. M.A.W. and E.S.B. supervised the project. Competing interests The authors declare no competing interests. Additional information Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41467-024-48765-5. Correspondence and requests for materials should be addressed to Jie Zhang. Peer review information Nature Communications thanks the anon- ymous, reviewer(s) for their contribution to the peer review of this work. A peer review ﬁle is available. Reprints and permissions information is available at http://www.nature.com/reprints Publisher’s note Springer Nature remains neutral with regard to jur- isdictional claims in published maps and institutional afﬁliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. © The Author(s) 2024 Article https://doi.org/10.1038/s41467-024-48765-5 Nature Communications|        (2024) 15:4480  12
Title: Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy
Authors: JJie Zhang1,2 , Jonathan Newman1,2, ZeguanWang2,3, Yong Qian2,3,Pedro Feliciano Ramos1,2, WeiGuo1,2, Takato Honda 1,2, Zhe Sage Chen 4,Changyang Linghu 5, Ralph Etienne-Cummings6, Eric Fossum7,Edward Boyden 2,3 & Matthew Wilson 1,2
Publisher: Nature Communications
Date: 27 May 2024 ​
Abstract: High-speed wide-field fluorescence microscopy has the potential to capturebiological processes with exceptional spatiotemporal resolution. However,conventional cameras suffer from low signal-to-noise ratio at high frame rates,limiting their ability to detect faint fluorescent events. Here, we introduce animage sensor where each pixel has individually programmable sampling speedand phase, so that pixels can be arranged to simultaneously sample at highspeed with a high signal-to-noise ratio. In high-speed voltage imaging experiments,our image sensor significantly increases the output signal-to-noise ratiocompared to a low-noise scientific CMOS camera (~2–3 folds). This signal-tonoiseratio gain enables the detection of weak neuronal action potentials andsubthreshold activities missed by the standard scientific CMOS cameras. Our camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions​