Deep learning in single-molecule imaging and analysis: recent advances and prospects

Single-molecule microscopy is advantageous in characterizing heterogeneous dynamics at the molecular level. However, there are several challenges that currently hinder the wide application of single molecule imaging in bio-chemical studies, including how to perform single-molecule measurements efficiently with minimal run-to-run variations, how to analyze weak single-molecule signals efficiently and accurately without the influence of human bias, and how to extract complete information about dynamics of interest from single-molecule data. As a new class of computer algorithms that simulate the human brain to extract data features, deep learning networks excel in task parallelism and model generalization, and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. In this perspective, we will highlight recent advances in the application of deep learning to single-molecule studies, discuss how deep learning has been used to address the challenges in the field as well as the pitfalls of existing applications, and outline the directions for future development.


Introduction
In recent years, driven by the interest to study cellular processes at the molecular level, a variety of single-molecule microscopic methods have been developed, including single-molecule localization microscopy (SMLM), 1 single-particle tracking (SPT), 2 single-molecule uorescence resonance energy transfer (smFRET), 3 single-molecule polarization imaging, 4 etc. By observing and analyzing the behavior of individual molecules directly, including their aggregation states, 5 kinetic characteristics, and conformation changes, 6,7 single molecule imaging can unveil structural and kinetic heterogeneities that are not accessible to conventional ensemble measurements. [8][9][10][11] Due to these unique advantages, single-molecule imaging has also attracted research interest from other elds, including electrochemistry, 12,13 materials science, 14 and pharmaceutical science. 15 Xiaolong Liu received his B. S. degree in Chemistry in 2019 from Harbin Institute of Technology. He is currently pursuing his PhD degree in chemistry at the Institute of Chemistry, Chinese Academy of Sciences (ICCAS), conducting singlemolecule imaging research under the supervision of Dr Xiaohong Fang. His research interests focus on single molecule imaging for membrane protein interaction analysis.
Dr. Yifei Jiang earned a Bachelor's degree in Chemical Physics from the University of Science and Technology of China and a PhD in Physical Chemistry from Clemson University. He did his post-doc with Prof. Daniel Chiu at the University of Washington, focusing on developing uorescent probes and spectroscopy methods for single exosome characterization. He is currently a professor at the Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences (CAS). His current research focuses on superresolution microscopy and its biomedical applications.
However, there are several barriers that hinder the wide application of single-molecule imaging in biochemical studies. Firstly, single-molecule imaging is a generally sensitive and time/labor-consuming process, which requires high stability of the instrument and extensive experience from the researcher. Run-to-run variation increases measurement errors and makes the result hard to interpret. Secondly, a single-molecule signal is oen weak and heterogeneous with various types of dynamics. The event of interest is also convolved with noise and photophysical kinetics, as well as instrument uctuation, which results in highly complex data. 16 Traditional algorithms that assume the data to follow a certain distribution might not work well with single-molecule data. 17 Thirdly, single-molecule imaging typically generates a large amount of data. Its data analysis method requires lots of time/effort from experienced users and the procedures are easily affected by human subjective factors, which affects the accuracy and the consistency of analysis.
Recently, as a new class of computer algorithms that simulate the human brain to extract data features, deep learning has been applied to a wide range of research elds with excellent performance. 18 Deep learning networks excel in task parallelism and model generalization and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. 19 Recent published studies that apply deep learning to single-molecule imaging and analysis have shown that, compared to previous algorithms, deep learning provides superior performances in terms of sensitivity, accuracy, and processing speed.
In this perspective, we will rst introduce the basic principles of single-molecule microscopy, in particular, current challenges in experiment automation and data processing. Then we will review recent advances in deep learning in singlemolecule studies and highlight how deep learning has been used to address the challenges in the eld. Finally, we will conclude with the current stage of deep learning in singlemolecule imaging and data analysis, discuss the pitfalls of the existing applications, and outline the directions for future development. It should be noted that deep-learning-assisted SMLM, including the single molecule localization method, [20][21][22] image reconstruction, 23,24 background estimation, 25 and point spread function (PSF) engineering, [26][27][28][29][30] has received broad attention and been reviewed extensively. 1,31-33 Equally important but oen neglected areas are single-molecule imaging automation and single-molecule feature recognition, which will be the focus of this review.

Basics of single-molecule fluorescence imaging and analysis
Single-molecule microscopy is a powerful tool to characterize structural and dynamic heterogeneities in biological systems, which mainly contains two tasks: imaging and data analysis. In the imaging process, researchers would like to fully preserve the dynamics of interest, which involves measuring the behavior of single molecules with high sensitivity and precision. In the data analysis process, researchers would like to extract as much information from the images as possible, which requires the use of highly specialized and carefully optimized algorithms. In this section, we will briey introduce the basics of singlemolecule uorescence imaging and analysis methods, and discuss the recent progress in this eld.

Single-molecule uorescence imaging
Single-molecule imaging relies on the efficient collection and detection of limited uorescence photons emitted by individual molecules. Under ideal conditions, such as labeled dyes immobilized on a clean coverslip or diffusing in a solution with a low uorescence background, a combination of a high numerical aperture objective and a highly sensitive detector can readily provide the signal-to-noise-ratio (SNR) required to detect single molecules. However, applying single-molecule imaging to more complex systems, such as live cells and thick tissues, places additional requirements on the imaging SNR and imaging depths. 7 In addition, in order to study biological processes that occur on a shorter time and length scales, it is also necessary to further improve the spatial and temporal resolutions of single-molecule imaging.
In general, single-molecule imaging methods enhance the imaging SNR and detection sensitivity by reducing the excitation/detection volumes using different types of uorescence microscopes. For example, total internal reection uorescence microscopy (TIRFM) 34-38 exploits evanescent waves to selectively excite molecules near the interface; confocal microscopy uses pinholes to lter out non-focal uorescence signals; light-sheet uorescence microscopy (LSFM) uses a 2D light sheet to illuminate and image samples in thin slices, etc. Among these methods, TIRFM has very shallow imaging depth and is most suitable for studying lateral structures/dynamics; confocal microscopy, as a point-scanning technique, offers 3D resolution but suffers from low imaging efficiency; LSFM, 39,40 on the other hand, combines wide-eld planar excitation with axial optical-sectioning, which offers a balanced performance between axial resolution and imaging speed. [41][42][43] In addition to the various excitation schemes, singlemolecule detection schemes can also be modied to extend the imaging depth and obtain additional information. For example, PSF engineering methods use conventional epi excitation schemes and modify the shape of the PSF to reect the axial position of the uorophore. By introducing cylindrical optics or phase plates into the detection light path, the conventional Gaussian-like PSF can be transformed into ellipse, double helix, and tetrapod shapes. 44-47 PSF engineering methods offer very good temporal resolution and extend the imaging depth to as deep as 20 mm, which is particularly useful for 3D imaging and particle tracking. 48 Hyperspectral imaging determines the spectra of individual molecules through dispersion of uorescence photons, which can provide information about structure and dynamic heterogeneities. [49][50][51][52] In addition, the emission polarization of uorescent probes can be used to study the orientation and the rotational movements of biomolecules. [53][54][55][56][57][58] The spatial resolution of conventional optical microscopy is limited by the diffraction of light. 59 Depending on the numerical aperture of the objective and imaging wavelength, the lateral and axial resolutions of uorescence microscopy are typically 200-300 nm and 500-600 nm, respectively. Driven by the interest in studying biological structures/processes below the diffraction limit, a variety of methods have been developed to further improve the spatial resolution of uorescence imaging, including single-molecule localization methods, such as stochastic optical reconstruction microscopy (STORM) 60 and photoactivated localization microscopy (PALM), 61 methods that exploit uorophores' non-linear response to excitation, such as stimulated emission depletion microscopy (STED) 62-64 and ground state depletion microscopy (GSD), 65 and postacquisition processing methods, such as super-resolution optical uctuation imaging (SOFI), 66 etc. Among these techniques, SMLM 67 has attracted particular research interest, as it offers high spatial resolution while using relatively simple instrumentation. 66,68,69 By combining SMLM with pointfunction engineering, 3D super-resolution imaging with a lateral and an axial resolution of 5 nm and 10 nm has been demonstrated, which greatly improves the level of detail provided by single-molecule imaging. 70,71 SMLM and its deep learning applications have been extensively reviewed. 1,[31][32][33][72][73][74][75][76][77] Due to the limited space, we will focus on single-molecule imaging and only mention SMLM briey in the review.
Overall, advances in imaging techniques have increased the detection sensitivity, imaging depth, and spatial and temporal resolution of single-molecule imaging. Due to the high sensitivity of the measurement, maintaining focus and minimizing sample dri are crucial for reducing measurement variations and obtaining reliable results. In addition, advanced applications, such as deep particle tracking and 3D single-molecule imaging, require careful optimization and calibration of the instruments. To address these challenges, we will discuss how deep learning has been used to set up single-molecule experiments, optimize imaging conditions, and improve the quality of the results in Sections 4.1 and 4.2.

Single-molecule imaging data analysis
Single-molecule imaging is well-suited to address heterogeneity and characterize unsynchronized sequences of events. Singlemolecule data contain a large amount of information, which can be used to reveal aggregation states, kinetic characteristics, and conformation changes of individual molecules. However, a single-molecule signal is typically weak and highly complex. The dynamics of interest can be easily confused with various types of noises, as well as photo-physical kinetics. Extracting information from single-molecule data is a challenging process that requires specialized algorithms and careful optimization of the analysis procedures.
Localization of single molecules in the images is the rst step of single-molecule data processing. Single-molecule localization allows obtaining basic information such as location, intensity, and orientation of single molecules, which can be used to visualize the subcellular structure and to construct the uorescence intensity and position traces. The conventional approach is to use a two-dimensional Gaussian distribution to t the PSF. Multiple iterations are performed using maximum likelihood estimation (MLE) 78 or nonlinear least squares (NLLS) 79 until the best Gaussian model is found. Such an iterative approach is usually time-consuming. There is also the wavelet segmentation algorithm, which converts the raw data into wave maps and performs single-molecule localization using a wavefront to accelerate the process. 80 Aer initial localization, analysis of the uorescence intensity traces can be used to obtain a variety of valuable information on biomolecule structures and functions (Fig. 1). For example, counting the number of steps in a photobleaching trajectory can be used to determine the single-molecule aggregation state (Fig. 1b), smFRET analysis can be used to study protein interactions ( Fig. 1c), single-molecule recognition through equilibrium Poisson sampling (SiMREPS) can be used to characterize the binding dynamics of bio-molecules (Fig. 1d), etc. 5 For diffusing molecules, the trajectory is constructed by linking localized positions between sequential frames, 81,82 which can be used to characterize the state of single molecules and their interactions with the microenvironment (Fig. 1a). 83,84 Many physical parameters associated with biological processes can be extracted from the analysis of the trajectories, such as total displacement, furthest distance from the starting point, connement ratio, local orientation, directional change, instantaneous velocity, mean curve rate, root mean square displacement (RMSD) and diffusion coefficient (D). 85,86 These parameters reect the state of single molecules and their interactions with the surroundings. For example, molecular diffusion models, such as Brownian motion, directional diffusion, conned diffusion, etc., are extensively used to analyze the interactions of proteins on the membrane. 7 Single-molecule data analysis is a challenging process by traditional methods. On one hand, single-molecule data oen contains a variety of dynamics, and do not follow a certain distribution. On the other hand, single-molecule imaging typically generates a large amount of data, which are easily inuenced by human bias with reduced accuracy and consistency. We will discuss how deep learning algorithms have been used to address these problems and facilitate single-molecule data analysis in Sections 4.3 and 4.4.

Deep learning algorithms
In recent years, with the development of computational hardware and algorithms, tremendous progress has been made in deep learning neural networks (DNNs). 87 Deep learning has been applied in various elds and plays an important role in high-throughput data processing. 18 Here, we will introduce the basic concepts of deep learning and hardware requirements.
The basic units of DNNs are neurons. 18 Each neuron is a simple operator that yields an output from multiple inputs. Multiple neurons in parallel form a layer of neurons, and the output of the neurons in one layer is used as the input of the neurons in the next layer, thus forming a neural network. The number of layers, the number of neurons in each layer, and the weights of neurons are all adjustable parameters in the model. The parameters are determined by learning a large amount of training data. Due to the advantages of task parallelism and model generalization, DNNs can be used to t nonlinear functions and simulate feature extraction functions of the human brain.
Deep neural networks can be divided into two main categories in terms of training methods: 88 supervised learning networks and unsupervised learning networks. Supervised learning feeds the model with already labeled data for training. The output targets of the training data are already known in advance, and the model only needs to iterate continuously so that the objective function converges to minimum error. The advantage of supervised learning is the high accuracy of the trained model. However, the data needs to be labeled in advance, which is difficult for some applications due to the lack of a priori knowledge. In contrast, an unsupervised learning network is a type of learning in which the training data does not need to be labeled in advance and the model automatically nds features and classies all the data. As a result, unsupervised learning performs well in cluster analysis and is able to nd small classes that traditional methods cannot nd.
The most widely used deep neural network is convolutional neural networks (CNNs). 89 CNNs are suitable for processing multidimensional data, such as images, audio signals, etc. CNNs generally consist of several types of network layers: input layer, convolutional layer, activation layer, pooling layer, and fully connected layer. The input layer feeds the raw or preprocessed data into the convolutional neural network. As the core layer in CNNs, the convolutional layer performs a sliding window operation using smaller convolutional kernels to detect different features, which is similar to the receptive eld in a biological visual system. The activation layer converts linear mapping into a nonlinear mapping using a nonlinear activation function, such as a rectied linear activation function or sigmoid function. The pooling layer is a down-sampling layer, sandwiched between successive convolutional layers, used to compress the number of parameters and reduce overtting. In a fully connected layer, all the neurons between two successive layers are interconnected with weights to map the learned features into the sample labeling space.
Recurrent neural networks (RNNs) are oen used for time series data, such as speech signals. 89 The RNN is a recursion of a neural network, which uses the previous output as well as a hidden layer, in which the information about all the past processed elements is preserved, as the input. 90 The memory layer adds a layer of weights with each propagation, which results in a reduced amount of previous information in the later memory (vanishing gradiant). To solve the problem of gradient disappearance, long-short-term memory (LSTM) networks have been developed. 91 By adding a forgetting gate, LSTMs chose which memory to remember or forget and can preserve longterm information. Long-term memory is propagated by linear summation operations so that gradient disappearance does not occur in back propagation. LSTMs have been shown to perform better than conventional RNNs in most problems. 91 A generative adversarial network (GAN), contains two sequential networks: the generative network and the discriminative network. 92 The generative network is used to generate data based on a probability distribution and the discriminative network is used to extract features from the generated data. The two models are trained to promote each other. As a type of GAN, the discriminator-generator network (DGN) uses two bidirectional long-short term memory networks (biLSTMs) as a generator and a discriminator respectively. 93,94 BiLSTMs can access both past and future contexts to improve the prediction results. The discriminator is used to map the input sequence to a hidden state vector, and then the generator recovers the input time sequence from this hidden state vector. The discriminator and generator are jointly trained to optimize the prediction accuracy, thus uncovering the hidden state behind a time series.
Considering that single-molecule imaging data is mainly images and time-series, CNN and RNN-based networks are wellsuited for single-molecule data analysis. In addition, in the case of single-molecule data with unknown features or with features that cannot be labeled, a GAN-based unsupervised network is particularly useful.

Applications of deep learning in single-molecule imaging and analysis
Deep learning has been applied to almost every stage of singlemolecule imaging and analysis, including single-molecule imaging automation, single-molecule localization, uorescence intensity and position trace analysis. By replacing human labor in single-molecule experiments and data processing, deep learning algorithms have reduced run-to-run variations induced by bias and human error, thus improving the accuracy of the measurement and the analysis. In addition, deep learning excels in handling nonlinear functions and extracting weak features that cannot be detected using conventional algorithms.
The application of deep learning in single molecule data analysis includes two stages: model training and experimental data analysis. Model training is time-and power-consuming. Once the model is trained, the analysis of experimental data only takes seconds to minutes. For most deep learning tasks, a personal computer with an appropriate conguration is enough. There are three parts that need to be considered: central processing unit (CPU), graphics processing unit (GPU), and random access memory (RAM). A deep learning model always processes a large amount of data. The performance of the CPU mainly limits the speed of data loading and preprocessing. Most mainstream CPUs can meet the requirements. Most deep learning models are trained on the GPU. An excellent GPU with a memory of no less than 8 GB can accelerate the speed of training, e.g., Nvidia RTX 1080 -3080 series and Titan series. Insufficient RAM will limit the processing speed of the CPU and GPU. The RAM should be larger than the GPU memory. We recommend RAM greater than 32 GB. There are several cloud computing platforms providing free GPUs, which facilitate the use of deep learning and project sharing, e.g., Amazon Web Service (AWS), Microso Azure, and Google Colaboratory. In Table 1, we have listed recent representative applications of deep learning in single-molecule imaging/analysis and summarized the key information, including network type, input/output of the model, training hardware and training time. In this part, we will review these applications in detail and compare the performances of the various deep learning algorithms.

Deep-learning-assisted single-molecule autofocus
A single-molecule data acquisition process requires a high level of instrument stability and a high level of experience from the researcher. On one hand, single-molecule imaging typically takes hours of continuous experiments to acquire enough data for statistical analysis. Focus variation and sample dri during the acquisition greatly affect the measurement accuracy. On the other hand, experimental procedures, such as searching for cells in the desired state or focusing onto the structure of interest, are prone to human error and require extensive practice. Deep learning can improve the performance of singlemolecule imaging by automating the experimental process and reducing the amount of human labor involved.
Autofocus is a very useful function in microscopic imaging. It can quickly nd the focal plane without human judgment and, in addition, prevent samples from defocusing during longtime imaging. Traditional real-time autofocus includes two main types: hardware-based autofocus and image-based autofocus. Hardware-based autofocus relies on an additional sensor that detects the back-reection signal from the coverslip to determine the focus dri and then performs re-focus. Lightley et al. 95 recently improved the working distance of the hardwarebased autofocus system by developing a CNN-based algorithm. A diode laser with a wavelength of 830 nm is focused onto a coverslip. The detector is located on the conjugate plane of the coverslip. The reected NIR laser is detected by using a camera and the spatial distribution of intensity is recorded (Fig. 2a). The shape of the distribution is inuenced by the focal condition. A CNN model is trained with the acquired images of various out-of-focus depths. The off-focus distance can be quickly calculated and corrected by analyzing the distribution shape during the imaging process (Fig. 2b). This method has been applied in SMLM and works well over a range of AE100 mm. The image-based auto-focusing takes a series of images along the Z-axis and determines the off-focus distance by calculating the sharpness of the feature edges.
Henry et al. 97 reported a single-shot focusing method based on deep learning, which relies on one or more off-axis illumination sources to nd the correct focal plane. While the idea of single-shot focusing is compelling, the requirement for an extra illumination source could limit its application in singlemolecule imaging. Li et al. 106 developed a deep learning model for autofocus of LSFM (Fig. 2c). Hundreds of defocused image stacks are acquired, each containing a series of images with various off-focus distances. For every image stack, two defocused images with a known off-focus distance are fed into the network for training. The known defocus distance served as the ground truth. Aer training, this model can determine the off-focus distance according to two defocused images in LSFM. This model has been demonstrated in the imaging of mouse forebrain and pig cochleae samples.
More recently, a new off-line autofocus method has been developed. Luo et al. 96 developed a post-imaging autofocus system called Deep-R based on the GAN. Different levels of outof-focus and in-focus images are used for training. The generative network takes the out-of-focus images as input data and outputs the in-focus images, and then the discriminative network takes the output of the generative network as the input and generates the out-of-focus images. The two networks are trained jointly. The in-focus image generated by the model is compared with the actual in-focus image to lter out the wrong models. Aer the training, with an out-of-focus image as the input, the generative network is able to generate the corresponding in-focus image quickly and accurately.

Deep-learning-assisted single-molecule image acquisition
The data acquisition process of single-molecule imaging is time-consuming and complex. To simplify this process and reduce the possibility of human error, Yasui et al. 98,99 built an automated single-molecule imaging platform AiSIS, based on deep learning. It consists of three key modules: an oil immersion feedback system, an autofocus system, and an automated cell search system. The autofocus system is hardware-based with the assistance of DNNs, which is briey described as follows. An iris is conjugated to the upper surface of the coverslip and the image of the iris is captured by a surface reection interference contrast lter (SRIC). When the image is out-of-focus, the image of the iris is blurred. Pre-acquired infocus and out-of-focus images are used to train the neural network. This neural network is used to determine whether it is out-of-focus and to perform focusing according to the image. Automated search for cells with suitable single-molecule density (1-3 molecules per mm 2 ) is also achieved by using a deep learning method. The model is pre-trained with singlemolecule images of suitable density. Fluorescent spots generated by cell fragments are also excluded by the deep learning model. Combined with their custom-built mechanics, they are able to fully automate the processes of well plate placement, dosing stimulation, and multi-well imaging without the need for human labor, which provides great advantages for highthroughput single-molecule imaging. The system is capable of imaging and analyzing 1600 cells in 96-well plates within one day.
Baddeley developed the Python-Microscopy Environment (PYME) 107 which is an integrated platform for high-throughput SMLM. Deep learning neural network Mask R-CNN is trained to detect nuclei for ROI selection automatically. Mask R-CNN is a exible framework for object instance segmentation, which has been applied in human pose estimation, tumor identication, artifact detection, etc. A dataset BBBC038v1, which contains a large number of segmented nuclei images, is used as training data. The system exploits data compression, distributed storage, and distributed analysis for automatic real-time localization analysis, which massively increases the throughput of SMLM to 10 000 cells a day.

Deep-learning-assisted single molecule uorescence intensity trace analysis
Here, we will focus on three types of single-molecule uorescence intensity trace analysis, which are single-molecule photobleaching step-counting analysis (smPSCA), smFRET, and SiMREPS, and discuss how deep-learning was applied to improve the performance of these methods.
Cellular proteins generally function as multimers, aggregates or protein complexes. SmPSCA has become a common method to count the number of uorescent proteins within a diffraction-limited-spot and determine the stoichiometry and aggregation state of the proteins. In photobleaching trajectories, the dynamics of interest are easily confused with various types of noise and photophysical kinetics, such as photoblinking, which are not accounted for in conventional analysis methods, such as the lter method, 108 threshold method, multiscale product analysis, motion t-test method, and step tting method. 109 Taking the temporal information into account, the hidden Markov model (HMM) can partially eliminate the interference of photoblinking. 110 However, HMM methods show a weak ability to correlate long-term events and require users to preset parameters such as initial states, state numbers and a transition matrix. All the methods above require the input of parameters based on prior knowledge of the biological system as well as the algorithms, which could be challenging for users and might affect the accuracy of the analysis. Xu et al. 16 reported the rst deep learning model to solve these problems in smPSCA, which is referred to as the convolutional and longshort-term memory deep learning neural network (CLDNN) (Fig. 3a). This model consists of both a convolutional layer and LSTM layer. Single-molecule photobleaching traces are used as input data, and the output is the number of steps. The convolutional layer is introduced to accurately extract features of steplike photobleaching events, and the LSTM layer is to remember the previous uorescence intensity for photoblinking identication. Manually labeled experimental data and articially synthesized data are used as the training sets. Once the model is trained, it can analyze a large amount of data quickly without setting any parameters. The CLDNN model effectively removes the interference of photoblinking and noise on bleaching step recognition. Compared to the previously reported algorithms for smPSCA, the CLDNN shows higher accuracy with even over 90% at a low SNR value (SNR ¼ 1.9), and higher computational efficiency with 2-3 orders of faster speed.
The CLDNN is a supervised-learning network. Training data need to be labeled manually, which is oen difficult to realize without human bias. Yuan et al. 100 developed an unsupervised neural network, DGN, which can be used not only for protein stoichiometry determination but also for the kinetic characteristics of protein aggregation state changes in live cells (Fig. 3b). The DGN model consists of two biLSTMs. Each biLSTM consists of two inverse LSTMs. The LSTM is suitable to analyze the change of the aggregation state, in which the previous state affects the prediction of the later data. In the traditional LSTM, the model predicts the current time point according to previous information. However, there is very little reference information for the rst data point, which leads to less accurate prediction. Therefore, a bi-directional LSTM is used so that there is data to refer to for both front and back feature extraction. In order to achieve unsupervised learning, two biLSTMs are used as a generator and discriminator, respectively. The discriminator identies the hidden state behind the input uorescence intensity traces, and then the generator generates uorescence intensity traces using the hidden state sequence from the discriminator. The generator and the discriminator are trained jointly. Aer training, the DGN exhibits excellent accuracy in counting photobleaching steps. At SNR ¼ 1.40, the DGN is able to achieve 79.6% accuracy, while conventional methods such as HMM can only achieve 30.9% accuracy. In addition, the DGN can recover the state path, which allows the dynamic information to be obtained from the analysis of uorescence intensity traces of live cells, including durations of protein association, transition rates during protein interactions and state occupancies of different protein aggregation states. The authors used the model to investigate the TGF-b receptor monomer and dimeric/ oligomeric state change under different conditions. They found that while the ligand TGF-b can drive the balance forward to receptor oligomer formation, disruption of lipid-ras by nystatin can make TGF-b receptor association or disassociation more active, and oligomers are difficult to stably exist.
Wang et al. 101 developed a deep learning convolutional neutral network (DLCNN) to recognize receptor monomers and complexes. When receptors form a complex, multiple uorophores are integrated into a diffraction volume to create an overly bright or abnormal spot, which can be used to identify the complex state. This model was trained with images of single quantum dot (QD) particles and aggregates. Aer training, it can visualize the complex formation of chemokine receptor CXCR4 in real time and reach an accuracy of >98% for identifying monomers and complexes. They also developed deepblinking ngerprint recognition (BFR) for identication of oligomeric states. 102 They labeled the CXCR4 receptor with carbon dots (CDs). According to the different aggregation states of the receptor, CD blinking creates different intensity ngerprints. Deep learning models extract the ngerprints and classify the receptor aggregation states. They demonstrate that the heterogeneous organizations of CXCR4 can be regulated by various stimuli at different degrees. For 42-residue amyloidb peptide (Ab42), it is difficult to probe individual aggregation pathways in a mixture because existing brils grow and new brils appear. A deep neural network (FNet) 111 was developed to split highly overlapping brils into single brils, which enables tracking of the changes of individual brils.
smFRET can be used to analyze protein interactions and achieve highly sensitive detection of targets. Performing smFRET requires preprocessing of images, and extracting, classifying and segmenting smFRET traces. 112 Traditionally, the selection of traces requires a lot of subjective judgment and is time-consuming. The two-color intensity trajectories of smFRET need to match the inverse relationship such that as one falls, the other rises. One of the major advantages of deep learning lies in fast feature recognition. Therefore, classication and analysis of smFRET trajectories using DNNs has been reported. Thomsen et al. 103 developed soware for smFRET data analysis based on DNNs: DeepFRET. This model includes the whole process from image pre-processing, extraction of trajectories, and selection of trajectories, to data analysis. The LSTM is used to learn the temporality of the data and propagate the learned information to the later frames. The introduction of the LSTM eliminates the interference of noise. The model was pre-trained using 150 000 simulated data that included all possible FRET states, inter-state transition probabilities, state dwell time etc. On the real data, the model was able to achieve an accuracy of over 95%, while using 1% of the time required by the traditional method.
SiMREPS uses uorescence to detect the specic binding and dissociation of the labeled molecules with xed targets (Fig. 4a). 113 The binding and dissociation of the molecule are reected as an increase or a decrease in the uorescence intensity (Fig. 4b). The rate constant of dissociation represents the strength of the binding, and the frequency of binding represents the concentration of the diffusing molecules. SiM-REPS requires classication of intensity traces based on resident ("on") time, which is easily inuenced by photobleaching, protein aggregation, and noise variation when analysed using traditional algorithms. Li et al. 104 developed a LSTM-based single-molecule uorescence trace classication model, i.e., an automatic SMFM trace selector (AutoSiM) (Fig. 4c). The authors applied this model to analyze DNA sequences with point mutations. A solution containing a DNA sequence with a pre-known proportion that exhibits point mutations, which randomly bind to complementary strands was immobilized on the surface of a glass plate. The DNA with a specic base mutation shows a shorter residence time in the uorescence state due to the reduced binding between the mutated DNA and the target. When used for the analysis of experimental data, the recognition specicity increased by 4.25 times compared to that by the conventional HMM method. The number of layers of the LSTM is adjusted in this model, with 7 layers for classication and 8 layers for segmenting useful trajectories. The model was trained with real experimental and synthetic FRET data. The FRET data classication network was able to achieve an accuracy of 90.2 AE 0.9%. To extend the applicability of the model, transfer learning was introduced. Only 559 manually analyzed FRET trajectories of a Mn 2+ sensor were used to complete the model training, which took less than 15 min. A classication accuracy of 91% was achieved for the experimental data using the transfer learning model.

Deep-learning-assisted single molecule position trace analysis
The traditional methods for extracting diffusion features are the mean square displacement (MSD) method 114,115 and HMM method. 116 In the MSD analysis, the slope and curvature of the MSD-Dt curve reect the D and diffusion mode of the molecule, respectively. Segmented MSD analysis can be used to detect the change of molecule diffusion states over time. The HMM is commonly used as a machine learning method to extract hidden state changes from noisy time series data. The HMM is typically combined with other parameter estimation methods, such as maximum likelihood estimation, Bayesian estimation, 86 and Rayleigh mixture distribution 17 for model selection. As discussed in the previous section, HMM methods require the input of parameters which are oen difficult to obtain in advance. In addition, both the MSD and the HMM methods assume that there is a clear mathematical relationship between MSD and Dt, which allows the diffusion parameters to be obtained by mathematical tting. However, sometimes, the diffusion model is not known in advance or the dynamics do not follow a simple diffusion pattern, which prevents such methods from accurately extracting information from the position traces. Recently, deep learning has been applied to the analysis of single molecule position traces, which can be classied into two categories, i.e., classication of single-molecule diffusion mode and construction of single-molecule diffusion ngerprints.
Granik et al. 105 used CNNs to classify diffusion trajectories. Three diffusion modes were analyzed: Brownian motion, continuous time random walk (CTRW) and fractional Brownian motion (FBM). FBM and CTRW have similar motion characteristics for shorter trajectories, but they obey different physical laws: FBM is associated with crowded cellular environments, while CTRW motion mainly occurs in trap-containing environments. The neural network was trained with 300 000 trajectories. The accuracy of the model was evaluated using real experiment trajectories. The diffusion of uorescent beads follows FBM in gel networks of different densities, and pure Brownian motion in water and glycerol solutions. The diffusion of proteins across the cell membrane is a combination of FBM and CTRW. Based on 100-step tracks of beads with two sizes, the network can distinguish two different populations, and the mean values are similar to predicted theoretical diffusion coefficients; however, the existence of two populations cannot be distinguished by TAMSD with 100 steps.
Due to the complexity of the cellular environment, singlemolecule diffusion is oen a combination of multiple models and varies over time. So far there is no unied model that can completely describe all the kinetic characteristics of singlemolecule diffusion. To address this problem, deep learning has been recently used to construct diffusion ngerprints for single molecule position traces. Pinholt et al. 117 proposed a singlemolecule diffusion ngerprinting method that integrates 17 single-molecule diffusion characteristics (Fig. 5). This approach creates an exclusive diffusion ngerprint for each type of singlemolecule diffusion, which allows better classication of different diffusion entities. The 17 characteristics include 8 features from HMM estimation: D of the four states and the respective residence times, two features from classical RMSD analysis: the diffusion constants describing irregular diffusion, four features based on trajectory shape: kurtosis, dimension, efficiency, and trappedness, and three features describing the general trend: the average speed, track duration, and MSD parameters. These features are partially overlapped and can be used to distinguish subtle differences between trajectories. A logistic regression classier is used to predict the experimental environment that generates such data. A linear discriminant analysis (LDA) is used to rank the most relevant features. Single molecule diffusion ngerprinting was applied to identify Thermomyces lanuginosus lipase (TLL) and L3 mutants. The mutant and wild type have almost the same catalytic rate. The step length distributions among the single-molecule trajectories were very similar and difficult to differentiate using conventional methods. The analysis of the diffusion ngerprints identied the feature that distinguishes the two enzymes: the residence time of the HMM diffusion state. The L3 mutant diffuses away from the generated product region in larger steps and spends more time in faster states. This allows the L3 mutant to have less end-product inhibition. This is also in agreement with the available experimental results. This approach of diffusion ngerprinting combined with multiple traditional characterization methods provides a more comprehensive understanding of different diffusion patterns. However, when the feature selection is not optimal, the accuracy of the classication is not very high. Replacing the simple logistic regression model with a CNN or LSTM could potentially improve the classication accuracy.
Overall, traditional feature-based methods of single-molecule diffusion analysis assume that the diffusion of particles obeys some basic physical diffusion patterns, which in reality is oen more complex. Analyzing a single feature of diffusion cannot reect the full information of single-molecule motion. Compared to intensity trajectories, the single-molecule diffusion trajectory has higher dimensions and therefore contains more features, which may not obey existing physical laws or models. Deep learning can be used to discover these weak features, achieve a comprehensive description of single-molecule diffusion, and truly establish a single-molecule diffusion ngerprint.

Summary and outlook
In summary, deep learning has performed well in singlemolecule experiment automation and data analysis. Automated data acquisition assisted by deep learning can greatly reduce variations induced by human error and improves the reliability of the measurement. Analysis by deep learning is objective, accurate and fast compared with that by conventional analytical methods. Most conventional analytical methods need to pre-know the mathematical model of variables. However, in reality, biomolecule activities oen do not strictly follow a specic mathematical distribution. By training, a deep learning method can nd the most appropriate, exact and nonlinear function for each variable, which cannot be achieved by conventional algorithms.
There are still some pitfalls to be solved for the application of deep learning in single molecule studies. (1) In order to obtain a good model, a large amount of data is required for training. Acquisition of such data is time-and labor-demanding. (2) Deep learning methods oen suffer from the problem of overtting. A model that learns well on training data may not be able to accurately handle unfamiliar data. Some methods have been developed to mitigate this problem, but it is still a tricky situation. (3) Deep learning is a black box; the distribution of features has no analytical form and steps in the algorithm cannot be correlated to the features, which makes it impossible to get an exact interpretation of the algorithm. (4) Deep learning is not easy to get started with. Performing deep learning requires extensive knowledge of the related algorithms and programming skills. Skilled scientists oen had difficulties tuning parameters and xing bugs, let alone the freshman. These problems limit the application of deep learning in singlemolecule imaging and analysis.
In the future, development of more advanced algorithms will reduce the requirement for training data volume and makes deep learning more user friendly. Construction of an authoritative single-molecule database can facilitate the generalization of deep learning methods. Not only does it help scientists to verify the accuracy of the methods, but it also contributes to building deeplearning models that are applicable to different instruments and experimental conditions. The standardization of instruments allows for the comparison of different research studies. For home-built microscopy systems, scientists should add more imaging parameters such as: SNR, laser power, TIRFM angle, etc. We should develop more convenient deep learning platforms and modularize different deep learning methods so that even inexperienced users can invoke them easily with a mouse click. Cloud computation platforms have dramatically lowered the barrier for deep learning applications, but more efforts are needed. Scientists who use deep learning for single-molecule data processing should share the code and package their models into easy-to-use applications. In addition, adding more single-molecule parameters (polarization, spectrum, phase, etc) to the deep learning model can help it extract less-obvious features with enhanced accuracy. By adding these advantages, deep learning can further improve the performance of single-molecule microscopy in a low SNR environment, providing a truly powerful tool set for biochemical applications. Further development of deep learningaided single molecule imaging should also contribute to clinical studies, including disease diagnoses, pathological investigations, and drug discovery.

Conflicts of interest
There are no conicts to declare.