A novel approach for evaluating the performance of real time quantitative loop-mediated isothermal amplification-based methods

Molecular diagnostic measurements are currently underpinned by the polymerase chain reaction (PCR). There are also a number of alternative nucleic acid amplification technologies, which unlike PCR, work at a single temperature. These ‘isothermal’ methods, reportedly offer potential advantages over PCR such as simplicity, speed and resistance to inhibitors and could also be used for quantitative molecular analysis. However there are currently limited mechanisms to evaluate their quantitative performance, which would assist assay development and study comparisons. This study uses a sexually transmitted infection diagnostic model in combination with an adapted metric termed isothermal doubling time (IDT), akin to PCR efficiency, to compare quantitative PCR and quantitative loop-mediated isothermal amplification (qLAMP) assays, and to quantify the impact of matrix interference. The performance metric described here facilitates the comparison of qLAMP assays that could assist assay development and validation activities.


Introduction
The use of molecular approaches for clinical diagnosis has increased over the past 30 years since the development of PCR [1] resulting in a wide variety of diagnostic applications [2][3][4]. Alternative nucleic acid amplification (NAA) technologies, reviewed by Craw and colleagues [5][6][7], utilising isothermal conditions offer a range of potential advantages over PCR, including speed and simplicity, and lend themselves to near patient and point of care diagnostic testing [8,9]. Loop-mediated isothermal amplification (LAMP) [10] is an example of an isothermal NAA technology that is typically faster than PCR [11][12][13] and reportedly less susceptible to common biological inhibitors [14][15][16][17]. LAMP has been successfully Abbreviations: Cq, quantification cycle; IDT, isothermal doubling time; MIQE, minimum information for the publication of quantitative real-time PCR experiments; NAA, nucleic acid amplification; qLAMP, quantitative loop-mediated amplification; qPCR, quantitative real-time polymerase chain reaction; t d , doubling time; Tt, threshold time.
utilised in the development of a wide range of diagnostic assays [18][19][20]. The application of LAMP to measure template abundance, originally established by real time quantitative PCR (qPCR), has been explored by a variety of laboratories [17,[20][21][22][23][24][25][26]. However, real time quantitative LAMP (qLAMP) is a relatively immature technology compared with qPCR and demonstrates poor quantitation capabilities below 1000 target copies [22,23,26]. Threshold time (T t ) [26] is a real-time quantitative LAMP measurement analogous to quantification cycle (C q ) [27] and is central to quantifying template abundance using qLAMP. While T t , allows quantification to be performed there is currently no agreed method for expressing assay performance or defining the magnitude of a difference between two results when conducting qLAMP; a role that 'PCR efficiency' has provided since the development of qPCR. Metrics for assay performance are essential to enable assay optimisation, evaluation of the impact of matrices [28][29][30][31] and to facilitate laboratory comparison.
An isothermal performance metric, analogous to PCR efficiency, has been proposed for real-time helicase dependent amplification (HDA) [32] reactions by Goldmeyer and colleagues [33] (2-point approach). We discuss the application of a more comprehensive metric termed isothermal doubling time (IDT) that utilised a standard curve-based approach to provide an equation (based on the slope of the T t against concentration) to define the magnitude of a difference in results and estimate assay performance which also benefits from evaluating the linear dynamic range.
Effective mechanisms for comparing the performance of NAA technologies such as qLAMP to that of established PCR-based approaches is central to the uptake of these technologies and establishing confidence in their diagnostic potential. This paper describes a methodology for assessing quantitative isothermal nucleic acid amplification performance that, similar to PCR efficiency for qPCR [34], can assign a relative value to any qLAMP assay. We investigate the application of this metric to target pathogens and assess the impact of inhibition on assay performance.

Artificial urine matrices
A panel of artificial urine matrices (Table S1) were developed from published work [35] that contained varying levels of urea concentration (500 or 1000 mM urea) which is a known PCR inhibitor [29]. For investigations into matrix impact, reactions were supplemented with 10% (v/v) of the respective artificial matrix.

Clinical purified DNA samples
All initial clinical sample purification and molecular screening activities were performed at University College London Hospitals (UCLH), London, United Kingdom. 24 pre-screened DNA samples with defined M. genitalium and C. trachomatis content (mix of positives and negatives) were provided blind by UCLH for comparative analysis at the approval of the Chair of the Camden and Islington Community Research Ethics Committee.
C. trachomatis testing was performed as part of a routine clinical diagnostic protocol. Testing process overview: cervical swabs, self-taken vaginal swabs or urine samples were collected and transported in 3 or 4 ml of APTIMA transport medium (Gen-Probe Incorporated, San Diego, USA) mixed with urine 1:1 (v/v) for routine C. trachomatis testing. The test sample (400 l) was analysed using the APTIMA CT assay on the TIGRIS ® platform (Gen-Probe Incorporated, San Diego, USA).
For initial M. genitalium analysis, 200 l samples were taken from materials previously tested for C. trachomatis, purified using a BioRobot 9604 automated workstation using the QIAamp ® Virus BioRobot ® 9604 Kit (QIAGEN, Hilden, Germany) and eluted in 50 l elution buffer (QIAGEN, Hilden, Germany). The eluate (7 l, equivalent to 28 l of the original volume) was analysed by a qPCR assay adapted from Jensen et al. [36] incorporating a mouse CMV (mCMV) internal control system [37].

qPCR assay design and conditions
qPCR assays were designed to detect the C. trachomatis cryptic plasmid (GenBank Accession #X07547), and the M. genitalium partially sequenced MgPa gene (GenBank Accession #X91072) from strain M2300. The real-time PCR assays ( Fig. 1, Table 1) were designed using Primer Express 2.0 (Life Technologies Ltd, Paisley, UK) using default design parameters and the sequences screened for homology using the BLASTn algorithm (http://blast.ncbi.nlm.nih.gov/) that revealed no database alignments likely to cause cross reactivity, other than the region of interest. HPLC purified oligonucleotide primers and FAM/TAMRA hydrolysis probes were synthesised by Eurofins Genomics (Ebersberg, Germany). HPLC purified FAM/NFQ TaqMan ® MGB probes were provided by Life Technologies Ltd (Paisley, UK). Preliminary primer/probe optimisation was conducted to determine optimal PCR conditions.
The qPCR reactions (20 l volume) were performed comprising 1× TaqMan ® Universal PCR Master Mix (Life Technologies Ltd, Paisley, UK), 900 nM forward and reverse primers, 200 nM 6-carboxyfluorescein (6-FAM) labelled hydrolysis probe, sample template (10 fold dilution series from ∼5 × 10 4 to ∼50 copies)    ). Water control assays were included in which template was omitted to assess the presence of contamination, all water control assays produced no detectable C q signal.

qLAMP assay design and conditions
qLAMP assays were designed to detect a similar region of the C. trachomatis cryptic plasmid and partial M. genitalium MgPa gene as that designed for the qPCR assays, however, due to design constraints exactly the same genetic region could not be targeted for the cryptic plasmid target ( Fig. 1 for qLAMP/qPCR amplicon comparisons). The qLAMP assays (Fig. 1, Table 1) comprising 4 core primers (F3/B3/FIP/BIP) and 2 loop primers (LF/LB) were designed using PrimerExplorer V4 (Eiken Chemical Co., Ltd., Tokyo, Japan) online software (http://primerexplorer.jp/ elamp4.0.0/index.html) using default design parameters. The sequences were screened for homology using the same approach as for qPCR primers.
The qLAMP reactions (20 l volume) were performed comprising 1× isothermal master mix (OptiGene Limited, Horsham, United Kingdom) containing proprietary intercalating fluorescent dye, 200 nM F3/B3/LF/LB primers, 800 nM FIP/BIP primers, sample template (∼5 × 10 4 to ∼50 copies per reaction) and, where appropriate, 10% (v/v) synthetic matrices (Table S1). All oligonucleotide primers were HPLC purified. Reactions were performed using the Applied Biosystems 7900HT Fast Real-Time PCR System (Life Technologies Ltd, Paisley, UK) under the following thermal cycling conditions: 65 • C/45 min (60 × 45 s cycles), and fluorescence monitored over the SYBR Green I spectral region. Data analysis was performed using Sequence Detection Software (SDS) version 2.4 (Life Technologies Ltd, Paisley, UK) with manual baseline/threshold settings to estimate Threshold Time (T t ). Water control assays were included in which template was omitted to assess the presence of contamination, all water control assays produced no detectable T t signal.

Matrix evaluation
Experiments were conducted to compare the performance of specific qLAMP assays with comparable qPCR assays in the presence of two different matrices (artificial urine matrix containing 50 or 100 mM urea final concentration, Table S1). Three independent experiments were performed per assay/NAA technology and each reaction performed in triplicate.

qLAMP and qPCR performance assessment
Standard curve analysis (log 10 transformed input copy number plotted against C q or T t ) was performed to evaluate the performance of the qPCR and qLAMP assays and enable calculation of PCR efficiencies [34] and isothermal doubling times (IDT = −0.301 m, whereby 0.301 corresponds on the base 10 logarithmic scale to a change in concentration of a factor of 2 and m = slope). The use of real time PCR instrument to undertake isothermal studies necessitates the conversion of the qPCR instrument output (C q ) to T t through the application of a time multiplier (cycle time), which in this experiment consisted of 45s cycles.

Clinical comparison
The 24 pre-screened clinical DNA samples (pathogen status determined by UCLH) comprising 18 positives (7× M. genitalium, 7× C. trachomatis and 2× C. trachomatis and M. genitalium) and 8 negatives, were used to investigate the C. trachomatis and M. genitalium assays using both NAA platforms on real clinical samples. Due to sample limitations single replicates of each clinical sample, comprising 1 l volumes, were interrogated using appropriate qPCR or qLAMP-based assays. This corresponded to 4 l of the original sample volume and was less than the initial sample analysed by UCLH with the M. genitalium qPCR (28 l) or the C. trachomatis Gen-Probe assay (400 l). To simulate how the respective technologies performed with clinical extracts in the presence of inhibition, a repeat measurement was performed in which artificial urine matrix to include 100 mM urea (final reaction concentration) was also added.

Establishing baseline performance
Standard curve-based analyses of the C. trachomatis and M. genitalium qPCR assays ( Fig. 2a and b, Table S2a) show that the assays achieved good linearity and precision (mean R 2 measurements of 1.00), displayed good levels of PCR sensitivity ( Fig. 3a and b) and are capable of repeatedly detecting ∼5 copies of target material under experimental conditions.
Comparable standard curve-based analyses of the qLAMP assays ( Fig. 2c and d, Table S2b) displayed a reduced dynamic range as compared to the qPCR assays which was characterised by a loss of linear response below the 1000 target copies input level. Variable R 2 measurements (0.82 and 0.97) and higher levels of error (Table  S3) for both assays highlight the lower levels of precision associated with qLAMP compared to qPCR. Both assays display good lower levels of detection sensitivity (Fig. 3c and d) and are capable of detecting 5 theoretical copies of target material under ideal conditions, although this was not quantitative below 1000 copies.

Assessing the impact of urea containing matrices
We investigated the impact of artificial matrices containing two concentrations of urea, a known PCR inhibitor [29] and found that urea had a differential impact upon the performance of both NAA technologies typified by a right shift in C q /T t (Fig. 2). 100 mM urea affects the cryptic plasmid qPCR assay by causing a positive shift in C q for the 50,000 copies sample from 23.50 ± 0.09 to 26.89 ±0.83 C q while a comparable qLAMP reaction demonstrates a T t shift from 387.24 ± 5.14 to 726.00 ± 12.95 s. While the presence of inhibitors produced an appreciable delay in qLAMP amplification, both assays were able to routinely detect ∼5 copies of template regardless of the inhibition levels ( Fig. 3c and d). In comparison, the qPCR assays were only appreciably affected by the presence of 100 mM urea which reduces the detection capabilities of both assays to above 500 template copies ( Fig. 3a and b).

Comparing NAA assay performance
Initial evaluation work demonstrated that the NAA technologies displayed different performance characteristics and responses to matrices. PCR efficiency and IDT metrics were used to compare intra-assay performance and investigate the impact of an inhibitor on qLAMP and qPCR assay performance. The PCR efficiency and IDT data correlates well with the initial matrix assessment work utilising performance metrics as characterised by increased C t /T t and lower PCR efficiency/higher IDT in the presence of inhibitors. PCR efficiency and IDT data suggested that the qPCR assays (Fig. 4a) were less affected by the presence of 50 mM urea than the comparable qLAMP assays (Fig. 4b). Reductions in PCR efficiencies (below 50%) were only observed with the 100 mM urea matrix samples. Matrices containing 100 mM urea had a pronounced impact on the performance of both qPCR assays characterised by a loss of assay sensitivity (Fig. 3a and b), increased C q and reduced efficiency (Fig. 4a).
In comparison, qLAMP assays show a minimum ∼1.5× increase in IDT that is unaffected by urea concentration (Fig. 4b) and suggests that urea is not responsible for the observed impact on assay performance.

Clinical evaluation
The performance of the two NAA technologies was further evaluated using the M. genitalium and C. trachomatis qLAMP/qPCR assays with clinical samples. 24 Clinical DNA extracts with defined M. genitalium and C. trachomatis status were evaluated by all assays in the presence/absence of 100 mM urea. A detailed assessment of diagnostic efficacy for either technology could not be performed due to sample limitations, however this study clearly demonstrated that both qPCR and qLAMP were able to detect their respective pathogen targets from clinical sample extracts. We demonstrated that the MgPa qPCR assay was more sensitive than the corresponding qLAMP assay (Table 2a) while the cryptic plasmid qLAMP assay demonstrated comparable performance to the cryptic plasmid qPCR assay (Table 2b). With the exception of one sample, the qLAMP assay's detection response (Table 2) was unaffected by the presence of the artificial matrices, while both qPCR assays were always inhibited when 100 mM urea matrix was present.

Discussion
Current molecular diagnostic standardisation activities are driven by international organisations such as the Laboratory Standards Institute (CLSI) [38] and user driven initiatives such as 'Minimum Information for Publication of Quantitative Real-Time PCR Experiments (MIQE) Guidelines' which aim to improve measurement comparability and traceability [27]. Metrics such as PCR efficiency, aid qPCR measurement and ultimately reproducibility but comparable tools available to evaluate qLAMP, and other isothermal assays, are limited. To this end, this study developed a performance metric (IDT) and compared it to the application of equivalent qPCR metrics.
Differences in performance were observed between the two NAA technologies, with qPCR displaying better sensitivity and quantitative precision when compared to qLAMP. However, the qualitative potential of qLAMP was not affected by the presence of artificial urine-type matrices containing urea, which represents an inhibitor found in urine-based clinical samples/extracts. In addition, this finding supports the applicability of LAMP to the direct testing of clinical urine samples without the need for timeconsuming extraction steps [12,16].
While our findings support the general assumption that qualitative LAMP amplification may be less susceptible to inhibitors  than qPCR, as others have observed [14,39], we have demonstrated that quantification using qLAMP is susceptible to matrix interference. Our data suggests that LAMP-based assays are susceptible to alternative inhibitory processes than those observed for qPCR as characterised by an increase in Isothermal Doubling Time (IDT) and T t shift that is not dependent upon urea concentration. These results support our previous findings [17] which demonstrated that inhibitors can exhibit different levels of inhibition on different nucleic acid amplification methods and Tani et al. [14] who observed that urea (up to 400 mM) has little effect on the quantitative performance of qLAMP, indicating that other matrix components are responsible for the observed inhibitory effect. The observed gross change in IDT would affect any quantitative measurements and require appropriate correction mechanisms such as matched calibrator and test matrices.
Performance comparison was achievable through the application of the commonly used PCR efficiency metric and our adaptation of the doubling time (t d ) [33] approach that we have termed IDT. IDT provides a mechanism by which a fold difference can be attributed to multiple target dilution points rather than two points as utilised to calculate t d . The standard curve-based approach provides an estimate of qLAMP assay performance that minimises biases associated with dual-point estimations. PCR is only able to perform a maximum of a doubling of all templates per cycle when it is at its optimum and thus PCR efficiency is reported as a percentage. qLAMP does not have the same cyclical constraint as PCR and thus IDT does not have a clearly defined physical maximum. IDT has the potential to be used for other quantitative isothermal technologies as long as there is a log linear relationship between T t and concentration such as observed with real-time nucleic acid sequence-based amplification (NASBA) and recombinase polymerase amplification (RPA) approaches [40,41].
IDT also provides users with a common baseline measurement to compare the performance of compatible isothermal NAA assays and support assay development activities such as optimisation or investigating the impact of inhibitors. Inhibition was demonstrated to affect the qLAMP and qPCR reactions through an increase in T t or C q respectively, as well as a reduction in PCR efficiency or increase in IDT.
IDT can be estimated using standard curves, as performed here, or to assess sample specific performance in the same way as qPCR when performing relative quantification [42]. This not only facilitates comparison between assays within the same laboratory, but also between laboratories and enables reporting of a key assay characteristic similar to qPCR efficiency; which is agreed to be an important factor that should be stated when performing qPCR [27].
This study highlighted the reduced precision associated with qLAMP when compared to qPCR (Table S3) and suggests further work on assay components such as enzymes and buffer systems is necessary to improve qLAMP performance. However, it should be remembered that qPCR has benefitted from over a decade of continuous development that contrasts with the relative immaturity of the LAMP.
LAMP has the potential to complement existing qPCR based techniques that are currently the gold standard within the molecular diagnostics field, but is limited by poor quantitation below 1000 target copies. This could be alleviated through the use of approaches similar to digital PCR, subject to sensitivity issues arising from current assayable sample volume limitations [17], or through improvements in LAMP chemistry, e.g. enzyme optimisation [43]. The metric proposed by this study provides a mechanism that can facilitate such research and development work and ultimately assist in the transfer of novel LAMP-based methods into diagnostic tests.

Conclusion
We demonstrate the application of a performance metric comparable to PCR efficiency in the evaluation of qLAMP assays which is potentially applicable to other isothermal NAA technologies. The isothermal doubling time metric described here facilitates the comparison of qLAMP assays that could assist assay development and validation activities within the developer and user communities.