Warfarin dosing algorithms: A systematic review
INTRODUCTION Warfarin remains the most commonly prescribed oral anticoagulant for the management of thromboembolic disorders. However, dosing remains challenging due to warfarin's narrow therapeutic index and highly variable clinical response. These dosing challenges usually result in a high frequency of adverse effects (thrombosis and bleeding) as well as an increased burden to the patient (e.g. more frequent monitoring), which could impact quality of life and lead to treatment discontinuation of an otherwise highly efficacious drug. To better predict an individual's warfarin dose requirements, numerous dose‐prediction algorithms based on demographic, clinical, environment and genetic factors have been developed. , Algorithms incorporating only demographic, clinical and environmental factors have been designated as clinical and those additionally incorporating genetic factors as pharmacogenetic . The availability of numerous published dosing algorithms provides a wealth of information but makes navigating the literature to identify which algorithms to use or recommend, and in which patients/populations difficult. This task becomes more complicated if it is not clear if the identified algorithms have low risks of bias, have been externally validated and/or have been evaluated for clinical utility. The Clinical Pharmacogenetics Implementation Consortium (CPIC) has recommended the use of 4 dosing algorithms. However, these may not be applicable to non‐Caucasians or for some categories of patients such as those with international normalized ratio (INR) targets outside of the 2–3 range. , , Previous reviews have attempted to describe available dosing algorithms but most, if not all, have had the main limitation of focusing on a limited number of algorithms. For example Saffian et al . evaluated 16 algorithms but these were only from studies that had: (i) evaluated the algorithm's predictive ability in the form of a high resolution scatterplot (observed vs predicted maintenance doses); (ii) used a validation dataset; and (iii) included at least 5 patients requiring warfarin at a dose >7 mg/d. Other reviews including those by Verhoef et al . (32 algorithms) and Shendre et al . (50 eligible studies) have been narrative in nature and consequently neither reported a detailed search strategy nor assessed the risk of bias of included studies. Additionally, both these narrative reviews included only pharmacogenetic algorithms and the Shendre et al . review additionally excluded studies that did not involve European or African ancestry populations. To methodologically assess and describe the knowledge base accumulated so far, we have undertaken this systematic review, which includes both clinical and pharmacogenetic algorithms with no population‐based exclusion criteria. Our aim was to accurately and comprehensively summarize which algorithms are available for which populations and the covariates (demographic, clinical, environmental, genetic), performances and the risk of bias of these algorithms. 

METHODS 2.1 Search strategy and selection criteria A predefined protocol (PROSPERO: CRD42019147995), based on the principles set in the CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) checklist, and PROBAST (Prediction model Risk Of Bias Assessment Tool), a tool meant to assess the risk of bias and applicability of prediction model studies was followed. This report adheres to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) statement (Table ). MEDLINE records (from 1946 to 22 August 2019) were searched using medical subject headings (MeSH terms) and text words related to “warfarin”, “algorithm” and “dosing” (Table ). A second MEDLINE search was conducted on 20 May 2020 to identify records published after our first search. Lists of references from the identified studies were hand‐searched to identify further eligible studies. Non‐English language studies were excluded. Observational (e.g. cohort studies) and interventional (e.g. randomised controlled trials) studies that developed, validated or assessed the impact/clinical utility of warfarin dosing algorithms in any warfarin‐treated population were included. For development studies, those that modelled at least 2 predictor variables (not counting dose and INR readings for dose‐revision algorithms) and either: (i) explicitly stated in their aims that they were developing a dosing algorithm; or (ii) reported dosing equations, nomograms, charts, tables, computer programs etc. that can be used to provide a daily or weekly dose, were included. In all cases, the included predictors had to have been reported. Unless enough details pertaining to algorithm development were available in an external validation or clinical utility assessment study or elsewhere (corresponding development study, other studies), external validation or clinical utility assessment studies were excluded. Additionally, for a clinical utility assessment study to be included, a comparison between a dosing algorithm with an alternative strategy (such as existing clinical practice) was a prerequisite. For the purposes of this review, clinical utility was defined as the demonstration that a dosing algorithm improved the quality of anticoagulation (based on the time spent in the therapeutic INR range) or lead to better clinical endpoints (such as fewer bleeding episodes). Not to be confused with the outcome to be predicted in the individual studies (i.e. the stable warfarin dose), the primary outcome of interest in this review was the warfarin dose‐prediction algorithm developed, and whether it was externally validated or evaluated for clinical utility in the included studies. 2.2 Data extraction and quality assessment One reviewer (I.G.A.) screened titles and abstracts of the retrieved bibliographic records for eligibility. For all stages, a second reviewer (R.O.) independently checked a random 10% of the records to check for consistency. Disagreements were resolved by consensus and because the first reviewer was consistent with regard to following agreed upon criteria, only the first reviewer continued reviewing the remaining records. A data extraction form was adapted from the CHARMS and PROBAST tools, piloted in a subset of randomly selected included papers and used to extract relevant information related to participants, predictors, outcome, analysis and results. When a single publication reported both development and external validation studies (and/or clinical utility assessments), or multiple algorithms, each study/algorithm was assessed separately. The exception was studies that reported the warfarindosing.com platform—although this platform incorporates multiple algorithms, it was not possible to separate the individual algorithms and so it was considered as 2, the clinical and pharmacogenetic Gage algorithms. Algorithm updating/extension studies in which new predictors were added to existing algorithms were considered as new algorithm development studies. To assess the methodological quality of each included development or external validation study, the 2 reviewers used the PROBAST tool. Although this tool focuses on prediction models that consider binary or time‐to‐event outcomes and studies that use generalized linear modelling, its authors encourage its use in studies that consider other outcomes and other machine learning techniques such as those explored in this review. It should, however, be tailored to these other outcomes/techniques as we did in Tables and Figure . For reasons detailed in Table , emphasis was placed on the assessment of the risk of bias in the analysis domain. We did not assess the methodological quality (and performance) of clinical utility (impact) assessment studies since these have been previously explored in several systematic reviews and meta‐analyses. , , , , , , , 2.3 Data synthesis This systematic review was qualitative in nature and no attempt to quantitatively synthesise studies by way of meta‐analysis was conducted. Consequently, heterogeneity measures and publication bias were not explored. The descriptive results (i.e. proportions and measures of central tendency and dispersion) are presented in structured tables, graphically and as a narrative summary. Where appropriate, the results were stratified according to the type of algorithm (clinical vs pharmacogenetic) and ethnic populations for which they were developed. For the purposes of stratifying by population, we used the 4 categories (White, Asian, Black and Mixed/Other) reported by 1 of the largest warfarin‐related studies to date, the International Warfarin Pharmacogenetics Consortium (IWPC) study. Where these race categories were unreported, country was used as a proxy (for example populations from China were categorized as Asian while populations from northern Europe as White). Regarding which algorithm would be relevant to a given population, we arbitrarily chose a 5% cut‐off, i.e. an algorithm that recruited at least 5% of a given population would be applicable to that population. These descriptive analyses were conducted in R (version 3.6.1). No sensitivity analyses were conducted. 2.4 Nomenclature of targets and ligands Key protein targets and ligands in this article are hyperlinked to corresponding entries in http://www.guidetopharmacology.org , the common portal for data from the IUPHAR/BPS Guide to PHARMACOLOGY. 

Search strategy and selection criteria A predefined protocol (PROSPERO: CRD42019147995), based on the principles set in the CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) checklist, and PROBAST (Prediction model Risk Of Bias Assessment Tool), a tool meant to assess the risk of bias and applicability of prediction model studies was followed. This report adheres to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) statement (Table ). MEDLINE records (from 1946 to 22 August 2019) were searched using medical subject headings (MeSH terms) and text words related to “warfarin”, “algorithm” and “dosing” (Table ). A second MEDLINE search was conducted on 20 May 2020 to identify records published after our first search. Lists of references from the identified studies were hand‐searched to identify further eligible studies. Non‐English language studies were excluded. Observational (e.g. cohort studies) and interventional (e.g. randomised controlled trials) studies that developed, validated or assessed the impact/clinical utility of warfarin dosing algorithms in any warfarin‐treated population were included. For development studies, those that modelled at least 2 predictor variables (not counting dose and INR readings for dose‐revision algorithms) and either: (i) explicitly stated in their aims that they were developing a dosing algorithm; or (ii) reported dosing equations, nomograms, charts, tables, computer programs etc. that can be used to provide a daily or weekly dose, were included. In all cases, the included predictors had to have been reported. Unless enough details pertaining to algorithm development were available in an external validation or clinical utility assessment study or elsewhere (corresponding development study, other studies), external validation or clinical utility assessment studies were excluded. Additionally, for a clinical utility assessment study to be included, a comparison between a dosing algorithm with an alternative strategy (such as existing clinical practice) was a prerequisite. For the purposes of this review, clinical utility was defined as the demonstration that a dosing algorithm improved the quality of anticoagulation (based on the time spent in the therapeutic INR range) or lead to better clinical endpoints (such as fewer bleeding episodes). Not to be confused with the outcome to be predicted in the individual studies (i.e. the stable warfarin dose), the primary outcome of interest in this review was the warfarin dose‐prediction algorithm developed, and whether it was externally validated or evaluated for clinical utility in the included studies. 

Data extraction and quality assessment One reviewer (I.G.A.) screened titles and abstracts of the retrieved bibliographic records for eligibility. For all stages, a second reviewer (R.O.) independently checked a random 10% of the records to check for consistency. Disagreements were resolved by consensus and because the first reviewer was consistent with regard to following agreed upon criteria, only the first reviewer continued reviewing the remaining records. A data extraction form was adapted from the CHARMS and PROBAST tools, piloted in a subset of randomly selected included papers and used to extract relevant information related to participants, predictors, outcome, analysis and results. When a single publication reported both development and external validation studies (and/or clinical utility assessments), or multiple algorithms, each study/algorithm was assessed separately. The exception was studies that reported the warfarindosing.com platform—although this platform incorporates multiple algorithms, it was not possible to separate the individual algorithms and so it was considered as 2, the clinical and pharmacogenetic Gage algorithms. Algorithm updating/extension studies in which new predictors were added to existing algorithms were considered as new algorithm development studies. To assess the methodological quality of each included development or external validation study, the 2 reviewers used the PROBAST tool. Although this tool focuses on prediction models that consider binary or time‐to‐event outcomes and studies that use generalized linear modelling, its authors encourage its use in studies that consider other outcomes and other machine learning techniques such as those explored in this review. It should, however, be tailored to these other outcomes/techniques as we did in Tables and Figure . For reasons detailed in Table , emphasis was placed on the assessment of the risk of bias in the analysis domain. We did not assess the methodological quality (and performance) of clinical utility (impact) assessment studies since these have been previously explored in several systematic reviews and meta‐analyses. , , , , , , , 

Data synthesis This systematic review was qualitative in nature and no attempt to quantitatively synthesise studies by way of meta‐analysis was conducted. Consequently, heterogeneity measures and publication bias were not explored. The descriptive results (i.e. proportions and measures of central tendency and dispersion) are presented in structured tables, graphically and as a narrative summary. Where appropriate, the results were stratified according to the type of algorithm (clinical vs pharmacogenetic) and ethnic populations for which they were developed. For the purposes of stratifying by population, we used the 4 categories (White, Asian, Black and Mixed/Other) reported by 1 of the largest warfarin‐related studies to date, the International Warfarin Pharmacogenetics Consortium (IWPC) study. Where these race categories were unreported, country was used as a proxy (for example populations from China were categorized as Asian while populations from northern Europe as White). Regarding which algorithm would be relevant to a given population, we arbitrarily chose a 5% cut‐off, i.e. an algorithm that recruited at least 5% of a given population would be applicable to that population. These descriptive analyses were conducted in R (version 3.6.1). No sensitivity analyses were conducted. 

Nomenclature of targets and ligands Key protein targets and ligands in this article are hyperlinked to corresponding entries in http://www.guidetopharmacology.org , the common portal for data from the IUPHAR/BPS Guide to PHARMACOLOGY. 

RESULTS We aimed to summarize which algorithms are available for which populations, and the covariates, performances and risk of bias of these algorithms. Figure illustrates the literature search and selection process; 10 035 records were identified of which 9435 were excluded based on the title and abstract. Of 600 full text records assessed for eligibility, 266 met the eligibility criteria and were included in the qualitative synthesis. Of these, 205, 123 and 32 articles, respectively, described algorithm development, external validation and clinical utility assessment. Some articles described both algorithm development and external validation ( n = 74), algorithm development and clinical utility assessment ( n = 10), external validation and clinical utility assessment ( n = 15) while 5 articles reported all 3 components (Figure ). The included articles described the development of 433 dosing algorithms, 481 external validations and 52 clinical utility assessments, whose characteristics are summarised in Table and detailed in Tables (algorithm development), Table (external validations), and Table (clinical utility assessments). Out of all date ranges investigated, the period during which most algorithm developments/evaluations were published was 2011–2015 in which 175 (41%), 224 (47%) and 18 (35%) algorithms were developed, externally validated and assessed for clinical utility, respectively (Table , Figure ). The median sample sizes for these studies were 229 (range 18–10 673), 125 (28–2181) and 234 (10–2343), respectively. Children were studied less often—algorithm developments, external validations and clinical utility assessments in children were 11 (3%), 26 (5%) and 3 (6%), respectively. Asia had the highest number of algorithm developments ( n = 175, 40%) and external validations ( n = 208, 43%), while North America had the highest rate of clinical utility assessments ( n = 25, 48%). Most of the developed algorithms included both clinical and genetic covariates ( n = 344, 79%), were mostly for dose‐initiation ( n = 373, 86%), were developed using multiple linear regression ( n = 280, 65%) and presented a regression formula that could be used to compute a weekly or daily dose ( n = 239, 55%). Of the developed algorithms, 111 (26%) and 30 (7%) algorithms were respectively externally validated or assessed for clinical utility at least once (Table ). The 5 most externally validated algorithms were all dose‐initiation pharmacogenetic algorithms and included those by the IWPC ( n = 72 external validations), Gage ( n = 46), Sconce ( n = 32), Wadelius ( n = 20) and Huang ( n = 19) while the 4 most clinically assessed algorithms were the Gage pharmacogenetic algorithm ( n = 8 clinical utility assessments), IWPC pharmacogenetic algorithm ( n = 7), Gage clinical algorithm ( n = 5) and Lenzini dose revision pharmacogenetic algorithm ( n = 4). Consequently, most external validations were conducted on pharmacogenetic ( n = 432 external validations, 90%) and dose initiation ( n = 443, 92%) algorithms, algorithms developed using multiple linear regression ( n = 458, 95%) and those that presented a regression formula ( n = 453, 94%). A similar trend was observed for the clinical utility assessments (Table ). 3.1 Study populations Of the 433 developed algorithms, 186 (43%), 210 (49%), 121 (28%) and 77 (18%) used datasets that included at least 5% White, 5% Asian, 5% Black and 5% Mixed/Other participants, respectively (Table ). For the studies including at least 5% White participants, the median percentage of White participants was 90% (range 33–100%; Table ). Corresponding values for Asian, Black and Mixed/Other cohorts were 100% (7–100%; Table ), 14% (5–100%; Table ) and 71% (5–100%; Table ), respectively. Other characteristics stratified by populations are detailed in Tables . For the developed algorithms that included at least 5% White participants in their datasets, 54 (29%) and 20 (11%) algorithms were respectively externally validated and assessed for clinical utility at least once (Table ). The corresponding values were 55 (26%) and 10 (5%) for Asians (Table ), 34 (28%) and 10 (8%) for Blacks (Table ), and 14 (18%) and 2 (3%) for Mixed/Other (Table ). It is important to note that these algorithms were not always validated in populations for whom they were developed. 3.2 Predictors During algorithm development, all 433 algorithms explored demographic, clinical, and environmental predictors and for these the median number of predictors included in the final algorithms was 5 (range 0–23). Conversely, only 346 algorithms explored genetic factors with the median number of genetic predictors included in the final algorithms being 3 (range 1–205). The predictors included in at least 10 algorithms are shown in Figure . Age (included in 401 algorithms), concomitant medications (270 algorithms, amiodarone in 201 algorithms), weight (229 algorithms) and sex (141 algorithms) were the 4 most common demographic/clinical/environmental predictors. Comorbidities were included in 100 algorithms and these included renal disease (42 algorithms), hepatic disease (40 algorithms), hypertension (27 algorithms) and diabetes mellitus (26 algorithms). The genes most frequently included in the pharmacogenetic algorithms were CYP2C9 (329 algorithms), VKORC1 (319 algorithms), CYP4F2 (92 algorithms) and APOE (11 algorithms). CYP2C9 variants included CYP2C9*2 (in 206 algorithms), CYP2C9*3 (316 algorithms) and other variants (64 algorithms), while VKORC1 variants included VKORC1‐1639G > A (270 algorithms), VKORC1 1173C > T , (75 algorithms), VKORC1 3730G > A (20 algorithms) and a number of others (34 algorithms). Figure shows numbers stratified by the 4 population categories. Age remained the most included demographic/clinical/environmental predictor in each of the 4 categories, while CYP2C9 remained the most common predictor in Whites, Blacks and the Mixed/Other population categories, but was overtaken by VKORC1 in Asians. The Asian algorithms that included the CYP2C9 gene ( n = 153) mostly focused on the CYP2C9*3 variant ( n = 149, 97%) as opposed to the CYP2C9*2 ( n = 44, 29%) or other CYP2C9 ( n = 7, 5%) variants. The corresponding proportions for inclusion of CYP2C9*3 vs CYP2C9*2 vs other CYP2C9 variants, respectively, were 99 vs 92 vs 19% (Whites, n = 153 algorithms), 99 vs 96 vs 36% (Blacks, n = 99) and 88 vs 95 vs 30% (Mixed/Other, n = 64). 3.3 Predictive performance A fit accuracy measure (the coefficient of determination, R 2 ) was the most commonly reported performance measure during both algorithm development (323 [75% of 433]) and external validation (261 [54% of 481]; Table ). The R 2 value represents the proportion of total interpatient variability in warfarin dose requirements that can be jointly accounted for by the variables included in an algorithm. For algorithm development, the median variability in warfarin dose explained by included predictors was 43% (range 2–96%). This was higher (median 45%, range 8–96%) when only pharmacogenetic algorithms were considered and lower (median 20%, range 2–83%) when only clinical factors were considered. The CYP2C9 and VKORC1 genes, respectively, accounted for a median of 7% (<1–50%) and median of 25% (1–59%) of the variability in warfarin dose requirements. The R 2 performance stratified by race is shown in Table . A consideration of the race‐specific proportions in each stratified analysis (Tables ) should be made when interpreting the race‐stratified performances. For example, for 24 studies that included at least 5% Black patients, the proportion of warfarin dose variability that can be attributed to VKORC1 is 23%. However, these 24 studies on average included a median of only 13% (range 5–100%) Black patients. When only the 3 studies that included only Black patients are considered, the median VKORC1 partial R 2 becomes 9% (range 7–10%). These partial R 2 values should also be cautiously interpreted since different computation approaches yield different results (Figure ). Regarding the precision (predictive accuracy) and bias measures, the most reported measures were the mean absolute and mean prediction errors, respectively, being reported 137 (32%) and 17 (4%) times (algorithm development) and 222 (46%) and 144 (31%) times (external validations). The median mean absolute errors for the algorithm development and external validations were respectively 1.23 and 1.20 mg/d, with Asians (0.70 and 0.96 mg/d) and Whites (1.29 and 1.30 mg/d) having lower values when compared to Blacks (1.55 and 1.39 mg/d). The preferred precision and bias measures (Table ) were underreported, each being reported only once and thrice for algorithm development and external validation, respectively. Clinical relevance and other fit accuracy, precision and bias measures are detailed in Tables and . Because most studies reported R 2 (a fit accuracy measure), we carried out a posthoc correlation analysis and included the studies that reported both the R 2 and a precision accuracy measure to determine whether R 2 could be a good proxy of predictive accuracy. For this purpose, we used the mean absolute error as the predictive accuracy measure because it was the most reported (its limitations as a predictive accuracy measure (Table ) not withstanding). A total of 216 algorithm developments and external validations reported both these 2 measures, with lower mean absolute errors being moderately associated with higher R 2 values (Pearson's product–moment correlation coefficient −0.390 [95% confidence intervals −0.494 to −0.274], Figure ). Tables and include the performance measures stratified according to whether algorithms were clinical or pharmacogenetic (direct comparisons are available for some algorithm pairs in Tables and ). To directly compare the performances of algorithms stratified according to the modelling technique and time of application (dose initiation or dose revision), we summarized the studies that, using the same dataset, included at least 2 algorithms that differed in these 2 characteristics. As expected, dose revision algorithms generally performed better than dose initiation algorithms (Table ). Multiple linear regression performed comparable to or even better than many other machine learning techniques (Table ). Although pharmacokinetic/pharmacodynamic algorithms (fitted using nonlinear mixed effect modelling) performed better than other algorithms, this is mainly attributable to their dose revision aspects (i.e. when used for dose initiation, performance was comparable). However, the numbers of direct comparisons were few, and the performance metrics used were probably suboptimal (Table ). 3.4 Risk of bias We focused on the assessment of the risk of bias in the analysis domain (Tables , , and ). During algorithm development, most developments had the number of participants per candidate predictor variable ≥20 ( n = 203, 47%), did not provide information on the handling of continuous and categorical predictors ( n = 291, 67%), probably included all enrolled participants in the analysis ( n = 229, 53%) and did not provide information on the handling of participants with missing data ( n = 233, 54%; Table ). Additionally, many algorithm developments relied on univariable ( n = 204, 47%) and multivariable ( n = 208, 48%) analysis during predictor selection, did not appropriately evaluate algorithm performance ( n = 232, 54%), did not account for model overfitting and optimism in algorithm performance ( n = 300, 69%), and did not provide enough information to assess whether predictors and their assigned weights in the final algorithms corresponded to the results reported in the multivariable analysis ( n = 220, 51%). Consequently, only 1 (<1%) algorithm was rated as having a low risk of bias (unclear n = 26, 6%; high n = 406, 94%). By contrast, most external validations included at least 100 participants with stable dose ( n = 329, 68%), all probably appropriately handled continuous and categorical predictors ( n = 481, 100%), mostly analysed all enrolled participants ( n = 309, 64%) although many did not provide enough information on the handling of participants with missing data ( n = 293, 61%). Most ( n = 273, 57%) reported the precision measures we considered appropriate for this review, although only 8 (<2%) were rated as having a low risk of bias (unclear n = 97, 20%; high n = 376, 78%). Although we did not focus on the risk of bias in the participant, predictors and outcome domains, the key risk of bias concerns in these domains are reported in Tables (algorithm development) and S7 (external validation). Of note, despite large variability, most algorithm developments ( n = 386, 89%) and external validations ( n = 433, 90%) provided stable dose definitions and/or referenced previous publications in which the same were provided. Lastly, for algorithm development, we also explored whether the outcome of stable dose was transformed during analysis since outcome transformation may affect the bias of the algorithm (Asiimwe, unpublished data). Most algorithm developments ( n = 228, 53%) did not incorporate any form of transformation, while 102 (24%) and 85 (20%) algorithm developments, respectively, incorporated logarithmic and square root transformations for reasons detailed in Table (for 18 [4%] algorithms, information was not available). The most common reason provided was to normalize dose and limit heteroscedasticity ( n = 120, 64%) with only 3 (2%) studies reporting clinical considerations as justification. 

Study populations Of the 433 developed algorithms, 186 (43%), 210 (49%), 121 (28%) and 77 (18%) used datasets that included at least 5% White, 5% Asian, 5% Black and 5% Mixed/Other participants, respectively (Table ). For the studies including at least 5% White participants, the median percentage of White participants was 90% (range 33–100%; Table ). Corresponding values for Asian, Black and Mixed/Other cohorts were 100% (7–100%; Table ), 14% (5–100%; Table ) and 71% (5–100%; Table ), respectively. Other characteristics stratified by populations are detailed in Tables . For the developed algorithms that included at least 5% White participants in their datasets, 54 (29%) and 20 (11%) algorithms were respectively externally validated and assessed for clinical utility at least once (Table ). The corresponding values were 55 (26%) and 10 (5%) for Asians (Table ), 34 (28%) and 10 (8%) for Blacks (Table ), and 14 (18%) and 2 (3%) for Mixed/Other (Table ). It is important to note that these algorithms were not always validated in populations for whom they were developed. 

Predictors During algorithm development, all 433 algorithms explored demographic, clinical, and environmental predictors and for these the median number of predictors included in the final algorithms was 5 (range 0–23). Conversely, only 346 algorithms explored genetic factors with the median number of genetic predictors included in the final algorithms being 3 (range 1–205). The predictors included in at least 10 algorithms are shown in Figure . Age (included in 401 algorithms), concomitant medications (270 algorithms, amiodarone in 201 algorithms), weight (229 algorithms) and sex (141 algorithms) were the 4 most common demographic/clinical/environmental predictors. Comorbidities were included in 100 algorithms and these included renal disease (42 algorithms), hepatic disease (40 algorithms), hypertension (27 algorithms) and diabetes mellitus (26 algorithms). The genes most frequently included in the pharmacogenetic algorithms were CYP2C9 (329 algorithms), VKORC1 (319 algorithms), CYP4F2 (92 algorithms) and APOE (11 algorithms). CYP2C9 variants included CYP2C9*2 (in 206 algorithms), CYP2C9*3 (316 algorithms) and other variants (64 algorithms), while VKORC1 variants included VKORC1‐1639G > A (270 algorithms), VKORC1 1173C > T , (75 algorithms), VKORC1 3730G > A (20 algorithms) and a number of others (34 algorithms). Figure shows numbers stratified by the 4 population categories. Age remained the most included demographic/clinical/environmental predictor in each of the 4 categories, while CYP2C9 remained the most common predictor in Whites, Blacks and the Mixed/Other population categories, but was overtaken by VKORC1 in Asians. The Asian algorithms that included the CYP2C9 gene ( n = 153) mostly focused on the CYP2C9*3 variant ( n = 149, 97%) as opposed to the CYP2C9*2 ( n = 44, 29%) or other CYP2C9 ( n = 7, 5%) variants. The corresponding proportions for inclusion of CYP2C9*3 vs CYP2C9*2 vs other CYP2C9 variants, respectively, were 99 vs 92 vs 19% (Whites, n = 153 algorithms), 99 vs 96 vs 36% (Blacks, n = 99) and 88 vs 95 vs 30% (Mixed/Other, n = 64). 

Predictive performance A fit accuracy measure (the coefficient of determination, R 2 ) was the most commonly reported performance measure during both algorithm development (323 [75% of 433]) and external validation (261 [54% of 481]; Table ). The R 2 value represents the proportion of total interpatient variability in warfarin dose requirements that can be jointly accounted for by the variables included in an algorithm. For algorithm development, the median variability in warfarin dose explained by included predictors was 43% (range 2–96%). This was higher (median 45%, range 8–96%) when only pharmacogenetic algorithms were considered and lower (median 20%, range 2–83%) when only clinical factors were considered. The CYP2C9 and VKORC1 genes, respectively, accounted for a median of 7% (<1–50%) and median of 25% (1–59%) of the variability in warfarin dose requirements. The R 2 performance stratified by race is shown in Table . A consideration of the race‐specific proportions in each stratified analysis (Tables ) should be made when interpreting the race‐stratified performances. For example, for 24 studies that included at least 5% Black patients, the proportion of warfarin dose variability that can be attributed to VKORC1 is 23%. However, these 24 studies on average included a median of only 13% (range 5–100%) Black patients. When only the 3 studies that included only Black patients are considered, the median VKORC1 partial R 2 becomes 9% (range 7–10%). These partial R 2 values should also be cautiously interpreted since different computation approaches yield different results (Figure ). Regarding the precision (predictive accuracy) and bias measures, the most reported measures were the mean absolute and mean prediction errors, respectively, being reported 137 (32%) and 17 (4%) times (algorithm development) and 222 (46%) and 144 (31%) times (external validations). The median mean absolute errors for the algorithm development and external validations were respectively 1.23 and 1.20 mg/d, with Asians (0.70 and 0.96 mg/d) and Whites (1.29 and 1.30 mg/d) having lower values when compared to Blacks (1.55 and 1.39 mg/d). The preferred precision and bias measures (Table ) were underreported, each being reported only once and thrice for algorithm development and external validation, respectively. Clinical relevance and other fit accuracy, precision and bias measures are detailed in Tables and . Because most studies reported R 2 (a fit accuracy measure), we carried out a posthoc correlation analysis and included the studies that reported both the R 2 and a precision accuracy measure to determine whether R 2 could be a good proxy of predictive accuracy. For this purpose, we used the mean absolute error as the predictive accuracy measure because it was the most reported (its limitations as a predictive accuracy measure (Table ) not withstanding). A total of 216 algorithm developments and external validations reported both these 2 measures, with lower mean absolute errors being moderately associated with higher R 2 values (Pearson's product–moment correlation coefficient −0.390 [95% confidence intervals −0.494 to −0.274], Figure ). Tables and include the performance measures stratified according to whether algorithms were clinical or pharmacogenetic (direct comparisons are available for some algorithm pairs in Tables and ). To directly compare the performances of algorithms stratified according to the modelling technique and time of application (dose initiation or dose revision), we summarized the studies that, using the same dataset, included at least 2 algorithms that differed in these 2 characteristics. As expected, dose revision algorithms generally performed better than dose initiation algorithms (Table ). Multiple linear regression performed comparable to or even better than many other machine learning techniques (Table ). Although pharmacokinetic/pharmacodynamic algorithms (fitted using nonlinear mixed effect modelling) performed better than other algorithms, this is mainly attributable to their dose revision aspects (i.e. when used for dose initiation, performance was comparable). However, the numbers of direct comparisons were few, and the performance metrics used were probably suboptimal (Table ). 

Risk of bias We focused on the assessment of the risk of bias in the analysis domain (Tables , , and ). During algorithm development, most developments had the number of participants per candidate predictor variable ≥20 ( n = 203, 47%), did not provide information on the handling of continuous and categorical predictors ( n = 291, 67%), probably included all enrolled participants in the analysis ( n = 229, 53%) and did not provide information on the handling of participants with missing data ( n = 233, 54%; Table ). Additionally, many algorithm developments relied on univariable ( n = 204, 47%) and multivariable ( n = 208, 48%) analysis during predictor selection, did not appropriately evaluate algorithm performance ( n = 232, 54%), did not account for model overfitting and optimism in algorithm performance ( n = 300, 69%), and did not provide enough information to assess whether predictors and their assigned weights in the final algorithms corresponded to the results reported in the multivariable analysis ( n = 220, 51%). Consequently, only 1 (<1%) algorithm was rated as having a low risk of bias (unclear n = 26, 6%; high n = 406, 94%). By contrast, most external validations included at least 100 participants with stable dose ( n = 329, 68%), all probably appropriately handled continuous and categorical predictors ( n = 481, 100%), mostly analysed all enrolled participants ( n = 309, 64%) although many did not provide enough information on the handling of participants with missing data ( n = 293, 61%). Most ( n = 273, 57%) reported the precision measures we considered appropriate for this review, although only 8 (<2%) were rated as having a low risk of bias (unclear n = 97, 20%; high n = 376, 78%). Although we did not focus on the risk of bias in the participant, predictors and outcome domains, the key risk of bias concerns in these domains are reported in Tables (algorithm development) and S7 (external validation). Of note, despite large variability, most algorithm developments ( n = 386, 89%) and external validations ( n = 433, 90%) provided stable dose definitions and/or referenced previous publications in which the same were provided. Lastly, for algorithm development, we also explored whether the outcome of stable dose was transformed during analysis since outcome transformation may affect the bias of the algorithm (Asiimwe, unpublished data). Most algorithm developments ( n = 228, 53%) did not incorporate any form of transformation, while 102 (24%) and 85 (20%) algorithm developments, respectively, incorporated logarithmic and square root transformations for reasons detailed in Table (for 18 [4%] algorithms, information was not available). The most common reason provided was to normalize dose and limit heteroscedasticity ( n = 120, 64%) with only 3 (2%) studies reporting clinical considerations as justification. 

DISCUSSION To facilitate a literature search of warfarin dosing algorithms by clinicians, guideline developers and/or policymakers, we have provided a comprehensive summary of existing algorithms ( n = 433), external validations ( n = 481) and clinical utility assessments ( n = 52) as well as described the populations for which they were developed. Using a threshold of at least 5% inclusion in a dataset, most algorithms were developed in Asians (49% of developed algorithms) and Whites (43%). Whereas 28% of the development datasets included Blacks, this group was under‐represented, the median percentage of Black participants in those datasets being only 14% (Asians and Whites, by contrast, comprised 100 and 90% [medians] of their corresponding algorithm development datasets). Similar trends were observed in the external validations and clinical utility assessments, with these results echoing previous reports of underrepresentation of minority groups. , The IWPC population categories were reported in the main results. As a result, Hispanic Caucasians were grouped with European Caucasians, sub‐Saharan Blacks with African Americans, Indians with Han‐Chinese, multiple sub‐populations under Mixed/Other etc.—groupings that simplify results' presentation but may be inaccurate in the context of personalized medicine. We therefore included locations (countries) from where the trial populations were recruited to serve as proxies for these sub‐populations, and further disparities were revealed. For example, despite 28% of the algorithm developments, 24% of the external validations and 31% of the clinical utility assessments including at least 5% Blacks, <1% algorithm developments, <1% external validations and none of the clinical utility assessments were conducted in sub‐Saharan Africa, results that re‐affirm our previous report that very few studies are conducted in Africa. Nonstatistical methods including clinical reasoning and literature support are recommended when deciding which candidate predictors to omit, combine or include during multivariable modelling. To make it easier for those developing new or updating existing algorithms, we also summarised the clinical, demographic, environmental, and genetic factors that are commonly included in warfarin dosing algorithms. The 4 most common demographic/clinical/environmental predictors were age, concomitant medications such as amiodarone, weight and sex being included in 93, 62, 53 and 32% of the 433 algorithms, respectively. By contrast, CYP2C9 , VKORC1 , and CYP4F2 variants were respectively included in 95, 92 and 27% of the 346 pharmacogenetic algorithms. All these are well‐established predictors whose mechanisms of action have been previously extensively reported. , In line with the ethnicity‐specific differences in minor allele frequencies of the various genetic variants, population‐specific differences were observed. For instance, CYP2C9*2 is almost absent in some Asian populations and it was included in only 29% of Asian algorithms that included the CYP2C9 gene ( n = 153; compared to being included in 92% of 153 White algorithms, 96% of 99 Black algorithms and 95% of 64 Mixed/Other algorithms). Other CYP2C9 variants (such as *5 , *6, *8 and *11 ) were mostly included in Black (36% of 99) and Mixed/Other (30% of 64) algorithms and less frequently in Asian (5% of 153) and White (19% of 153) algorithms. Despite a higher inclusion of these other CYP2C9 variants in studies employing at least 5% Blacks, 36% may still be a low figure given the importance of these African‐specific variants. When undertaking multivariable modelling, other population‐ and/or clinical setting‐specific considerations such as availability and cost of predictors should also always be considered. Our third objective was to evaluate the performances of these algorithms. As reported previously, the coefficient of determination ( R 2 ) was the most common performance measure (reported in 75% of algorithm developments and 54% of external validations). Based on R 2 , the median contribution of clinical factors (20%) and VKORC1 (25%) was similar to previous estimates , although CYP2C9 ’s contribution (7%) was lower (previously estimated at 12 and 15% ). Among the first of 2 key cautions is, like for all the other performance measures, these summary estimates were descriptive in nature since we did not conduct a formal quantitative synthesis, which with the preferred measures (Table ) and methods (such as individual participant data meta‐analysis ) is possible. Because of the descriptive nature of the study, different algorithms using the same or overlapping datasets was also of little concern. The second cautionary warning is that R 2 is a fit accuracy and not a prediction accuracy measure, the former of which is of less relevance when evaluating the value of prediction algorithms. , , For example, fit accuracy measures will mostly focus on the correct relative ordering of the dose predictions while predictive accuracy measures will also require that these predictions be close to the doses actually required by the patients. During a posthoc correlation analysis aimed at determining if R 2 could be a good proxy of 1 of the predictive accuracy measures (the mean absolute error, MAE), a moderate correlation coefficient (−0.39) was observed, which further questions the use of fit‐accuracy measures given that predictive accuracy measures are available. Among the predictive accuracy distance measures (Table ), the MAE is preferred to the [root] mean squared error mainly because it is less sensitive to outliers —and this would be an additional reason to prioritize it over R 2 , which is also highly sensitive to outliers. For example, the exclusion of only 1 outlier (<0.1% of 1010 participants) in the IWPC internal validation cohort improved the performance of the pharmacogenetic algorithm from an R 2 of 33 to 43% while in Langley's study ( n = 75), the R 2 increased from 9 to 31% when 2 outliers were excluded. To reiterate and for the reasons above, all the reported R 2 values, especially those that approach 100%, should be interpreted cautiously. Despite the mean absolute error being preferable to the above‐mentioned measures, it remains a distance measure whose limitations we detail in Table . In the same table, we describe an unbiased predictive accuracy ratio measure (derived from the logarithm of the ratio of the predicted dose to the actual dose , ), which we consider to be most appropriate. Unfortunately, and excluding our own study (Asiimwe, unpublished data), this measure was not used. We conducted some comparisons in performance based on populations, modelling techniques and the time an algorithm is applied but these results should again be cautiously interpreted because few studies reported direct comparisons and the performance measures used were likely to be inappropriate. For example, it may be misleading to conclude that Asians are better dosed compared to Whites or Blacks based on mean absolute error (or any distance measure that ignores the actual dose required by a patient) since as explained in Table , an MAE of 1 mg/d in a small value (e.g. 2 mg/d) may be clinically more important than a larger error (e.g. 2 mg/d) in an even larger value (e.g. 5 mg/d). With these precautions in mind, multiple linear regression (the most commonly used technique) seemed to perform comparable or even better than other supervised machine learning techniques as previously observed. A further understanding of these other techniques, and a more thorough comparison with multiple linear regression, is nevertheless recommended. For instance, artificial neural networks can capture very complex relationships, while pharmacokinetic/pharmacodynamic‐based techniques model both temporal and quantitative aspects of warfarin response and do not exclude unstable patients, which may be beneficial to warfarin dosing. Lastly, we assessed the risk of bias of existing algorithms with focus on the analysis domain. Less than 2% of both algorithm developments and external validations had a low risk of bias. Only our study was ranked as having a low risk of bias during algorithm development and although this could be because of self‐evaluation, we mainly attribute it to following the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) guidelines, which if followed correctly would result in a low risk of bias rating with the risk of bias assessment tool that we used. It is of concern that although the TRIPOD guidelines were published in 2015, none of the other 163 studies reported from 2016 onwards refer to its use. In the context of not adhering to current methodological recommendations, warfarin dosing algorithms may not be unique. , The consequences of most of the design flaws have been previously described in detail. One key issue that has received less attention is data transformation (done in 44% of the algorithm developments). As discussed by Keene, we also discourage data‐driven decisions to transform or not and recommend that the logarithmic transformation be preferred because it produces a proportional/multiplicative scale that is clinically relevant and easy to interpret. , A slightly more complex method to fit nontransformed dose using a proportional/multiplicative scale is to estimate the parameters of a linear algorithm using nonlinear (log–log) modelling , as we previously did (Asiimwe, unpublished data). In agreement with the CPIC guidelines, we recommend the IWPC and Gage clinical and pharmacogenetic algorithms since these have been the most externally validated and clinically assessed algorithms. Specifically, clinical utility assessments have concluded that they are better than fixed dose‐initiation approaches. , , , , , , , Recent debate has mainly focused on whether pharmacogenetic‐guided dosing strategies are better than clinical‐guided strategies with randomized controlled trials such as Kimmel et al ., Pirmohamed et al . and Gage et al . providing conflicting results—we neither quantitatively synthesized (both benefit and safety) nor assessed the risk of bias of the clinical utility studies since we felt these have been previously explored in more detail in several systematic reviews and meta‐analyses. , , , , , , , In addition to the CPIC guidance, before using these or other algorithms, clinicians, guideline developers and/or policymakers are reminded to ensure their applicability to their respective populations. For example, the above pharmacogenetic algorithms, despite including Black patients, may not be appropriate for Blacks because they exclude important Black‐specific genetic variants. Clinicians are also reminded that other numerous dosing algorithms that were not assessed in this review exist and may be more appropriate depending on the clinical setting. For example, we excluded many algorithms that rely only on current dose and measured INR levels to make this review more manageable, while the Food and Drug Administration dosing table was excluded because the methods used to derive this table are not publicly available. Nevertheless, these algorithms are less likely to perform better than those that incorporate predictors additional to existing dose/measured INR while the Food and Drug Administration table has been assessed in several studies , , , , , , and its performance is not better than the pharmacogenetic algorithms we have reported here. In addition to using heterogeneous and nonspecific racial categories, presenting mainly descriptive results, and excluding algorithms that rely only on current dose and measured INR levels, our study had other limitations. Specifically, we did not include non‐English articles, which could have affected geographical representation. For example, we excluded 9 Chinese studies during title/abstract screening and non‐English articles are less likely to be indexed in MEDLINE. Although we tried identifying other studies through reference list searching, using only the MEDLINE database also limited the number of studies that we could include in this review. Lastly, we relied on single‐reviewer extraction; a second reviewer, nevertheless, confirmed consistency based on a random selection of 10% of the included papers. For further research, novel/existing algorithms may need to be developed or updated and externally validated following the recommended guidelines such as TRIPOD. More attention needs to be paid to under‐represented populations such as minority ethnic groups and children (only 3% developed algorithms) to reduce health disparities. Moreover, although newer directly acting oral anticoagulants have been developed, warfarin is likely to remain the preferred choice for some of these groups. In conclusion, this systematic review provides a comprehensive summary of the algorithms available for different populations and their associated covariates (demographic, clinical, environmental and/or genetic), performances and risk of bias from these algorithms. Most of these algorithms have been developed for Asians and Whites, most have neither been externally validated nor assessed for clinical utility and either have a high or unclear risk of bias, which makes their reliability for clinical use uncertain. Future research should focus on developing prediction algorithms for under‐represented populations and externally validating and assessing the clinical utility of these and already existing algorithms. Algorithm development and assessment should follow current methodological recommendations to improve reliability and applicability. 

M.P. receives other research funding from various organisations including the EU Commission. He has also received partnership funding for the following: MRC Clinical Pharmacology Training Scheme (co‐funded by MRC and Roche, UCB, Eli Lilly and Novartis); a PhD studentship jointly funded by EPSRC and Astra Zeneca; and grant funding from Vistagen Therapeutics. He has also unrestricted educational grant support for the UK Pharmacogenetics and Stratified Medicine Network from Bristol‐Myers Squibb and UCB. He has developed an HLA genotyping panel with MC Diagnostics, but does not benefit financially from this. None of these of additional funding sources have been used for the current paper. 

I.G.A., E.J.Z., A.L.J. and M.P. conceived and designed the study. I.G.A. and R.O. performed data search, screening and analysis. A.L.J. and M.P. supervised the study. I.G.A. wrote the manuscript. A.L.J. and M.P. interpreted the results and made manuscript revisions. All authors read and approved the final manuscript. 

TABLE S1 Preferred Reporting Items for Systematic Reviews and Meta‐Analyses: The PRISMA Statement 1 TABLE S2 MEDLINE search strategy TABLE S3 Tailoring PROBAST (Prediction model Risk Of Bias ASsessment Tool) 2 to the systematic review TABLE S4 Relevant performance measures TABLE S5 Clinical and pharmacogenetic algorithms TABLE S6 Quality assessment (algorithm development) TABLE S7 External validations TABLE S8 Clinical utility assessments a TABLE S9 Algorithms that have been externally validated or assessed for clinical utility at least once a TABLE S10 Summary characteristics of algorithm developments, external validations and clinical utility assessments for studies that included at least 5% White participants TABLE S11 Summary characteristics of algorithm developments, external validations and clinical utility assessments for studies that included at least 5% Asian participants TABLE S12 Summary characteristics of algorithm developments, external validations and clinical utility assessments for studies that included at least 5% Black participants TABLE S13 Summary characteristics of algorithm developments, external validations and clinical utility assessments for studies that included at least 5% Mixed/Other participants TABLE S14 Algorithms that have been externally validated or assessed for clinical utility at least once a (studies that included at least 5% Whites) TABLE S15 Algorithms that have been externally validated or assessed for clinical utility at least once a (studies that included at least 5% Asians) TABLE S16 Algorithms that have been externally validated or assessed for clinical utility at least once a (studies that included at least 5% Blacks or 5% Mixed/Other participants) TABLE S17 Performance measures stratified by race TABLE S18 Dose initiation (a priori) vs dose revision (a posteriori) algorithms TABLE S19 A comparison of various modelling techniques TABLE S20 Summary results of the risk of bias assessment (analysis domain) TABLE S21 Reasons for logarithmic or square‐root transformations (n = 187). FIGURE S1 Computing R 2 and partial R 2 using set representation. FIGURE S2 A breakdown of the research articles included in the systematic review. FIGURE S3 Most commonly included predictors (Figure 2) stratified according to the 4 population categories. FIGURE S4 Correlation between a fit accuracy measure ( R 2 ) and a predictive accuracy measure (mean absolute error). Click here for additional data file. 

