text:
HomeIssuesSpecialtiesFor Authors
JAMA Network Open
Performance Matrix of Machine-Learning Models for Predicting Opioid Overdose in Medicare Beneficiaries
The 4 prediction performance matrixes in the validation sample are the area under the receiver operating characteristic curve (AUC) or C statistic (A); the precision-recall curves, which have improved performance if they are closer to the upper right corner or above the other method (B); the number needed to evaluate (NNE) by different cutoffs of sensitivity (C); and alerts per 100 patients by different cutoffs of sensitivity (D).
DNN indicates deep neural network; GBM, gradient boosting machine; LASSO, least absolute shrinkage and selection operator–type regularized regression; MLR, multivariate logistic regression; and RF, random forest.
Calibration Performance of Gradient Boosting Machine (GBM) and Deep Neural Network (DNN) by Risk Group
Risk subgroups were classified into 3 groups using the optimized threshold in the validation sample (n = 186 686): low risk (score below the optimized threshold), medium risk (predicted score between the optimized threshold, identified by the Youden index, and the top fifth percentile score), and high risk (predicted score in the top fifth percentile). The dashed line indicates the overall observed overdose rate without risk stratifications.
Top 50 Important Predictors for Opioid Overdose Selected by Gradient Boosting Machine
Rather than P values or coefficients, the gradient boosting machine reports the importance of predictors included in a model. Importance is a measure of each variable's cumulative contribution toward reducing square error, or heterogeneity within the subset, after the data set is sequentially split according to that variable. Thus, importance reflects a variable's significance in prediction. Absolute importance is then scaled to give relative importance, with a maximum importance of 100. For example, the top 10 important predictors identified from the gradient boosting machine model included total opioid dose (eg, >1500 morphine milligram equivalent [MME] during 3 months), diagnosis of alcohol use disorders or substance use disorders (AUD/SUD), mean daily opioid dose (eg, >32 MME), age, disability status, total number of opioid prescriptions (eg, >4), beneficiary's state residency (eg, Florida, Kentucky, or New Jersey), type of opioid use (eg, with mixed schedules), total number of benzodiazepine prescription fills (eg, >3), and cumulative days of early prescription refills (eg, >19 days). ED indicates emergency department; FFS: fee-for-service.
Prediction Performance of Gradient Boosting Machine and Deep Neural Network Models in the Validation Sample Divided Into Risk Subgroupsa
Comparison of Prediction Performance Between Centers for Medicare & Medicaid Services Measures and Deep Neural Network Measures Over a 12-Month Period
Supplement.
eAppendix. Machine Learning Approaches Used in the Study
eTable 1. Diagnosis Codes for the Exclusion of Patients With Malignant Cancers Based on the National Committee for Quality Assurance (NCQA)'s Opioid Measures in 2018 Healthcare Effectiveness Data and Information Set (HEDIS)
eTable 2. Diagnosis Codes for Identifying Opioid Overdose
eTable 3. Other Diagnosis Codes Used to Identify the Likelihood of Opioid Overdose
eTable 4. Summary of Predictor Candidates (n=268) Measured in 3-month Windows for Predicting Subsequent Opioid Overdose
eTable 5. Opioid Overdose and Sociodemographic Characteristics Among Medicare Beneficiaries (n=560,057), Divided Into Training, Testing, and Validation Samples
eTable 6. Prediction Performance Measures for Predicting Opioid Overdose, Across Different Machine Learning Methods With Varying Sensitivity and Specificity
eTable 7. Comparison of Prediction Performance Using Any of Centers for Medicaid & Medicaid Services (CMS) High-Risk Opioid Use Measures vs. Deep Neural Network (DNN) and Gradient Boosting Machine (GBM) in the Validation Sample (n=166,580) Over a 12-month Period
eFigure 1. Sample Size Flow Chart of Study Cohort
eFigure 2. Illustrations of Two Study Designs: 3-month Windows for Measuring Predictor Candidates and Overdose Events
eFigure 3. Classification Matrix and Definition of Prediction Performance Metrics
eFigure 4. Prediction Performance Matrix Across Machine Learning Approaches in Predicting Opioid Overdose Risk in the Subsequent 3 Months: Sensitivity Analyses Including the Information Measured in All the Historical 3-Months Windows
eFigure 5. Scatter Plot Between Deep Neural Network (DNN) and Gradient Boosting Machine (GBM)'s Prediction Scores
eFigure 6. Top 50 Important Predictors for Opioid Overdose Selected by Random Forest (RF)
eReferences
Substance Abuse and Mental Health Services Administration Center for Behavioral Health Statistics and Quality. Results from the 2016 National Survey on Drug Use and Health: detailed tables. https://www.samhsa.gov/data/sites/default/files/NSDUH-DetTabs-2016/NSDUH-DetTabs-2016.pdf. Published September 15, 2017. Accessed February 15, 2019.
Centers for Disease Control and Prevention. National Center for Health Statistics, 2016: multiple cause of death data, 1999-2017. http://wonder.cdc.gov/mcd.html. Accessed January 27, 2019.
Rudd RA, Seth P, David F, Scholl L. Increases in drug and opioid-involved overdose deaths - United States, 2010-2015. MMWR Morb Mortal Wkly Rep. 2016;65(50-51):1445-1452. doi:10.15585/mmwr.mm655051e1PubMedGoogle ScholarCrossref
Seth P, Scholl L, Rudd RA, Bacon S. Overdose deaths involving opioids, cocaine, and psychostimulants - United States, 2015-2016. MMWR Morb Mortal Wkly Rep. 2018;67(12):349-358. doi:10.15585/mmwr.mm6712a1PubMedGoogle ScholarCrossref
Florence CS, Zhou C, Luo F, Xu L. The economic burden of prescription opioid overdose, abuse, and dependence in the United States, 2013. Med Care. 2016;54(10):901-906. doi:10.1097/MLR.0000000000000625PubMedGoogle ScholarCrossref
Roberts AW, Gellad WF, Skinner AC. Lock-in programs and the opioid epidemic: a call for evidence. Am J Public Health. 2016;106(11):1918-1919. doi:10.2105/AJPH.2016.303404PubMedGoogle ScholarCrossref
The US Congressional Research Service. The SUPPORT for Patients and Communities Act (P.L.115-271): Medicare Provisions. https://www.everycrsreport.com/files/20190102_R45449_231fb05ad093244bc8b91a84133fe310b2892ebe.pdf. Updated January 2, 2019. Accessed January 28, 2019.
Webster LR, Webster RM. Predicting aberrant behaviors in opioid-treated patients: preliminary validation of the Opioid Risk Tool. Pain Med. 2005;6(6):432-442. doi:10.1111/j.1526-4637.2005.00072.xPubMedGoogle ScholarCrossref
Ives TJ, Chelminski PR, Hammett-Stabler CA, et al. Predictors of opioid misuse in patients with chronic pain: a prospective cohort study. BMC Health Serv Res. 2006;6:46-55. doi:10.1186/1472-6963-6-46PubMedGoogle ScholarCrossref
Becker WC, Sullivan LE, Tetrault JM, Desai RA, Fiellin DA. Non-medical use, abuse and dependence on prescription opioids among U.S. adults: psychiatric, medical and substance use correlates. Drug Alcohol Depend. 2008;94(1-3):38-47. doi:10.1016/j.drugalcdep.2007.09.018PubMedGoogle ScholarCrossref
Hall AJ, Logan JE, Toblin RL, et al. Patterns of abuse among unintentional pharmaceutical overdose fatalities. JAMA. 2008;300(22):2613-2620. doi:10.1001/jama.2008.802PubMedGoogle ScholarCrossref
Centers for Disease Control and Prevention (CDC). Overdose deaths involving prescription opioids among Medicaid enrollees - Washington, 2004-2007. MMWR Morb Mortal Wkly Rep. 2009;58(42):1171-1175.PubMedGoogle Scholar
White AG, Birnbaum HG, Schiller M, Tang J, Katz NP. Analytic models to identify patients at risk for prescription opioid abuse. Am J Manag Care. 2009;15(12):897-906.PubMedGoogle Scholar
Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study. Ann Intern Med. 2010;152(2):85-92. doi:10.7326/0003-4819-152-2-201001190-00006PubMedGoogle ScholarCrossref
Edlund MJ, Martin BC, Devries A, Fan MY, Braden JB, Sullivan MD. Trends in use of opioids for chronic noncancer pain among individuals with mental health and substance use disorders: the TROUP study. Clin J Pain. 2010;26(1):1-8. doi:10.1097/AJP.0b013e3181b99f35PubMedGoogle ScholarCrossref
Sullivan MD, Edlund MJ, Fan MY, Devries A, Brennan Braden J, Martin BC. Risks for possible and probable opioid misuse among recipients of chronic opioid therapy in commercial and medicaid insurance plans: the TROUP Study. Pain. 2010;150(2):332-339. doi:10.1016/j.pain.2010.05.020PubMedGoogle ScholarCrossref
Bohnert AS, Valenstein M, Bair MJ, et al. Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA. 2011;305(13):1315-1321. doi:10.1001/jama.2011.370PubMedGoogle ScholarCrossref
Volkow ND, McLellan TA, Cotto JH, Karithanom M, Weiss SR. Characteristics of opioid prescriptions in 2009. JAMA. 2011;305(13):1299-1301. doi:10.1001/jama.2011.401PubMedGoogle ScholarCrossref
Webster LR, Cochella S, Dasgupta N, et al. An analysis of the root causes for opioid-related overdose deaths in the United States. Pain Med. 2011;12(suppl 2):S26-S35. doi:10.1111/j.1526-4637.2011.01134.xPubMedGoogle ScholarCrossref
Cepeda MS, Fife D, Chow W, Mastrogiovanni G, Henderson SC. Assessing opioid shopping behaviour: a large cohort study from a medication dispensing database in the US. Drug Saf. 2012;35(4):325-334. doi:10.2165/11596600-000000000-00000PubMedGoogle ScholarCrossref
Peirce GL, Smith MJ, Abate MA, Halverson J. Doctor and pharmacy shopping for controlled substances. Med Care. 2012;50(6):494-500. doi:10.1097/MLR.0b013e31824ebd81PubMedGoogle ScholarCrossref
Rice JB, White AG, Birnbaum HG, Schiller M, Brown DA, Roland CL. A model to identify patients at risk for prescription opioid abuse, dependence, and misuse. Pain Med. 2012;13(9):1162-1173. doi:10.1111/j.1526-4637.2012.01450.xPubMedGoogle ScholarCrossref
Gwira Baumblatt JA, Wiedeman C, Dunn JR, Schaffner W, Paulozzi LJ, Jones TF. High-risk use by patients prescribed opioids for pain and its role in overdose deaths. JAMA Intern Med. 2014;174(5):796-801. doi:10.1001/jamainternmed.2013.12711PubMedGoogle ScholarCrossref
Hylan TR, Von Korff M, Saunders K, et al. Automated prediction of risk for problem opioid use in a primary care setting. J Pain. 2015;16(4):380-387. doi:10.1016/j.jpain.2015.01.011PubMedGoogle ScholarCrossref
Zedler B, Xie L, Wang L, et al. Development of a risk index for serious prescription opioid-induced respiratory depression or overdose in Veterans' Health Administration patients. Pain Med. 2015;16(8):1566-1579. doi:10.1111/pme.12777PubMedGoogle ScholarCrossref
Cochran G, Gordon AJ, Lo-Ciganic WH, et al. An examination of claims-based predictors of overdose from a large Medicaid program. Med Care. 2017;55(3):291-298. doi:10.1097/MLR.0000000000000676PubMedGoogle ScholarCrossref
Carey CM, Jena AB, Barnett ML. Patterns of potential opioid misuse and subsequent adverse outcomes in Medicare, 2008 to 2012. Ann Intern Med. 2018;168(12):837-845. doi:10.7326/M17-3065PubMedGoogle ScholarCrossref
Glanz JM, Narwaney KJ, Mueller SR, et al. Prediction model for two-year risk of opioid overdose among patients prescribed chronic opioid therapy. J Gen Intern Med. 2018;33(10):1646-1653. doi:10.1007/s11606-017-4288-3PubMedGoogle ScholarCrossref
Rose AJ, Bernson D, Chui KKH, et al. Potentially inappropriate opioid prescribing, overdose, and mortality in Massachusetts, 2011-2015. J Gen Intern Med. 2018;33(9):1512-1519. doi:10.1007/s11606-018-4532-5PubMedGoogle ScholarCrossref
Zedler BK, Saunders WB, Joyce AR, Vick CC, Murrelle EL. Validation of a screening risk index for serious prescription opioid-induced respiratory depression or overdose in a US commercial health plan claims database. Pain Med. 2018;19(1):68-78. doi:10.1093/pm/pnx009PubMedGoogle ScholarCrossref
Oliva EM, Bowe T, Tavakoli S, et al. Development and applications of the Veterans Health Administration's Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide. Psychol Serv. 2017;14(1):34-49. doi:10.1037/ser0000099PubMedGoogle ScholarCrossref
Iams JD, Newman RB, Thom EA, et al; National Institute of Child Health and Human Development Network of Maternal-Fetal Medicine Units. Frequency of uterine contractions and the risk of spontaneous preterm delivery. N Engl J Med. 2002;346(4):250-255. doi:10.1056/NEJMoa002868PubMedGoogle ScholarCrossref
Hsich E, Gorodeski EZ, Blackstone EH, Ishwaran H, Lauer MS. Identifying important risk factors for survival in patient with systolic heart failure using random survival forests. Circ Cardiovasc Qual Outcomes. 2011;4(1):39-45. doi:10.1161/CIRCOUTCOMES.110.939371PubMedGoogle ScholarCrossref
Gorodeski EZ, Ishwaran H, Kogalur UB, et al. Use of hundreds of electrocardiographic biomarkers for prediction of mortality in postmenopausal women: the Women's Health Initiative. Circ Cardiovasc Qual Outcomes. 2011;4(5):521-532. doi:10.1161/CIRCOUTCOMES.110.959023PubMedGoogle ScholarCrossref
Chen G, Kim S, Taylor JM, et al. Development and validation of a quantitative real-time polymerase chain reaction classifier for lung cancer prognosis. J Thorac Oncol. 2011;6(9):1481-1487. doi:10.1097/JTO.0b013e31822918bdPubMedGoogle ScholarCrossref
Amalakuhan B, Kiljanek L, Parvathaneni A, Hester M, Cheriyath P, Fischman D. A prediction model for COPD readmissions: catching up, catching our breath, and improving a national problem. J Community Hosp Intern Med Perspect. 2012;2(1):9915-9921. doi:10.3402/jchimp.v2i1.9915PubMedGoogle ScholarCrossref
Chirikov VV, Shaya FT, Onukwugha E, Mullins CD, dosReis S, Howell CD. Tree-based claims algorithm for measuring pretreatment quality of care in Medicare disabled hepatitis C patients. Med Care. 2017;55(12):e104-e112.PubMedGoogle ScholarCrossref
Thottakkara P, Ozrazgat-Baslanti T, Hupf BB, et al. Application of machine learning techniques to high-dimensional clinical data to forecast postoperative complications. PLoS One. 2016;11(5):e0155705. doi:10.1371/journal.pone.0155705PubMedGoogle ScholarCrossref
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55-63. doi:10.7326/M14-0697PubMedGoogle ScholarCrossref
Bossuyt PM, Reitsma JB, Bruns DE, et al; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. doi:10.1136/bmj.h5527PubMedGoogle ScholarCrossref
Herzig SJ, Rothberg MB, Cheung M, Ngo LH, Marcantonio ER. Opioid utilization and opioid-related adverse events in nonsurgical patients in US hospitals. J Hosp Med. 2014;9(2):73-81. doi:10.1002/jhm.2102PubMedGoogle ScholarCrossref
Unick GJ, Rosenblum D, Mars S, Ciccarone D. Intertwined epidemics: national demographic trends in hospitalizations for heroin- and opioid-related overdoses, 1993-2009. PLoS One. 2013;8(2):e54496. doi:10.1371/journal.pone.0054496PubMedGoogle ScholarCrossref
Larochelle MR, Zhang F, Ross-Degnan D, Wharam JF. Rates of opioid dispensing and overdose after introduction of abuse-deterrent extended-release oxycodone and withdrawal of propoxyphene. JAMA Intern Med. 2015;175(6):978-987. doi:10.1001/jamainternmed.2015.0914PubMedGoogle ScholarCrossref
Fulton-Kehoe D, Sullivan MD, Turner JA, et al. Opioid poisonings in Washington State Medicaid: trends, dosing, and guidelines. Med Care. 2015;53(8):679-685. doi:10.1097/MLR.0000000000000384PubMedGoogle ScholarCrossref
Yang Z, Wilsey B, Bohm M, et al. Defining risk of prescription opioid overdose: pharmacy shopping and overlapping prescriptions among long-term opioid users in medicaid. J Pain. 2015;16(5):445-453. doi:10.1016/j.jpain.2015.01.475PubMedGoogle ScholarCrossref
Edlund MJ, Martin BC, Fan MY, Braden JB, Devries A, Sullivan MD. An analysis of heavy utilizers of opioids for chronic noncancer pain in the TROUP study. J Pain Symptom Manage. 2010;40(2):279-289. doi:10.1016/j.jpainsymman.2010.01.012PubMedGoogle ScholarCrossref
Lange A, Lasser KE, Xuan Z, et al. Variability in opioid prescription monitoring and evidence of aberrant medication taking behaviors in urban safety-net clinics. Pain. 2015;156(2):335-340. doi:10.1097/01.j.pain.0000460314.73358.ffPubMedGoogle ScholarCrossref
Jann M, Kennedy WK, Lopez G. Benzodiazepines: a major component in unintentional prescription drug overdoses with opioid analgesics. J Pharm Pract. 2014;27(1):5-16. doi:10.1177/0897190013515001PubMedGoogle ScholarCrossref
Liu Y, Logan JE, Paulozzi LJ, Zhang K, Jones CM. Potential misuse and inappropriate prescription practices involving opioid analgesics. Am J Manag Care. 2013;19(8):648-665.PubMedGoogle Scholar
Logan J, Liu Y, Paulozzi L, Zhang K, Jones C. Opioid prescribing in emergency departments: the prevalence of potentially inappropriate prescribing and misuse. Med Care. 2013;51(8):646-653. doi:10.1097/MLR.0b013e318293c2c0PubMedGoogle ScholarCrossref
Mack KA, Zhang K, Paulozzi L, Jones C. Prescription practices involving opioid analgesics among Americans with Medicaid, 2010. J Health Care Poor Underserved. 2015;26(1):182-198. doi:10.1353/hpu.2015.0009PubMedGoogle ScholarCrossref
Sharabiani MT, Aylin P, Bottle A. Systematic review of comorbidity indices for administrative data. Med Care. 2012;50(12):1109-1118. doi:10.1097/MLR.0b013e31825f64d0PubMedGoogle ScholarCrossref
Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8-27. doi:10.1097/00005650-199801000-00004PubMedGoogle ScholarCrossref
Gordon AJ, Lo-Ciganic WH, Cochran G, et al. Patterns and Quality of Buprenorphine Opioid Agonist Treatment in a Large Medicaid Program. J Addict Med. 2015;9(6):470-477. doi:10.1097/ADM.0000000000000164PubMedGoogle ScholarCrossref
HipXChange. Area deprivation index datasets. https://www.hipxchange.org/ADI. Accessed November 13, 2018.
County Health Rankings and Roadmaps. Use the data. http://www.countyhealthrankings.org/explore-health-rankings/use-data. Accessed November 3, 2018.
Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York, NY: Springer; 2008.
Chu A, Ahn H, Halwan B, et al. A decision support system to facilitate management of patients with acute gastrointestinal bleeding. Artif Intell Med. 2008;42(3):247-259. doi:10.1016/j.artmed.2007.10.003PubMedGoogle ScholarCrossref
Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015;10(3):e0118432. doi:10.1371/journal.pone.0118432PubMedGoogle ScholarCrossref
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-845. doi:10.2307/2531595PubMedGoogle ScholarCrossref
Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015;19:285-290. doi:10.1186/s13054-015-0999-1PubMedGoogle ScholarCrossref
Tufféry S. Data Mining and Statistics for Decision Making. West Sussex, UK: John Wiley & Sons; 2011. doi:10.1002/9780470979174
Fluss R, Faraggi D, Reiser B. Estimation of the Youden Index and its associated cutoff point. Biom J. 2005;47(4):458-472. doi:10.1002/bimj.200410135PubMedGoogle ScholarCrossref
Centers for Medicare & Medicaid Services (CMS). Announcement of calendar year (CY) 2019 Medicare Advantage capitation rates and Medicare Advantage and Part D payment policies and final call letter. https://www.cms.gov/Medicare/Health-Plans/MedicareAdvtgSpecRateStats/Downloads/Announcement2019.pdf. Accessed November 6, 2018.
Goldstein BA, Pencina MJ, Montez-Rath ME, Winkelmayer WC. Predicting mortality over different time horizons: which data elements are needed? J Am Med Inform Assoc. 2017;24(1):176-181. doi:10.1093/jamia/ocw057PubMedGoogle ScholarCrossref
Hendrick RE, Helvie MA. Mammography screening: a new estimate of number needed to screen to prevent one breast cancer death. AJR Am J Roentgenol. 2012;198(3):723-728. doi:10.2214/AJR.11.7146PubMedGoogle ScholarCrossref
Lutgendorf MA, Stoll KA. Why 99% may not be as good as you think it is: limitations of screening for rare diseases. J Matern Fetal Neonatal Med. 2016;29(7):1187-1189.PubMedGoogle Scholar
Errors in Methods, Results, Table 1, Figure 2, and Supplement
Machine Learning Website
Substance Use and Addiction Toxicology Opioids Health Care Economics, Insurance, Payment Health Informatics
Sign Up for Emails Based on Your Interests
Get the latest research based on your areas of interest.
Conflicts of Interest Disclosure
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Yes, I have potential conflicts of interest.
No, I do not have potential conflicts of interest.
Limit 3600 characters or approximately 600 words
The following information is required and must be completed in order to submit a comment:
Your comment submission was successful.
Please allow up to 2 business days for review, approval, and posting.
2 Comments for this article
Addiction is a Medical Problem?
Thomas Hilton, PhD | Retired, NIH/NIDA
I do not doubt the sincerity of the authors in trying to prevent opioid overdose among the elderly, but I find it to be quite a stretch to tacitly suggest that overdose is due to medical mismanagement, and therefore can be prevented by an algorithm's alert. Many of the variables in the equation have been associated with intentional as well as accidental overdose. This fact is acknowledged in the limitations section but inadequately discussed as undermining AI application in clinical practice.
An AI shotgun cannot match an attending physician's need for a more nuanced understanding of their patient than an algorithm can produce even if it has a full magazine of data in the chamber - which is unlikely.
CONFLICT OF INTEREST: None Reported
Error in Text, Table 1, Figure 2 and Supplement
Jenny Lo-Ciganic, PhD | University of Florida, College of Pharmacy
In the Original Investigation titled "Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose among Medicare Beneficiaries with Opioid Prescriptions", published March 22, 2019, there were minor errors in the texts, Table 1, Figure 2, and Supplement. We identified these errors when duplicating the analyses for a separate project. When presenting our risk stratification results for the GBM and DNN models, we inadvertently presented results defining high risk differently in the GBM and DNN models, using the 5th percentile in one model and 10th percentile in the other. We have now corrected these errors in the text, Table 1 and Figure 1 by consistently using the 5th percentile of scores. We also provide the results using the 10th percentiles of scores for both GBM and DNN in the Online Supplement. None of the conclusions or interpretations are affected. This article has been corrected.
1. Lo-Ciganic W, Huang JL, Zhang HH et al. Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions. JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968
View Correction
Lo-Ciganic W, Huang JL, Zhang HH, et al. Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions. JAMA Netw Open. Published online March 22, 20192(3):e190968. doi:10.1001/jamanetworkopen.2019.0968
Original Investigation
Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions
Wei-Hsuan Lo-Ciganic, PhD1; James L. Huang, PhD1; Hao H. Zhang, PhD2; et al Jeremy C. Weiss, MD, PhD3; Yonghui Wu, PhD4; C. Kent Kwoh, MD5; Julie M. Donohue, PhD6; Gerald Cochran, PhD7; Adam J. Gordon, MD, MPH7,8; Daniel C. Malone, PhD9; Courtney C. Kuza, PhD10; Walid F. Gellad, MD, MPH10,11,12
1Department of Pharmaceutical Outcomes & Policy, College of Pharmacy, University of Florida, Gainesville
2Department of Mathematics, University of Arizona, Tucson
3Carnegie Mellon University, Heinz College, Pittsburgh, Pennsylvania
4Department of Health Outcomes & Biomedical Informatics, University of Florida, College of Medicine, Gainesville
5Division of Rheumatology, Department of Medicine, and the University of Arizona Arthritis Center, University of Arizona, Tucson
6Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania
7Program for Addiction Research, Clinical Care, Knowledge, and Advocacy, Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City
8Informatics, Decision-Enhancement, and Analytic Sciences Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, Utah
9Department of Pharmacy, Practice and Science, College of Pharmacy, University of Arizona, Tucson
10Center for Pharmaceutical Policy and Prescribing, Health Policy Institute, University of Pittsburgh, Pittsburgh, Pennsylvania
11Department of Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania
12Center for Health Equity Research Promotion, Veterans Affairs Pittsburgh Healthcare System, Pittsburgh, Pennsylvania
JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968
Key Points español 中文 (chinese)
Question Can machine-learning approaches predict opioid overdose risk among fee-for-service Medicare beneficiaries?
Findings In this prognostic study of the administrative claims data of 560 057 Medicare beneficiaries, the deep neural network and gradient boosting machine models outperformed other methods for identifying risk, although positive predictive values were low given the low prevalence of overdose episodes.
Meaning Machine-learning algorithms using administrative data appear to be a valuable and feasible tool for more accurate identification of opioid overdose risk.
Importance Current approaches to identifying individuals at high risk for opioid overdose target many patients who are not truly at high risk.
Objective To develop and validate a machine-learning algorithm to predict opioid overdose risk among Medicare beneficiaries with at least 1 opioid prescription.
Design, Setting, and Participants A prognostic study was conducted between September 1, 2017, and December 31, 2018. Participants (n = 560 057) included fee-for-service Medicare beneficiaries without cancer who filled 1 or more opioid prescriptions from January 1, 2011, to December 31, 2015. Beneficiaries were randomly and equally divided into training, testing, and validation samples.
Exposures Potential predictors (n = 268), including sociodemographics, health status, patterns of opioid use, and practitioner-level and regional-level factors, were measured in 3-month windows, starting 3 months before initiating opioids until loss of follow-up or the end of observation.
Main Outcomes and Measures Opioid overdose episodes from inpatient and emergency department claims were identified. Multivariate logistic regression (MLR), least absolute shrinkage and selection operator–type regression (LASSO), random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN) were applied to predict overdose risk in the subsequent 3 months after initiation of treatment with prescription opioids. Prediction performance was assessed using the C statistic and other metrics (eg, sensitivity, specificity, and number needed to evaluate [NNE] to identify one overdose). The Youden index was used to identify the optimized threshold of predicted score that balanced sensitivity and specificity.
Results Beneficiaries in the training (n = 186 686), testing (n = 186 685), and validation (n = 186 686) samples had similar characteristics (mean [SD] age of 68.0 [14.5] years, and approximately 63% were female, 82% were white, 35% had disabilities, 41% were dual eligible, and 0.60% had at least 1 overdose episode). In the validation sample, the DNN (C statistic = 0.91; 95% CI, 0.88-0.93) and GBM (C statistic = 0.90; 95% CI, 0.87-0.94) algorithms outperformed the LASSO (C statistic = 0.84; 95% CI, 0.80-0.89), RF (C statistic = 0.80; 95% CI, 0.75-0.84), and MLR (C statistic = 0.75; 95% CI, 0.69-0.80) methods for predicting opioid overdose. At the optimized sensitivity and specificity, DNN had a sensitivity of 92.3%, specificity of 75.7%, NNE of 542, positive predictive value of 0.18%, and negative predictive value of 99.9%. The DNN classified patients into low-risk (76.2% [142 180] of the cohort), medium-risk (18.6% [34 579] of the cohort), and high-risk (5.2% [9747] of the cohort) subgroups, with only 1 in 10 000 in the low-risk subgroup having an overdose episode. More than 90% of overdose episodes occurred in the high-risk and medium-risk subgroups, although positive predictive values were low, given the rare overdose outcome.
Conclusions and Relevance Machine-learning algorithms appear to perform well for risk prediction and stratification of opioid overdose, especially in identifying low-risk subgroups that have minimal risk of overdose.
In 2016, 11.8 million American individuals reported using prescription opioids nonmedically,1 and an estimated 115 individuals died each day from opioid overdose.2-4 The annual cost of misuse or abuse of opioids exceeds $78.5 billion, including the costs of health care, lost productivity, substance abuse treatment, and the criminal justice system.5
In response, health systems, payers, and policymakers have developed programs to identify and intervene in individuals at high risk of problematic opioid use and overdose. These programs, whether outreach calls from case managers, prior authorizations, referrals to substance use disorder specialists, dispensing of naloxone hydrochloride, or enrollment in lock-in programs, can be expensive to payers and burdensome to patients. The determination of who is at high risk is a factor in the size and scope of these interventions and the resources expended. Yet, the definition of high risk is variable, ranging from a high-dose opioid (defined using various cut points) to the number of pharmacies or prescribers that a patient visits. These criteria, for example, determine how Medicare beneficiaries are selected into so-called lock-in programs in Medicare, also called the Comprehensive Addiction and Recovery Act (CARA) drug management programs.6 These programs will soon be required for all Part D plans.7
These current measures of high risk were derived from studies that used traditional statistical methods to identify risk factors for overdose rather than predict an individual's risk.8-31 However, individual risk factors may not be strong predictors of overdose risk.32 Moreover, traditional statistical approaches have limited ability to handle nonlinear risk prediction and complex interactions among predictors. For example, receipt of a high-dose opioid is a well-known overdose risk factor, but the complex interactions between opioid dose, substance use disorders, mental health, emergency department visits, prescriber characteristics, and socioeconomic variables may yield greater predictive power than one factor alone. The few previous studies focused on predicting opioid overdose (rather than simply identifying risk factors) either had suboptimal prediction performance24,28,31 or used case-control designs that were unable to measure true overdose incidence and may not adequately calibrate algorithms to real data for rare outcomes such as overdose.22,25,30
Machine learning is an alternative analytic approach to handling complex interactions in large data, discovering hidden patterns, and generating actionable predictions in clinical settings. In many cases, machine learning is superior to traditional statistical techniques.33-38 Machine learning has been widely used in activities from fraud detection to genomic studies but, to our knowledge, has not yet been applied to address the opioid epidemic. Our overall hypothesis was that a machine-learning algorithm would perform better in predicting opioid overdose risk compared with traditional statistical approaches.
The objective of this study was to develop and validate a machine-learning algorithm to predict opioid overdose among Medicare beneficiaries with at least 1 opioid prescription. Based on the prediction score, we stratified beneficiaries into subgroups at similar overdose risk to support clinical decisions and improved targeting of intervention. We chose Medicare because of the high prevalence of prescription opioid use and the availability of national claims data and because the program will require specific interventions targeting individuals at high risk for opioid-associated morbidity.6,7
Design and Sample
This prognostic study was conducted between September 1, 2017, and December 31, 2018. The University of Arizona Institutional Review Board approved the study. This study followed the Standards for Reporting of Diagnostic Accuracy (STARD) and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines.39,40
We included prescription drug and medical claims for a 5% random sample of Medicare beneficiaries between January 1, 2011, and December 31, 2015. We identified fee-for-service adult beneficiaries without cancer who were US residents and received 1 or more opioid prescriptions during the study period. We excluded beneficiaries who (1) filled only parenteral opioid prescriptions and/or cough or cold medication prescriptions containing opioids, (2) had malignant cancer diagnoses (eTable 1 in the Supplement), (3) received hospice, (4) ever enrolled in Medicare Advantage plans (because their health care use in Medicare Advantage may not be observable), or (5) had their first opioid prescription after October 1, 2015 (eFigure 1 in the Supplement). An index date was defined as the date of a patient's first opioid prescription between April 1, 2011, and September 30, 2015. Once eligible, beneficiaries remained in the cohort, regardless of whether they continued to receive opioid prescriptions, until they were censored because of death or the end of observation.
Outcome Variables: Opioid Overdose
We identified any occurrence of fatal or nonfatal opioid overdose (prescription opioids or other opioids, including heroin), defined in each 3-month window after the index prescription using the International Classification of Diseases, Ninth Revision, and International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10), codes for overdose (eTable 2 in the Supplement) from inpatient or emergency department settings.14,41-44 Overdose was defined with either an opioid overdose code as the primary diagnosis (80% of identified overdose episodes) or other drug overdose or substance use disorder code as the primary diagnosis (eTable 3 in the Supplement) and opioid overdose as the nonprimary diagnosis (20% of identified overdose episodes), as defined previously.14 Sensitivity analyses using opioid overdose as the primary diagnosis and capturing any opioid overdose diagnosis code in any position yielded similar results.
Predictor Candidates
We compiled 268 predictor candidates, informed by the literature (eTable 4 in the Supplement).8-31 Patient, practitioner, and regional factors were measured at baseline in the 3 months before the first opioid prescription fill and in 3-month windows after initiating prescription opioids. We chose a 3-month window in accordance with the literature and to be consistent with the quarterly evaluation period commonly used by prescription drug monitoring programs and health plans.13,14,45 In the primary analysis, we used the variables measured in each 3-month period (eg, the first) to predict overdose risk in each subsequent 3-month period (eg, the second) (eFigure 2A in the Supplement). In sensitivity analyses, instead of using a previous 3-month period to predict overdose in the next period, we included information collected in all of the historical 3-month windows to predict opioid risk for each 3-month period for each person (eFigure 2B in the Supplement).
The predictor candidates also included a series of variables related to prescription opioid and relevant medication use: (1) total and mean daily morphine milligram equivalent (MME),17 (2) cumulative and continuous duration of opioid use (ie, no gap >32 days between fills),45 (3) total number of opioid prescriptions overall and by active ingredient, (4) type of opioid based on the US Drug Enforcement Administration's Controlled Substance Schedule (I-IV) and duration of action, (5) number of opioid prescribers, (6) number of pharmacies providing opioid prescriptions,11,17,23 (7) number of early opioid prescription refills (refilling opioid prescriptions >3 days before the previous prescription runs out),46 (8) cumulative days of early opioid prescription refills, (9) cumulative days of concurrent benzodiazepines and/or muscle relaxant use, (10) number and duration of other relevant prescriptions (eg, gabapentinoids), and (11) receipt of methadone hydrochloride or buprenorphine hydrochloride for opioid use disorder.19,47-50
Patient sociodemographic characteristics included age, sex, race/ethnicity, disability as the reason for Medicare eligibility, receipt of low-income subsidy, and urbanicity of county of residence. Health status factors (eg, number of emergency department visits) were derived from the literature and are listed in eTable 4 in the Supplement.13,16,51-54 Practitioner factors included opioid prescriber's sex, specialty, mean monthly opioid prescribing volume and MME, and mean monthly number of patients receiving opioids. Many beneficiaries had more than 1 opioid prescriber, in which case the practitioner prescribing the highest number of opioids was designated as the primary prescriber. Regional factors (eg, percentage of households below the federal poverty level) included variables obtained from publicly available resources, including the Area Health Resources Files, Area Deprivation Index data sets,55 and County Health Rankings data.56
Machine-Learning Approaches and Prediction Performance Evaluation
Our primary goal was risk prediction, and the secondary goal was risk stratification (ie, identifying patient subgroups at similar overdose risk). First, we randomly and equally divided the cohort into training (developing algorithms), testing (refining algorithms), and validation (evaluating algorithm's prediction performance) samples. In both the primary and sensitivity analyses (eFigure 2 in the Supplement), we developed and tested prediction algorithms for opioid overdose using 5 commonly used machine-learning approaches: multivariate logistic regression, least absolute shrinkage and selection operator–type regression (LASSO), random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN). Previous studies consistently showed that these methods yield the best prediction results57,58; the eAppendix in the Supplement describes the details for each approach used. Given that beneficiaries may have multiple opioid overdose episodes, we present the results from a patient-level random subset (ie, using one 3-month period with predictor candidates measured to predict risk in the subsequent 3 months for each patient) from the validation data for ease of interpretation. Episode-level performance was the same as the patient-level results.
To assess discrimination performance (ie, the extent to which patients who were predicted to be high risk exhibited higher overdose rates compared with those who were predicted to be low risk), we compared the C statistic (or area under the receiver operating curve) and precision-recall curves59 across different methods from the validation sample using the DeLong Test.60 Given that overdose events are rare outcomes and C statistics do not incorporate information about outcome prevalence, we reported other metrics, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio, negative likelihood ratio, number needed to evaluate (NNE) to identify 1 overdose episode, and estimated rate of alerts, to thoroughly assess the algorithms' prediction ability (eFigure 3 in the Supplement).61,62 To compare performance across methods, we presented and assessed these metrics at the optimized prediction threshold that balances sensitivity and specificity, as identified by the Youden index.63 Furthermore, because no single threshold is suitable for every purpose, we also presented these metrics at multiple other levels of sensitivity and specificity (eg, arbitrarily choosing 90% sensitivity) to enable risk-benefit evaluations of potential interventions that use different thresholds defining high risk.
On the basis of the distribution of individuals' estimated probability of an overdose event, we classified beneficiaries in the validation sample into low risk (predicted score below the optimized threshold), medium risk (score between the optimized threshold and top fifth percentile), or high risk (the top fifth percentile of scores, chosen according to clinical utility). We evaluated calibration plots (the extent to which the predicted overdose risk agreed with the observed risk) by the 3 risk groups.
To ensure clinical utility, we reported the predictors with the strongest effect. Because no standardized methods exist to identify individual important predictors from the DNN model, we reported the top 50 important predictors from the GBM and RF models. We also compared our prediction performance over a 12-month period with any of the 2019 Centers for Medicare & Medicaid Services opioid safety measures, which are meant to identify high-risk individuals or utilization behavior in Medicare.64 These simpler decision metrics were constructed from factors identified from previous studies using traditional approaches (eg, multivariate logistic regression). These measures included 3 metrics: (1) high-dose use, defined as higher than 120 MME for 90 or more continuous days; (2) 4 or more opioid prescribers and 4 or more pharmacies; and (3) concurrent opioid and benzodiazepine use for 30 or more cumulative days. In addition to using 3- and 12-month windows, we conducted a sensitivity analysis using a 6-month window in DNN to examine whether the prediction quality changes with different time horizons.65
We compared the patient characteristics by overdose status and by training, testing, and validation sample with unpaired, 2-tailed t test, χ2 test and analysis of variance, or corresponding nonparametric tests, as appropriate. We assessed correlations between 2 variables using Pearson correlation coefficient (r). Statistical significance was defined as 2-tailed P < .05.
All analyses were performed using SAS, version 9.4 (SAS Institute Inc); Python, version 3.6 (Python Software Foundation); and Salford Predictive Modeler software suite, version 8.2 (Salford System).
Patient Characteristics
Beneficiaries in the training (n = 186 686), testing (n = 186 685), and validation (n = 186 686) samples had similar characteristics and outcome distributions (approximately 63% were female, 82% were white, 35% had disabilities, and 41% were dual eligible; the mean [SD] age was 68.0 [14.5] years (eTable 5 in the Supplement). Overall, 3188 beneficiaries (0.6%) had at least 1 opioid overdose episode during the study period.
Prediction Performance of Machine-Learning Algorithms
Figure 1 summarizes 4 prediction performance measures of each model. The DNN (C statistic = 0.91; 95% CI, 0.88-0.93) and GBM (C statistic = 0.90; 95% CI, 0.87-0.94) algorithms outperformed the LASSO (C statistic = 0.84; 95% CI, 0.80-0.89), RF (C statistic = 0.80; 95% CI, 0.75-0.84), and multivariate logistic regression (C statistic = 0.75; 95% CI, 0.69-0.80) methods for predicting opioid overdose (P < .001). In addition, DNN and GBM had similar prediction performance, and DNN had the best precision-recall performance (Figure 1B), based on an area under the curve of 0.036. Sensitivity analyses including all the historical 3-month windows yielded similar results (eFigure 4 in the Supplement).
eTable 6 in the Supplement shows the prediction performance measures across different levels (90%-100%) of sensitivity and specificity for each method. At the optimized sensitivity and specificity, as measured by the Youden index, GBM had a sensitivity of 86.8%, specificity of 81.1%, PPV of 0.22%, NPV of 99.9%, NNE of 447, and 24 positive alerts per 100 beneficiaries. Similarly, at the optimized sensitivity and specificity, DNN had a sensitivity of 92.3%, specificity of 75.7%, PPV of 0.18%, NPV of 99.9%, NNE of 542, and 22 positive alerts per 100 beneficiaries (Figure 1C and D; eTable 6 in the Supplement). If sensitivity were instead set at 90% (ie, attempting to identify 90% of individuals with actual overdose episodes), GBM had a specificity of 72.3%, PPV of 0.16%, NPV of 99.9%, NNE of 631 to identify 1 overdose, and 28 positive alerts generated per 100 beneficiaries; DNN had a specificity of 77.0%, PPV of 0.19%, NPV of 99.9%, NNE of 525, and 23 positive alerts per 100 beneficiaries (eTable 6 in the Supplement). If specificity were set at 90% (ie, identifying 90% of individuals with actual nonoverdose), GBM had a sensitivity of 74.7%, PPV of 0.41%, NPV of 99.9%, NNE of 245, and 9 positive alerts per 100 beneficiaries; DNN had a sensitivity of 70.3%, PPV of 0.34%, and NPV of 99.9%, NNE of 294, and 10 positive alerts per 100 beneficiaries. Overall, DNN's prediction scores were highly correlated with the GBM's prediction scores (r = 0.73 for all patients, 0.73 for those without overdose episodes, 0.80 for those with overdose episodes; eFigure 5 in the Supplement).
Risk Stratification Using Predicted Probability
Using the GBM algorithm, 144 860 (77.6%) of the sample were categorized into low risk, 32 415 (17.4%) into medium risk, and 9411 (5.0%) into high risk for overdose (Table 1). Among all 91 beneficiaries with an overdose episode in the sample, 54 (59.3%) were captured in the high-risk group. Similarly, using the DNN algorithm, 9747 individuals (5.2%) were predicted to be high risk, capturing 56 overdose episodes (61.5%). Among the 142 180 individuals (76.2%) categorized as low risk, 99.99% did not have an overdose. Figure 2 depicts the actual overdose rate for individuals in each of the 3 risk groups. Across both the GBM and DNN models, those in the high-risk group had 7 to 8 times the risk of overdose compared with those in the lower-risk groups (observed overdose rate of GBM: 0.57% [high risk], 0.08% [medium risk], and 0.01% [low risk]; observed overdose rate of DNN: 0.57% [high risk], 0.07% [medium risk], and 0.01% [low risk]). Again, depicted is the negligible rate of overdose in the low-risk subgroups, representing more than three-quarters of the sample.
Figure 3 shows the most important predictors (n = 50) identified by the GBM model, such as total MME, history of any substance use disorder, mean daily MME, age, and Medicare disability status. eFigure 6 in the Supplement shows the most important predictors (n = 50) identified by the RF model.
Table 2 compares the performance of the DNN algorithm measures with the Centers for Medicare & Medicaid Services opioid safety measures. By targeting the high-risk group, DNN's algorithm captured approximately 90% of 297 individuals with actual overdose episodes (NNE = 56) in a 12-month period, with 14 917 (8.9%) of the overall cohort being misclassified as having overdose, whereas the Centers for Medicare & Medicaid Services measures captured 30% of 297 individuals actual overdose episodes (NNE = 108), with 5.51% of the overall cohort being misclassified as overdose. The GBM and DNN algorithms performed similarly (eTable 7 in the Supplement). Sensitivity analyses using 6-month windows yielded similar C statistic with an improved PPV compared with using 3-month windows (eg, C statistic = 0.89; 95% CI, 0.87-0.90; PPV = 0.36% in DNN).
Using national Medicare data, we developed machine-learning models with strong performance for predicting opioid overdose. The GBM and DNN models achieved high C statistic (>0.90) for predicting overdose risk in the subsequent 3 months after initiation of treatment with prescription opioids and outperformed traditional classification techniques. As expected in a population with very low prevalence of the outcome, the PPV of the models was low; however, these algorithms effectively segmented the population into 3 risk groups according to predicted risk score, with three-quarters of the sample in a low-risk group with a negligible overdose rate and more than 90% of individuals with overdose captured in the high- and medium-risk groups. The ability to identify such risk groups has important potential for policymakers and payers who currently target interventions based on less accurate measures to identify patients at high risk.
We identified 7 previously published studies of opioid prediction models, each focused on predicting a different aspect of opioid use disorder and not applying advanced machine learning. The studies predicted a 12-month risk of opioid use disorder diagnosis using private insurance claims13,22; 2-year risk of clinical, electronic medical record–documented problematic opioid use in a primary care setting24; 12-month risk of overdose or suicide-associated events using data from the Veterans Health Administration 31; 6-month risk of serious prescription opioid–induced respiratory depression or overdose using data from the Veterans Health Administration and claims data from Insurance Management Services private insurance25,30; and 2-year risk of fatal or nonfatal overdose using electronic medical record data.28 These studies had several key limitations, including use of case-control designs unable to calibrate to population-level data with the true incidence rate of overdose; measuring predictors at baseline rather than over time; capturing only the first overdose episode; inability to identify complex or nonintuitive relationships (interactions) between the predictors and outcomes; and having suboptimal prediction performance (with a C statistic of up to 0.72 in non–case-control designs). The present study overcomes these limitations using a population-based sample and machine-learning methods. To our knowledge, this study is the first to predict overdose risk in the subsequent 3-month period after initiation of treatment with prescription opioids as opposed to 1-year or longer period.
The extant literature in predicting health outcomes often focuses on C statistics rather than the full spectrum of prediction performance. This study found high C statistics (>0.90) from machine-learning approaches. However, although opioid overdose represents a particularly important outcome, it is a rare outcome, especially in the Medicare population. Relying on C statistics alone may lead to overestimating the advantages of a prediction tool or underestimating the costs of clinical resources involved. For a preimplementation evaluation of a clinical prediction tool, it is recommended that researchers report sensitivity and at least 1 other metric (eg, PPV, NNE, or estimated alert rate) to present a more complete picture of the performance characteristics of a specific model.59,61 In this study, the NNE value using DNN and GBM algorithms is similar to other commonly used cancer screening tests, such as annual mammography to prevent 1 breast cancer death (NNE = 233-1316, varying by subgroups with different underlying risk).66
Unlike sensitivity and specificity, which are properties of the test alone, the PPV and NPV are affected by the prevalence of the outcome in the population tested. Low outcome prevalence leads to low PPV and high NPV, even in tests with high sensitivity and specificity, and could limit the clinical utility of a prediction algorithm such as ours because of false-positives. Other tests with good discrimination have low PPV because of overall prevalence, including trisomy 21 screening in 20- to 30-year-old women (prevalence of approximately 1:1200),67 with a PPV of 1.7% at a test with sensitivity higher than 99% and specificity higher than 95%. Despite the low PPV in this study, our risk stratification strategies may more efficiently guide the targeting of opioid interventions among Medicare beneficiaries compared with exisiting measures. This strategy first excludes most (approximately 75%) prescription opioid users with negligible overdose risk from burdensome interventions like pharmacy lock-in programs and specialty referrals. Targeting medium- and/or high-risk groups can capture nearly all (90%) overdose episodes by focusing on only 25% of the population, which greatly frees up resources for payers and patients. For those in the high- and medium-risk groups, although most will be false-positives for overdose given the overall low prevalence, additional screening and assessment may be warranted. Although certainly not perfect, these machine-learning models allow interventions to be targeted to the small number of individuals who are at greater risk, and these models are more useful than other prediction criteria that have considerably more false-positives.
The study has important limitations. First, patients may obtain opioids from nonmedical settings, which are not captured in claims data. Second, this study captured overdose episodes in medical settings and missed overdose episodes that occurred outside of medical settings, which are not captured in claims data. Third, the study relied on administrative billing data that lacked laboratory results and sociobehavioral information. This limitation can be addressed in the future with more robust linked data. In addition, although the study was novel in measuring overdose risk in the subsequent 3 months after initiation of prescription opioids, it used older data with complete claims capture; translation into real-time risk scores can be complicated by the lag in claims completion after the time of visit. Fourth, our focus was on predicting opioid overdose, and not opioid misuse, which is difficult to measure solely from claims data. Fifth, prediction algorithms and findings derived from the fee-for-service Medicare population may not generalize to individuals enrolled in Medicare Advantage plans or to other populations with different demographic profiles or programmatic features. However, the models may have better prediction performance in settings in which overdose is less rare (eg, Medicaid).
This study demonstrates the feasibility and potential of machine-learning prediction models with routine administrative claims data available to payers. These models have high C statistics and good prediction performance and appear to be valuable tools for more accurately and efficiently identifying individuals at high risk of opioid overdose.
Accepted for Publication: February 1, 2019.
Published: March 22, 2019. doi:10.1001/jamanetworkopen.2019.0968
Correction: This article was corrected on July 5, 2019, to fix data errors in Methods, Results, Table 1, Figure 2, and eAppendix and eTable 7 in the Supplement.
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Lo-Ciganic W-H et al. JAMA Network Open.
Corresponding Authors: Wei-Hsuan Lo-Ciganic, PhD, Department of Pharmaceutical Outcomes & Policy, College of Pharmacy, University of Florida, PO Box 100496, 1225 Center Dr, HPNP Bldg, Room 3338, Gainesville, FL 32610-0496 (wlociganic@cop.ufl.edu).
Author Contributions: Drs Lo-Ciganic and Huang had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Lo-Ciganic, Zhang, Kwoh, Donohue, Cochran, Gordon, Malone, Gellad.
Drafting of the manuscript: Lo-Ciganic, Gellad.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Lo-Ciganic, Huang, Zhang, Weiss, Wu.
Obtained funding: Lo-Ciganic, Zhang, Kwoh, Donohue, Cochran, Gordon, Malone, Gellad.
Administrative, technical, or material support: All authors.
Supervision: Lo-Ciganic, Gellad.
Conflict of Interest Disclosures: Drs Lo-Ciganic, Huang, Zhang, Kwoh, Donohue, Cochran, Gordon, Malone, Kuza, and Gellad reported grants from the National Institutes of Health (NIH)/National Institute on Drug Abuse during the conduct of the study. Dr Lo-Ciganic reported grant support from the Pharmaceutical Research and Manufacturers of America Foundation during the conduct of the study. Dr Wu reported grants from the NIH during the conduct of the study. Dr Kwoh reported grant funding from AbbVie and EMD Serono; grants and personal fees from EMD Serono; and personal fees from Fidia, Regulus, GSK, Taiwan Liposome Company Inc, Kolon Tissue Gene, and Express Scripts outside of the submitted work. No other disclosures were reported.
Funding/Support: This study was supported by grant R01DA044985 from the NIH/National Institute on Drug Abuse and by a 2017 Pharmaceutical Research and Manufacturers of America Foundation Research Starter Award (Dr Lo-Ciganic).
Role of the Funder/Sponsor: The funders or sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The views presented here are those of the authors and do not necessarily represent the views of the US Department of Veterans Affairs or the US government.
Get the latest from JAMA Network Open


Truck unit Iveco Stralis with manipulator – Max. power – 476 hp max. Speed – 81 km / h. Lighting, interior lights, speedometer, tachometer, fuel gauge, temperature, indicator lights and turn signals, mirrors, swing bridge, camera operator, marks, dust, animation supports. Washable.
Semi-wood chipper – Bunker 10,000 liters. Lighting, work indicators, the level of the hopper, swivel bridges, tracks, dust, animation (spotlights, console lid, hood, motor, bearing, trunk, drums). Washable.


Black and White Crappie up to 14"
"The trophy has been busy in Arkansas catching monster slabs!!!"
Black and White Crappie up to 14.5"
Combo trip with Action Fishing Trips featuring Jim Erickson and Keith Sutton.
"Oops sorry, it's 'sacalait' not 'crappie' to these boys."
"Just a little late morning Lake Hamilton scouting with my wife and younger son."
"Check out the fish'in shirts on the boys, look'in good!!!"
The longest crappie ever caught from my boat.
Didn't anyone tell you, you can't catch crappie like that in the middle of the summer.
Way to go Mark. Wow, that slab's going on the wall!!!
Another picture can be viewed on the Wallhangers page.
Black and White Crappie up to 13"
Black and White Crappie up to 15.25"
Black and White Crappie up to 13.5"
Black and White Crappie up to 16" - 1lb.15oz. and 2lbs.3oz.
Black and White Crappie up to 16.25" / 2.5lbs.
Crappie up to 16.5" / 2.75 lbs.


text:
Chinese FM: US is Trying to use Nuclear Non-Proliferation Agreements as Geopolitical Tool Against Russia and China
EurasiaAnglo 5China
By Drago Bosnic	 On Apr 25, 2020
BEIJING/WASHINGTON, D.C. – The United States tries to use international cooperation in nuclear power as its geopolitical instrument, Chinese Foreign Ministry Spokesman Geng Shuang commented on Friday on the US Department of Energy's strategy, which mentions the need for ousting Russia and China from international nuclear power markets.
"We've taken note of this report. It argues that in contrast to the United States, China and Russia fail to observe non-proliferation requirements, which is absolutely not true. The United States uses a variety of pretexts for upsetting normal cooperation between other countries in nuclear power," Geng said, according to TASS.
"US officials have been saying in public that they will use cooperation in nuclear power as a geopolitical instrument," he added.
"China officially dismisses such politicization [of cooperation in nuclear power] and it will never agree with such accusations. Aware of its responsibility China will be developing nuclear power and also contributing to stable international cooperation in this field," Geng concluded.
Earlier the US Department of Energy published a strategy in which it proclaimed as one of its main tasks the ousting of Russia and China from the international markets of nuclear power and technologies. Also, it voiced the intention to impede cooperation by Moscow and Beijing with East European and African countries in this field.
The US violates its own international obligations by suspending contributions to the World Health Organization (WHO), undermining global cooperation, the Chinese Foreign Ministry spokesman said on Friday, commenting on the statement by US Secretary of State Mike Pompeo regarding further suspension of WHO funding.
"Attacks on the WHO and attempts to discredit it by the United States are baseless," the diplomat said.
"The US is paying its dues to the WHO as a legal entity, and it must adhere to its obligations as a member state of the organization. The suspension of funding by Washington is a violation of its fundamental obligations, namely before other members of the organization. Such actions undermine global cooperation and may lead to serious consequences," the spokesman stressed.
"Supporting the WHO can save many lives and stop the spread of the pandemic," Geng Shuang concluded.
On Wednesday, Pompeo said on Fox News that the US might suspend its funding of the WHO indefinitely.
ChinaGeopoliticsnuclear arsenalRussiaUS Imperialism
Drago Bosnic 1419 posts 0 comments
IRAQ: Official Warns of US Plot to Transfer ISIS Terrorists From Syria to Iraq's Al-Anbar
WHO Ramps Up Covid-19 Alarms: People with coronavirus are not protected from reinfection
Woman killed during Capitol riot actually urged others to stand down
Up to 65% Increase in Deaths Among 18-49 Year Olds in the U.S. During 2021, the Year…
The China Distraction and U.S. Destabilization

redpajama_set_name: RedPajamaCommonCrawl


The H-1B cap season is around the corner, and the premium processing for previously filed H-1B petitions has commenced on March 12th, 2019. USCIS has started the premium processing only for the H-1B petitions received on or before Dec 21st, 2018.
The usual processing time for H-1B petitions is around 5-6 months, which is very long. So USCIS has started the premium processing service to minimize the time taken for processing the employment-based petitions. Any petition filed under premium processing will take only 15 calendar days for completion of the process.
This 15-day period will begin once the USCIS receives the petition at the processing center. Upon processing the petition USCIS will send any one of the following as a response to the petition within 15 days.
In case you have received an RFE or NOID, you are responsible for sending an RFE/ NOID response to the USCIS for further processing. A new 15-day period will begin on the day USCIS receives the response to the RFE.
USCIS charges a huge amount of fee for the premium processing of your petitions. If USCIS fails to provide a response within the specified 15 days, it would refund your fee, but your petition will still be eligible for the premium processing.
USCIS has suspended the premium processing of H-1B petitions in the previous year in order to clear all the pending petitions. Recently USCIS has resumed this process for all the H-1B petitions filed on or before Dec 21st, 2018. This latest update from the USCIS has brought in a little hope for the suspended the premium processing of H-1B aspirers from different countries. However, there is no update regarding the premium processing of the petitions received after this date.


text:
Q: Loop through array of arrays of string with spaces I'm trying to loop through an array that contains other arrays and these arrays consist of strings with spaces. The problem is that I can't seem to preserve the spacing in the string. The string with spaces are either divided into multiple items if I change IFS to \n or all the elements of the array are seen as 1 item if I leave IFS unchanged here's some sample code:
#!/bin/sh
low1=("AA  QQ" "BB  LL")
low2=("CC" "DD")
low3=("EE" "FF")
high=(low1 low2 low3)

for high_item in ${high[@]}
do
   eval arrayz=\${$high_item[@]}
   #IFS=$'\n'
   for item in $arrayz
   do
      echo $item
   done
done

Output:

AA
QQ
BB
LL
CC
DD
EE
FF

As you can see the elements "AA QQ" and "BB LL" have been split.
If I uncomment the line that sets IFS to \n I get the following:

AA QQ BB LL
CC DD
EE FF

Now "AA QQ" and "BB LL" are concatenated!
Is there anyway I can preserve these elements just as they original are...I need the output to look like that:

AA QQ
BB LL
CC
DD
EE
FF

A: I think you meant that the output should look like:
AA  QQ
BB  LL
CC
DD
EE
FF

i.e.:
${low1[0]}
${low1[1]}
${low2[0]}
${low2[1]}
${low3[0]}
${low3[1]}

This could be accomplished using:
#!/bin/bash

low1=("AA  QQ" "BB  LL")
low2=("CC" "DD")
low3=("EE" "FF")
high=(low1 low2 low3)

for high_item in ${high[@]}
do
    x="${high_item}[@]" # -> "low1[@]"
    arrays=( "${!x}" )

    #IFS=$'\n'
    for item in "${arrays[@]}"
    do
        echo "$item"
    done
done

And please always use #!/bin/bash for bash scripts.
Explanation: ${!x} is indirect variable expansion. It evaluates to the value of variable with a name contained in $x.
For our needs, x needs to have the [@] suffix for array expansion as well. Especially note that it is x=${high_item}[@] and not x=${high_item[@]}.
And you have to evaluate it in array context; otherwise, it wouldn't work as expected (if you do arrays=${!x}).
Ah, and as final note: IFS doesn't make any difference here. As long as you are working on quoted arrays, IFS doesn't come into play.

A: Replace eval with indirect parameter expansion, and you'll get what I think you want (although it doesn't match your current given output:
for high_item in "${high[@]}"
do
    arrayz="$high_item[@]"
    # arrayz is just a string like "low1[@]"
    for item in "${!arrayz}"
    do
        echo $item
    done
done

Note the need to quote the array expansion in the inner loop to preserve the whitespace in elements of low1.

redpajama_set_name: RedPajamaStackExchange


text:
The 1955 William & Mary Indians football team represented William & Mary during the 1955 college football season.

Schedule

NFL Draft selections

References

William and Mary
William
William & Mary Tribe football seasons

redpajama_set_name: RedPajamaWikipedia


text:
Lily Arkens, Staff Writer, Advanced Journalism
Notre Dame's hockey team is currently undefeated at 14-0-0. They are also ranked number one in the state and in the Fox River Classic Conference.
Notre Dame won over rival BayPort in overtime with a score of 2-1, giving them the Boys Game of the Week title under Wisconsin Prep Hockey.
"We have a great group of leaders. We also have a bunch of selfless players that are all holding each other accountable for improving daily. I am more proud of our group for the way they are handling success and preparing for big games knowing we are a targeted and circled team each season," said Head Coach Cory McCracken.
McCracken has been in this program for 16 years and has "changed and adapted the program" over the years as a coach.
This year's senior captains are Hunter Bill, Joseph Gertbitz, Brenden Gruber, Quinn Koszarek and Michael Mcintee.
"I have noticed that one strength of our team is the brotherhood we have developed. Even with the first-year players, we have welcomed a great group of kids who fit right in," said Gerbitz.
Koszarek's leadership skills have changed in the way he views things and the way he treats his teammates.
"You can say one thing to one person, but then you might not be able to say it to another teammate," said Koszarek.
Sophomore Sam Kappell has scored the most goals this year, leading the Tritons with a total of 17. Sophomore Drew Shock has added his fair share to the scoring by assisting on 23 of the Tritons goals, and senior Hunter Bill is leading the team in points with an outstanding 29.
Kappell won Green Bay Press-Gazette Athlete of the Week in early December and described it as "a great feeling."
Sophomore Anton Widas is leading the team with a Goalie Save Percentage of .968, which converts to 96%. Goalie Save Percentage is to measure the amount of saves a goal has by the number of shots.
"I've built better relationships with some teammates than others, like Lleyton Jaschke. Although he is my number one competitor, I've gotten really close to him, and now he is a very good friend of mine. It is very important to have a relationship with the whole team since hockey is very much a team game," said Widas.
Widas praised Jaschke, saying, "Lleyton has been playing well this year and doing very well against the higher-skilled teams."
McCracken has evolved his coaching style over the past several years to keep it on the modern side.
"Our players keep evolving and want to play the modern game. The modern game is puck possession and is played with skill and speed. We try to incorporate what we see in the juniors, college and pro games," said the coach.
This team plays an up-tempo game style that is fun for them. McCracken likes to make it an environment where the kids are excited to play and come to the rink every day. When the team has success he notices that individual players also have success.
"The success this group has is not by accident. If someone could show me a harder-working or more committed group of young men, I would be very surprised," said McCracken.
Lily Arkens, Staff Writer
Lily Arkens is a sophomore at NDA. She is Football Team Manager and during the summer in her free time, she loves to golf with her grandpa every Friday....
Dominic D'Angelo Discovers his Passion for Sports Broadcasting
NDA's Trista Fayta to Play Basketball at Illinois State
Gracie Grzesk to Play Basketball at UW-Madison
Fall Sports Season Wraps Up, Winter Season Looks Promising for Basketball, Hockey
Junior Girls Win PowderPuff Trophy, Bragging Rights
Grace Durkin: FRCC's Top Golfer Decides to Play at Loyola in Chicago


text:
1-minute hike: Mansell Mountain on Mount Desert Island
April 9, 2013 1-Minute Adventures
Difficulty: Moderate. The hike hovers around 2 miles in length, slightly more and slightly less, depending on the route you take.
How to get there: Drive to Mount Desert Island on Route 3. After crossing the causeway, stay to the right and follow Route 102 toward Somesville and Southwest Harbor. After reaching Southwest Harbor, turn right onto Seal Cove Road, the bang another right onto Long Pond Road. Drive to the end of Long Pond Road to a parking area on the shore of Long Pond, just past the water pumping station. You will start on Long Pond Trail; the trailhead is to the left of the water pumping station, if you are facing the lake. (Two other trails that lead to Beech Mountain leave from this parking area.)
Information: Mansell Mountain, which rises 949 feet above sea level on Mount Desert Island, can be explored by a number of hiking trails in a section of Acadia National Park located near the town of Southwest Harbor.
A popular route is to start on the Long Pond Trail, take the second left at about 0.2 mile to hike up the Perpendicular Trail to the summit, and descend the mountain on the Razorback Trail (located at the termination of the Perpendicular Trail, past the junction with the Mansell Mountain Trail. The Razorback Trail will enter the forest and end at the Gilley Trail. A left leads to the Cold Brook Trail, which will bring you to Long Pond Trail. Turn right and head back to the parking area. The loop is about 2.5 miles long, according to www.mainetrailfinder.com.
Another option (and the one I took), is to descend the mountain via Mansell Mountain Trail, a steep trail that will lead you quickly to the base of the mountain and Cold Brook Trail, which will lead you back to Long Pond and the parking area.
There are also trails that link to Great Notch and the nearby Bernard Mountain.
Mansell Mountain, named after Sir Robert Mansell, has a confusing history. Sir Robert Mansell, born in 1565, was married to Miss Roper, a maid of honor in Queen Anne's household, according to "Mountains of Maine: Intriguing Stories Behind Their Names" by Steve Pinkham. After being knighted in 1596, he worked his way up the line to become commander of the Royal Navy. In 1622, he received a deed to Mount Desert Island for a "note of one hundred and ten pounds." Samuel de Champlain had already named the island, but Mansell claimed it for England and renamed it Mount Mansell Island, which the colonists of Massachusetts called it for 70 years until the name Mount Desert Island was restored.
In the 1800s, the Reverend Benjamin DeCosta visited the island and renamed Brown Mountain, near Somes Sound, to Mount Mansell. Then George Dorr came along and renamed it Norumbega Mountain (the name the mountain retains today). Later, he bestowed the east peak of Western Mountain with the name Mansell Mountain (the name it retains today), according to Pinkham.
It was to this peak that the Perpendicular Trail was built in 1933-34 by the Civilian Conservation Corps, according to Mainetrailfinder.com. Though there are several trails on the slopes of Mansell Mountain, the best-known trail is the Perpendicular Trail, which includes more than 300 granite steps. Dogs are not allowed on this trail due to a steep section that includes ladder and iron rungs.
The summit of Mansell Mountain is wooded, but a few great outlooks are located along the Perpendicular Trail, Mansell Mountain Trail and Razorback Trail.
To use the trails and other resources of Acadia National Park, the first national park east of the Mississippi, visitors must pay an entrance fee upon entry May through October. Pass cost varies depending on several factors. For information about park passes, visit www.nps.gov/acad/planyourvisit/index.htm.
For general information, visit www.nps.gov/acad/index.htm, call 288-3338 or e-mail acadia_information@nps.gov.
Personal note: No one had ever suggested this mountain to me, and I'm surprised. The Perpendicular Trail is a treat — stone staircases winding around boulders and through rockslides. One portion of the trail climbs near flowing water that creates tiny waterfalls at it travels down the mountain, and another section of the trail climbs through a rock slide. While there are rock stairs on other mountains in the park — Dorr Mountain and Huguenot Head, to name a couple — Mansell Mountain is less likely to be crowded during the summer, as it lies in Southwest Harbor, on the less-traveled side of Mount Desert Island.
About Aislinn Sarnacki
← Don't forget to attend Bangor area trails meeting Thursday	Where in the woods am I? Sunkhaze Meadows →


text:
For the Summer Olympics, there are 29 venues that have been or will be used for basketball. The first official venue was outdoors under the auspices of the International Basketball Federation although the game was played indoors. Being played on tennis courts of clay, the final was held in a heavy rain which turned the courts into mud. Since those games, the basketball games have been played indoors.

References

Venues

Basketball
Olympic venues


text:
Home > Canon > PC Printers Series > PC795
Canon FX-6 Premium Compatible Black Laser Toner Cartridge (1559A002)
The compatible toner cartridge replaces Canon FX-6 Toner Cartridge and is compatible with Canon FAX L1000 and LaserClass 3170 3175 series printers. The product yields 6,000 pages at 5% coverage. Our compatible Canon units come with a 100% satisfaction guarantee.
Canon E-20 Premium Compatible Black Laser Toner Cartridge (1492A002AA)
The compatible toner cartridge replaces Canon E20 Toner Cartridge and is compatible with Canon FC 200 220 300 310 330 336 500 and PC 140 150 160 170 210 300 310 320 325 330 400 420 425 428 430 530 550 700 710 720 730 735 740 745 760 770 775 785 790 795 800 860 880 890 900 920 921 940 950 980 981 series printers. The product yields 2,000 pages at 5% coverage. Our compatible Canon units come with a 100% satisfaction guarantee.
The compatible toner cartridge replaces Canon E40 Toner Cartridge and is compatible with Canon FC 200 220 300 310 330 336 500 and PC 140 150 160 210 300 310 320 325 330 400 420 425 428 430 530 550 700 710 720 730 735 740 745 760 770 775 785 790 795 800 860 880 890 900 920 921 940 950 980 981 series printers. The product yields 3,000 pages at 5% coverage. Our compatible Canon units come with a 100% satisfaction guarantee.
Canon PC795 Printer Review:
The Canon PC795 laser copier is designed for the home or small office user. The Canon PC795 laser copier is a very economical copy machine. It comes with Energy Star compliant and requires no warm up time before making copies. The Canon PC795 also comes with auto-shutoff feature to further help in saving energy costs.
The Canon PC795 measures 13.3 inches X 19.1 inches X 17.3 inches and weighs about 55 pounds. It features a 30-page automatic document feeder for quickly scanning multiple pages, as well as a a flatbed scanning surface for media that will not fit into the feeder.
Paper capacity is 250 sheets in the main drawer with a 50-sheet bypass tray to accommodate alternate media types or sizes for the PC795 model. The unit supports reduction and enlargement from 70% to 141%.
The PC795 uses Canon's single cartridge system, which combines the toner and the drum into a single copier toner cartridge. There are three toner models for the Canon PC795 machine. The FX-6 comes with 6,000 page yield at 5% coverage, the E40 comes with 3,000 page yield at 5% coverage, and the E20 comes with 2,000 page yield at 5% coverage. The use of compatible cartridges does not void the printer warranty.
The Canon PC795 has a copying speed of about 12 pages per minute. Overall, the Canon PC795 is a solid copy machine with enough paper capacity and speed to be shared among a small group of users in a small office.


text:
Shaw Contract announces winners of its 2020 Design Awards
Dezeen staff | 9 December 2020
Dezeen promotions: US flooring designer Shaw Contract has announced the winners of its 2020 Design Awards, which include an office for Adidas in Hong Kong and a school for homeless children in the USA.
Shaw Contract recognised seven winners and a People's Choice award in the 15th edition of its Design Awards, which celebrates impactful living, working, learning and healing spaces around the world.
Each of the winners was awarded with a $5,000 charitable donation in their studio's name to an organisation of their choice.
Top: The Suave office was the People's Choice winner. Above: Envisione Studio's design for the Adidas Global Sourcing Headquarters in Hong Kong was one of seven award winners. Photo by Roy Esterhuysen
"The Awards place a spotlight on a diverse range of talents from across the globe, celebrating design in action and purposefully rewarding the innovative and truly inspiring work being done by the architecture and design community," said Shaw Contract.
"Design defines and influences action, healing, idea generation and socialization. It gives us and our experiences meaning and purpose," it continued.
"Shaw Contract is obsessed with the environments that define the human experience, and celebrating those spaces that make a smarter impact for our world is an imperative."
Core Collective Architects' design for the Eternal Women's Health facility was one of the winners. Photo by Adam Gibson Photographer
The seven winners were selected from 43 category finalists, which were chosen from hundreds of entries by a panel of judges from around the world.
Alongside these winners, the public also chose the winner of the People's Choice award: an office in Kuala Lumpur designed by Box Design Studio.
The Suave office by Box Design Studio was the public's favoutite project. Photo is by Lawrence Choo
Designed for reinsurance company Hannover Re, The Suave office combines areas of open-plan workspaces with a series of formal and informal meeting spaces.
"Oftentimes, an employee's work requires different brain states between active group work and solitude think time," said the studio.
"From huddle rooms to thinking pods, the variety of spaces enables quick switching between conversation and concentration."
The headquarters for Shui On Land was one of two projects designed by Robarts Spaces to win. Photo is by Wangyi
International architecture and design studio Robarts Spaces won awards for two of its office projects: a workspace for International Consulting Firm and a headquarters for Shui On Land.
Two further office projects received awards, including Envisione Studio's design for the Adidas Global Sourcing Headquarters in Hong Kong and Escape from Sofa's workspace for the Onur Group in Ankara.
Positive Tomorrows School for Homeless Children was also a winner. Photo is by Simon Hurst Photography
MA Architecture won an award for its colourful Positive Tomorrows School for Homeless Children in Oklahoma City.
Designed to have the feel of a home rather than of an institution, the school features different spaces named the Family Room, Living Room, and Backyard.
Japara "The Regent", Aged Care Facility was one of two projects in Australia to win awards. Photo is by Tatjana Plitt
The final two awards were presented to projects in Australia. The Japara "The Regent", Aged Care Facility in Melbourne by Spowers was designed to connect residents to the surrounding nature reserve as part of an overall strategy to enhance the life of the residents.
In Hobart, Core Collective Architects created a homely, crafted and functional tenancy fit-out for the Eternal Women's Health, which is a healthcare facility for women at all stages of life.
Read more about all the winners on Shaw Contract's Design Awards website.
Partnership content
This article was written by Dezeen for Shaw Contract as part of a partnership. Find out more about Dezeen partnership content here.
More promotions
Competition: win a Flo monitor arm by Colebrook Bosson Saunders
Ressence's Type 3BBB is a monochromatic oil-filled watch
Design should "be part of life and not separate from it" ...
Franchising gives architects "better balance between home ...
Call for final degree project entries to IE School of ...
Kew and the Museum of Architecture launch treehouse design ...
Rieder showcases concrete facade panels in Germany's largest ...
Thermo-ash cladding gives tone and texture to Dutch forest house


The first step in a successful Adwords Advertising Campaign is to set and measure appropriate goals. There are a few ways to set goals and we recommend using a combination of the methods we will be discussing.
What is an Analytics/Adwords Goal?
Defining a goal is a fundamental element of any online marketing plan. Without this information, it's almost impossible to evaluate the effectiveness of your online business and marketing campaigns.
Only you can choose the right goal for your business but a good place to start is to find out what your desired end result is.
Let's assume you have an ecommerce business that sells widgets. A purchase is your desired end result however, this isn't your only goal.
Discovering the path of purchase is often times just as important as the actual purchase. Once you discover what your ideal customer does on your website, you can set goals for each step of the conversion path and find ways to get more website visitors to take that exact path.
Let's see an example of that with our widget business.
A website visitor reaches your home page. Navigates to your about page. Then to a specific product page. Followed by your return policies and finally checks out.
With this information you can build each step of this path and redesign/test your website to get the most visitors possible to take this path. For this tutorial, we won't show you how to build out your conversion funnel. To keep things simple, we will show you how to create simple goals and get comfortable with goal tracking.
In Analytics, go to Admin, select the property you want and select Goals from the far right column.
Choose a name you can easily identify such as Reached checkout, Made a Purchase, Thank You Confirmation, etc. You will be given three options, the most common type of goal is a Destination Goal and hit continue.
A destination is when someone reaches a particular website page. This is commonly used for eCommerce (order confirmation page), contact forms (thank you page) and/or a download (document.pdf). For example, if you want to track how many times your contact form is completed, you can copy the URL of the thank you page given to the user after hitting submit. You would then paste the URL in the destination URL. Selecting a value is optional. The funnel feature is an advanced option.
Verify your goal and hit submit or continue.
You can make several goals for important conversions on your website. After completing this you can import your Analytics goals into your Adwords account.


The student visa for Canada is document based and in most cases an interview is not required. Applicants need to show that they can pay for their entire education whilst in the country, including living expenses. The funds shown should be in movable or liquid assets and a detailed cover letter explaining your reasons to study in Canada, your plans after graduation and your financial plan should be included. Certain health requirements must also be fulfilled when you applying for a student visa to Canada. The visa processing time for Canada can be 1-2 months and hence students should apply early.
If you plan to apply to a Canadian college through the Student Partners Program (SPP), there are additional recommended steps that you can take. For example, you may want to show evidence of purchase of a Special Guaranteed Investment Certificate (GIC) from a participating Canadian financial institution of $10,000 CAD to cover the living expenses for the first year in Canada.