text:
Cyberpharmacies (29) Medicine 2.0: Social Media, Open, Participatory, Collaborative Medicine (1228) Crowdsourcing and Mechanical Turks (84)
Crowdsourcing Black Market Prices For Prescription Opioids
Nabarun Dasgupta 1 ; Clark Freifeld 1, 2 ; John S Brownstein 1, 2, 3 ; Christopher Mark Menone 1 ; Hilary L Surratt 4 ; Luke Poppish 5 ; Jody L Green 5 ; Eric J Lavonas 5 ; Richard C Dart 5, 6
Article Authors Cited by (42) Tweetations (51) Metrics
Nabarun Dasgupta1, MPH, PhD;
Clark Freifeld1,2, MS;
John S Brownstein1,2,3, PhD;
Christopher Mark Menone1, MESc;
Hilary L Surratt4, PhD;
Luke Poppish5, MBA;
Jody L Green5, PhD;
Eric J Lavonas5, MD;
Richard C Dart5,6, MD, PhD
1Epidemico, Boston, MA, United States
2Boston Children's Hospital, Children's Hospital Informatics Program, Harvard Medical School, Boston, MA, United States
3Boston Children's Hospital, Department of Pediatrics, Harvard Medical School, Boston, MA, United States
4Nova Southeastern University, Fort Lauderdale, FL, United States
5RADARS System, Rocky Mountain Poison and Drug Center, Denver Health, Denver, CO, United States
6Department of Emergency Medicine, University of Colorado School of Medicine, Aurora, CO, United States
Nabarun Dasgupta, MPH, PhD
Epidemico
268 Newbury Street
Phone: 1 9192603808
Fax:1 8666571107
Email: nabarund@gmail.com
Background: Prescription opioid diversion and abuse are major public health issues in the United States and internationally. Street prices of diverted prescription opioids can provide an indicator of drug availability, demand, and abuse potential, but these data can be difficult to collect. Crowdsourcing is a rapid and cost-effective way to gather information about sales transactions. We sought to determine whether crowdsourcing can provide accurate measurements of the street price of diverted prescription opioid medications.
Objective: To assess the possibility of crowdsourcing black market drug price data by cross-validation with law enforcement officer reports.
Methods: Using a crowdsourcing research website (StreetRx), we solicited data about the price that site visitors paid for diverted prescription opioid analgesics during the first half of 2012. These results were compared with a survey of law enforcement officers in the Researched Abuse, Diversion, and Addiction-Related Surveillance (RADARS) System, and actual transaction prices on a "dark Internet" marketplace (Silk Road). Geometric means and 95% confidence intervals were calculated for comparing prices per milligram of drug in US dollars. In a secondary analysis, we compared prices per milligram of morphine equivalent using standard equianalgesic dosing conversions.
Results: A total of 954 price reports were obtained from crowdsourcing, 737 from law enforcement, and 147 from the online marketplace. Correlations between the 3 data sources were highly linear, with Spearman rho of 0.93 (P<.001) between crowdsourced and law enforcement, and 0.98 (P<.001) between crowdsourced and online marketplace. On StreetRx, the mean prices per milligram were US$3.29 hydromorphone, US$2.13 buprenorphine, US$1.57 oxymorphone, US$0.97 oxycodone, US$0.96 methadone, US$0.81 hydrocodone, US$0.52 morphine, and US$0.05 tramadol. The only significant difference between data sources was morphine, with a Drug Diversion price of US$0.67/mg (95% CI 0.59-0.75) and a Silk Road price of US$0.42/mg (95% CI 0.37-0.48). Street prices generally followed clinical equianalgesic potency.
Conclusions: Crowdsourced data provide a valid estimate of the street price of diverted prescription opioids. The (ostensibly free) black market was able to accurately predict the relative pharmacologic potency of opioid molecules.
J Med Internet Res 2013;15(8):e178
doi:10.2196/jmir.2810
opioids; black market; economics; drug abuse; surveillance; crowdsourcing; Internet; Silk Road; StreetRx; RADARS System; police; law enforcement
The United States has a high level of concern with the diversion and public health consequences associated with the nonmedical use of prescription opioid analgesics [1-3]. Prescription opioid analgesics diverted from the pharmaceutical supply chain may end up being resold in open-air markets, in clubs and bars, or more subtly between friends and relatives [4].
Street price data have many applications. They are routinely collected by law enforcement agencies, which rely on accurate street prices for agents to be credible buyers or sellers in undercover operations. On a policy level, the Drug Enforcement Administration cited street price data to justify assigning buprenorphine to Schedule III, a lesser category of regulation than methadone, oxycodone, and morphine [5]. Street prices for cocaine and heroin have been used as indicators of intervention impact in public policy [6,7], including as inputs for modeling the impact of policy decisions, understanding the profits and risks in the drug trade [8,9], informing debates on prohibition [10], evaluating the impact of interdiction, informing the timing of public health efforts [11,12], and understanding the impact of globalization and economic recession on drug street prices [13-15]. Behavioral economic studies in controlled settings have been used extensively in the last decade to explain and predict human behaviors associated with addictive disorders, focusing on impulse control (discounting) and the relative likeability of substances [16,17]. In pharmaceutical development, recent guidelines for pre-approval abuse liability studies for new pharmaceuticals increase reliance on laboratory behavioral economic assessments with drug users to determine differences in willingness to pay [18], but these data from controlled settings are not connected to real-world black market street price data in post-marketing surveillance. Finally, behavioral economics research has become an important tool in understanding decision making, drug dependence, and treatment choices for a variety of reinforcing substances [19-21].
Although street price data are collected by local law enforcement, they have only occasionally been reported at a national level and are rarely made available for public health research [22]. The standard federal government source for this information, the Department of Justice's National Drug Intelligence Center (NDIC), was closed in June 2012 as part of a broader realignment of federal drug policy with no notice of future data availability.
An earlier study by our group suggested the Internet was an infrequent source of diverted drugs [23]. More recently, several anonymous online marketplaces operate via Tor hidden services (distributed traffic software enabling online anonymity) or using other identity-masking techniques. One such marketplace is Silk Road, where controlled substances can be purchased with a reasonable expectation of anonymity for both the purchaser and seller. We hypothesized that Silk Road could be a source of information on street prices for diverted prescription drugs.
Given the interest but lack of scientific efforts to collect street price information, we sought to evaluate whether online crowdsourcing could be used to measure black market street prices. Crowdsourcing is a method for harnessing distributed human intelligence, where small pieces of independently derived information are systematically collected, often using electronic tools [24]. Crowdsourcing has been used in applied biomedical research to rapidly and efficiently complete tasks that would otherwise require large amounts of time, for example to evaluate medical pictograms [25] and multilanguage patient information [26], process patient narratives [27], collect soil samples across a large area [28], identify malaria parasites in slides [29], and others [30-32]. We have previously demonstrated that electronic crowdsourcing techniques can be used to produce rapid estimates in fields as varied as infectious disease incidence and illegal wildlife trade [33-35]. We hypothesized that the same could be true for street prices. Any use of the Internet as a source of information for public health research requires careful validation against established data sources and an understanding of biases that may be present in the data. Therefore, we conducted an experiment to cross-validate 2 sources of online street price data (StreetRx and Silk Road) with a more traditional survey of law enforcement officers.
Launched on November 1, 2010, StreetRx is a collection of databases and websites, which gathers, organizes, and displays street price data on diverted pharmaceutical controlled substances for public health research purposes (see Figure 1) [36]. Site users anonymously submit prices they paid or heard were paid for diverted prescription drugs, specifying the drug formulation, dose, and the US city or state in which the transaction occurred. Date of entry is automatically collected. The system supports product codes and location information for the United States, Canada, and the United Kingdom. A visual photo identification feature for each formulation aids accuracy of reporting. In order to mitigate concerns about self-incrimination, the submitter can choose to identify the source of the information in 3 ways: personal experience, heard it from "someone who isn't me" (SWIM), or the Internet. Links to information on drug treatment, overdose prevention, harm reduction, safe disposal, and pain management are also provided. Site visitors can query and view submitted prices at the city level using a map interface. Users have submitted links to online pharmacies, news media, government reports, and other public sources. StreetRx averages roughly 200 visitors and 20 street price submissions per day.
Submissions for opioid analgesics that were received from the United States between January 1 and June 30, 2012, and contained data about formulation and dose strength were considered for this report. Based on previous crowdsourcing experiments, we deemed it necessary to have a systematic way to reduce noise in the data and identify less credible submissions. Outlier prices identified by site users as "cheap" or "overpriced" on a 5-point visual analog scale were excluded; approximately one quarter of all submissions were rated in these 2 categories. Because duplicate submissions for the same drug from the same IP address less than 10 seconds apart most likely indicated submission errors, only one of the dyad was retained.
StreetRx is written in PHP programming language, with OpenLayers and jQuery user interface components. The data are stored in a MySQL relational database, on a scalable, secure hosting service with a proven track record of managing traffic spikes and high user load. Because the hosting provider also specializes in politically controversial content, the system is designed to resist attempts at being shut down due to objections to the site content. It relies on map tiles from Google Maps, but uses OpenLayers to render the map interface. The site also contains Google Analytics to track visitor volume and other statistics.
Law Enforcement Data
Reference data for street prices were obtained from the Researched Abuse, Diversion and Addiction Related Surveillance (RADARS) System Drug Diversion program, which collects data from approximately 280 police agencies in 49 US states on a quarterly basis. Methods of the RADARS System Drug Diversion Program, which is operated by the Center for Applied Research on Substance Use and Health Disparities, Nova Southeastern University (Miami, Florida), have been described previously [37]. For this study, a subset of 125 law enforcement reporters was selected based on prior consistency of reporting, level of diversion activity, and geographic distribution [38]. These agencies were from 46 US states, but were not sampled in a way to make them nationally representative. A standardized electronic form was developed to collect data about the prices paid for specific drug formulations and strengths. Each reporter received the survey quarterly, in April 2012 (covering January through March 2012) and July 2012 (covering April through June 2012). Respondents were instructed to provide prices for the most common dosage strength they encountered for each opioid during the time period of interest, and not to respond for opioids with no encounters during the time period.
Dark Internet Online Marketplace Data
Silk Road is an anonymous online marketplace structured as a Tor hidden service (see Figure 2) [39]. Prospective buyers access Silk Road through a distributed network, which provides anonymity to the IP addresses of both the buyer's Web client and the Silk Road server [40]. Silk Road uses Bitcoin (BTC), an international peer-to-peer digital currency, for payments. Bitcoin prices were converted to US dollars using the weighted average price posted on a Bitcoin exchange website on the day the sale was posted to Silk Road [41]. The exchange rate between Bitcoin and US dollars during October 2012 was approximately 11 BTC to 1 USD. A subject matter expert (author CM) manually collected ("scraped") prices per milligram for prescription opioids in the "Opioids" section from October 1 through October 31, 2012, and collected these data on a standardized electronic data collection form. Only posts that specified that the product would be shipped from the United States were scraped. No data cleaning steps were performed and no effort was made to purchase the drugs online. No "stealth listings" (unsearchable and unlinked listings that are accessible only by buyers who have been given the URL) were scraped.
Figure 1. Screenshot of StreetRx - features street price data on diverted pharmaceutical controlled substances for public health research purposes.
Figure 2. Screenshot of Silk Road - an anonymous online marketplace where drugs, fireworks, and stolen goods are sold.
Opioids Studied
The following drugs were initially considered for inclusion in the study: oral/sublingual dosing forms of buprenorphine, hydrocodone, hydromorphone, methadone, morphine, oxycodone, oxymorphone, tapentadol, tramadol, and transdermal patch formulations of fentanyl. Because only a small number of reports were available for tapentadol and transdermal fentanyl, these opioids were excluded from further analysis.
Geometric means and 95% confidence intervals of the price per milligram were calculated for each opioid. Correlation between systems was assessed by comparing opioid-specific means using the nonparametric Spearman rank correlation coefficient (rho), and tested with the null hypothesis that data from each pair of systems were independent with a two-tailed significance threshold of 0.05. Data management and analysis were performed using STATA version 12. In a preplanned secondary analysis, we used a standardized equianalgesic dosing conversion table [42] to convert the strength of each formulation to milligram of morphine equivalent, and compared prices on this basis. The predicted potency was calculated by dividing the mean price per milligrams for each opioid by that of morphine.
Ethics and Disclosure Statement
Law enforcement data used in this study were reviewed by the Colorado Multiple Institutional Review Board (IRB), which also provides overall ethical oversight to the RADARS System. The Drug Diversion program is classified as exempt by the Nova Southeastern University IRB, as it does not constitute human subjects research.
Data from 954 StreetRx reports, 737 Drug Diversion reports, and 147 postings on Silk Road were analyzed. The most reports were received for oxycodone and hydrocodone in each system (Table 1). Correlation between the 3 data sources was high. The Spearman correlation of prices per milligram between crowdsourced and law enforcement prices was 0.93 (P<.001; Figure 3, top frame), and the correlation between crowdsourced and online black market prices was 0.98 (P<.001; Figure 3, middle frame). Data from law enforcement and the online black market were also highly correlated (rho=0.90, P=.002; Figure 3, bottom frame).
With the exception of morphine, there was no significant difference between the mean price per milligram of each opioid between the 3 data sources (Figure 4). The price per milligram of morphine was greater in the law enforcement survey at US$0.67/mg (95% CI 0.59-0.75), compared with $0.52/mg (95% CI 0.40-0.68) in the crowdsourced data and US$0.42/mg (95% CI 0.37-0.48) for the online black market (P=.048).
Street prices paid for different opioids generally followed the rank order of oral equianalgesic opioid potency clinically used for rotation of opioid analgesics (Figure 4 and Table 2). The predicted potency or desirability relative to morphine was calculated by dividing the mean prices for each opioid by that of morphine. When compared to published clinical conversion guides, the predicted relative potency from crowdsourced data were similar. Oxymorphone and oxycodone had predicted potencies that were statistically indistinguishable from clinical conversion factors. Hydromorphone, hydrocodone, and methadone were valued higher on the street per milligram than in the clinic, while tramadol was valued lower on the street.
Table 1. Mean black market street prices and equianalgesic potency, US dollars per milligram, from online and law enforcement data sources, United States, 2012.
Drug StreetRx Crowdsourcing Drug Diversion Survey Silk Road Marketplace
n Mean, US$ (95% CI) n Mean, US$ (95% CI) n Mean, US$ (95% CI)
Hydromorphone 75 3.29 (2.74-3.96) 54 4.47 (3.57-5.59) 14 3.55 (3.09-4.08)
Buprenorphine 34 2.13 (1.69-2.69) 81 2.35 (1.97-2.80) 12 2.58 (2.13-3.13)
Oxymorphone 38 1.57 (1.27-1.95) 43 1.64 (1.29-2.10) 6 1.58 (0.73-3.43)
Methadone 21 0.96 (0.71-1.29) 81 1.16 (1.01-1.37) 3 0.93 (0.65-1.34)
Oxycodone 454 0.97 (0.90-1.04) 181 0.86 (0.78-0.93) 43 0.99 (0.83-1.18)
Hydrocodone 228 0.81 (0.74-0.89) 179 0.90 (0.84-0.97) 46 0.97 (0.90-1.05)
Morphine 83 0.52 (0.40-0.68) 81 0.67 (0.59-0.75)a 16 0.42 (0.37-0.48)a
Tramadol 21 0.05 (0.03-0.07) 37 0.09 (0.07-0.12) 7 0.02 (0.01-0.03)
aMorphine values differ between Drug Diversion Survey and Silk Road based on statistical test for possibility of random error (P<.05), but not between StreetRx and the other data sources.
Table 2. Mean street prices from crowdsourced data, adjusted for potency relative to morphine, United States, 2012.
Drug Crowdsourced Data from StreetRx Predicted Relative Potency Clinical Equianalgesic Potencyb
n Mean, US$ (95% CI) (95% CI)a Milligrams
Hydromorphone 75 3.29 (2.74-3.96) 6.3 (5.8-6.8) 4
Oxymorphone 38 1.57 (1.27-1.95) 3.0 (2.9-3.2) 3
Methadone 21 0.96 (0.71-1.29) 1.8 (1.8-1.9) 1.5
Oxycodone 454 0.97 (0.90-1.04) 1.9 (1.5-2.2) 2
Hydrocodone 228 0.81 (0.74-0.89) 1.5 (1.3-1.8) 1
Morphine 83 0.52 (0.40-0.68) 1.0 1
Tramadol 21 0.05 (0.03-0.07) 0.1 (0.07-0.13) 0.3
aPredicted relative potency refers to the potency or desirability as predicted by the street price relative to morphine. It was calculated by standardizing the price per milligram for each opioid against that of morphine. These numbers do not distinguish oral from other routes of administration, nor take into account time-release mechanisms. They should not be used for clinical conversion.
bSource: United States Veterans Administration/Department of Defense Clinical Practice Guideline for the Management of Opioid Therapy for Chronic Pain, 2012.
Figure 3. Correlation between the data sources: StreetRx reports, Drug Diversion survey, and Silk Road postings.
Figure 4. Mean price per milligram of each opioid analgesic, between the data sources. Numbers at the bottom of each bar indicate sample size.
Data about the street price of diverted prescription opioid medications can be useful to policymakers and public health officials, but timely and accurate data are rarely available publicly. In this paper, we present findings of a national analysis of street price data for prescription opioid analgesics. Our findings show consistent prices per milligram across 3 independent sources, and that prices for different opioid active ingredients on the black market reflect their clinically established potency. We also demonstrate the feasibility of validating crowdsourced data in the absence of a "gold standard" and document the emergence of a hidden online marketplace where drugs are sold.
Our findings are among the first to document the consistency of prices per milligram among diverted opioid analgesics. Earlier researchers, referring to heroin and cocaine, noted that "the most striking characteristics of drug prices are their high levels and extreme variability over time and space" [43]. Prescription drug prices in the United States "are affected by numerous variables, including availability, demand, law enforcement investigations, area of the country, and the relationship between the purchaser and the seller" [22]. While we observed the expected relative stability of prices during the 6 months of observation, further information is needed to understand changes over time.
For the most part, previous research has focused on the prices and purity of illicitly manufactured drugs like heroin and cocaine [13-15,43-47]. In interpreting our results, we suggest three major differences in which prescription drugs may differ from models of illicitly manufactured drugs. First, drugs are considered by economists to be "experience goods" where buyers pay before discovering the quality of the product (such as an amusement park ride or restaurant meal). Pharmaceutical manufacturing controls create a highly uniform product where these variations are not a factor. Second, retail drug transactions for illicitly manufactured drugs often occur with standardized prices, such as "dime bags". In these scenarios, the seller can modify the purity and weight of the product to maintain their margins. While counterfeit prescription drugs are possible, for the most part this strategy to improve profit margins is largely limited with pharmaceutical drugs. Third, urban markets for illicitly manufactured drugs are highly competitive with many small sellers creating intense competition for lower prices. Under these and other collective pressures, a seller's reputation for quality is paramount in order to stay in business selling illicitly manufactured drugs [48]. With prescription drugs, however, these three inputs are constrained—the quality is standardized and can be assessed immediately by the buyer. Therefore, the reputation of quality of a seller may be less important than with illicit drugs. Could this create a lower threshold for initiation of selling leftover medications?
In contrast to illicitly manufactured drugs, the different prescription opioids are nearly perfect "interchangeable goods", from the economist's perspective (but perhaps not the pharmaceutical industry's); it is difficult to distinguish opioids of the same potency such as heroin (diacetylmorphine) and hydromorphone. This means that we cannot look at the prices of any single prescription opioid in isolation, but must also see what is happening with the prices of other opioid molecules. We found only 2 studies that examined street prices for opioid analgesics, neither of which focused on online sources. One study found a 10x linear association between the pharmacy price and the street price of prescription opioid analgesics in Vancouver, British Columbia [49]. A study of drug users in eastern Kentucky suggested that OxyContin may serve as a form of currency, possibly as a proxy for social capital associated with having the means to afford daily use [50].
More studies have examined Internet pharmacies. These studies concluded that the pharmacies (whether operating legally or illicitly) were found to be rarely used sources for diverted prescription drugs [4,23,51,52]. In a fundamental shift from the itinerant Internet pharmacies of the last decade, recent social network technology has allowed the emergence of online marketplaces that allow sellers to cultivate their reputation by amassing positive feedback while keeping transactions anonymous and difficult to monitor. Often the site's owners charge a commission or per-transaction fee. Monthly transactions on Silk Road have been estimated at US$1.9 million [40], and drugs, fireworks, and stolen goods are routinely listed. Calls from US senators to shut down the sites have resulted in broad social attention [53], but the sites continue to operate. Using these sites for academic research has been limited [40], but they offer benefits. Online black market data have less opportunity for recall bias because it is possible to quantify actual transactions instead of reports of prices. However, given the moderate difficulty in accessing hidden sites relative to the open Internet, there is likely to be selection bias in terms of who uses these sites. In addition, online black markets are under pressure from authorities and may blink in and out of existence without warning, adding additional difficulty for data collection.
Websites designed for research-quality data collection via crowdsourcing and data mining are likely to cost less per report than traditional surveys and can be rapidly adapted to collect new information [54]. However, the credibility of law enforcement surveys is perceived to be higher. Other validation sources against which to compare street price data on prescription opioids are limited. With the closing of NDIC and the absence of a gold standard, we relied on triangulating street prices from 3 different sources, with our results showing remarkable consistency and robustness of prices across opioid molecules. In situations where a gold standard is not available and the behavior under question is illicit, crowdsourcing and data mining provide alternative strategies for collecting information.
There are several limitations of this study that bear mentioning. Others have pointed out the need to take into account bulk purchasing when modeling prices of illicitly manufactured drugs [44]. To address this, StreetRx site users were asked to note if the price they were reporting was a bulk purchase of more than 10 units. Only 10% of those submitting data positively indicated bulk purchasing; however, the question was not mandatory and we cannot exclude the possibility that others did not answer the question for separate reasons. Another limitation of this analysis was the insufficient sample size to analyze fentanyl and tapentadol data. The greatest limitation, however, lies in the sample size and non-contemporaneous data collection. Our research team was not aware of Silk Road until later in the study, but we felt that it was an important data source that was worth documenting. Geographic analyses, or at least controlling for geographic variation, were deemed to be beyond the scope of the present analysis, but an area of future research. We also did not account for the site users' history of addiction in the analyses, despite basic research suggesting that the point in the progression of addictive disease may influence willingness to pay. We hope to explore these topics in future analyses. Connecting drug prices to behaviors and health outcomes is a direction of future research.
Finally, we note that the use of equianalgesic ratios in clinical practice should be undertaken with caution, as should the interpretation of our results using these conversion numbers. In this analysis, we do not know if the opioids were diverted for self-medication, euphoria, or preventing withdrawal. The equianalgesic conversion factors were designed with opioid rotation for pain in mind, and the relative desirability for abuse or withdrawal prevention may be different. Various equianalgesic potency tables have been proposed [42,55,56], and the predicted potency conversion factors may reflect some of this variation. However, for the most part the ostensibly free black market was able to accurately predict the relatively pharmacologic potency of opioid molecules.
Crowdsourcing and data mining are efficient ways to collect data about street prices in an era of Internet-based social networks. These data can inform pharmacoeconomic modeling and policy analysis, and may shed light on which new controlled pharmaceutical formulations have desirability relative to others when they hit the street. While this study represents an initial foray into collecting systematic economic data for modeling black markets for prescription drugs, the methodology could be extended in the future by connecting the data to health outcomes.
The authors wish to thank Karen Lowitz for assistance in preparing this manuscript. This research was funded by the RADARS System (Denver, Colorado, USA).
StreetRx is entirely funded by the RADARS System (Denver, Colorado, USA), an independent nonprofit operation of the Rocky Mountain Poison and Drug Center (RMPDC), a division of Denver Health. StreetRx is operated under contract by Epidemico (Boston, Massachusetts, USA), a health data collection and analytics company. Authors of this paper include employees of the RADARS System and Epidemico. All datasets used in this publication are publicly available by contacting the RADARS System.
McDonald DC, Carlson KE. Estimating the prevalence of opioid diversion by "doctor shoppers" in the United States. PLoS One 2013;8(7):e69241 [FREE Full text] [CrossRef] [Medline]
Berge KH, Dillon KR, Sikkink KM, Taylor TK, Lanier WL. Diversion of drugs within health care facilities, a multiple-victim crime: patterns of diversion, scope, consequences, detection, and prevention. Mayo Clin Proc 2012 Jul;87(7):674-682 [FREE Full text] [CrossRef] [Medline]
Rigg KK, Kurtz SP, Surratt HL. Patterns of prescription medication diversion among drug dealers. Drugs (Abingdon Engl) 2012;19(2):144-155 [FREE Full text] [CrossRef] [Medline]
Substance Abuse and Mental Health Services Administration. Results from the 2010 National Survey on Drug Use and Health: Volume I (NSDUH Series H-38A, HHS Publication No. SMA Findings). In: Summary of National Findings. Rockville, MD: Health and Human Services (Office of Applied Studies); 2010:11-4856.
Drug Enforcement Administration (DEA)' Department of Justice. Schedules of controlled substances: rescheduling of buprenorphine from schedule V to schedule III. Final rule. Fed Regist 2002 Oct 7;67(194):62354-62370. [Medline]
Nordt C, Stohler R. Low-threshold methadone treatment, heroin price, police activity and incidence of heroin use: the Zurich experience. Int J Drug Policy 2009 Nov;20(6):497-501. [CrossRef] [Medline]
Kleiman MA. Enforcement swamping: A positive-feedback mechanism in rates of illicit activity. Mathematical and Computer Modelling 1993 Jan;17(2):65-75. [CrossRef]
Minron J. The effect of drug prohibition on drug prices: evidence from the markets for cocaine and heroin. Review of Economics and Statistics 2003;85(3):522-530.
Brown G, Silverman L. The retail price of heroin: estimation and applications. Journal of the American Statistical Association 1974;69(347):595-606.
Bretteville-Jensen AL. To legalize or not to legalize? Economic approaches to the decriminalization of drugs. Subst Use Misuse 2006;41(4):555-565. [CrossRef] [Medline]
Bright DA, Ritter A. Retail price as an outcome measure for the effectiveness of drug law enforcement. Int J Drug Policy 2010 Sep;21(5):359-363. [CrossRef] [Medline]
Caulkins JP, Tragler G, Wallner D. Optimal timing of use reduction vs. harm reduction in a drug epidemic model. Int J Drug Policy 2009 Nov;20(6):480-487. [CrossRef] [Medline]
Costa Storti C, De Grauwe P. Globalization and the price decline of illicit drugs. Int J Drug Policy 2009 Jan;20(1):48-61. [CrossRef] [Medline]
Caulkins JP. The global recession's effect on drug demand--diluted by inertia. Int J Drug Policy 2011 Sep;22(5):374-375. [CrossRef] [Medline]
Bretteville-Jensen AL. Illegal drug use and the economic recession--what can we learn from the existing research? Int J Drug Policy 2011 Sep;22(5):353-359. [CrossRef] [Medline]
Bickel WK, Miller ML, Yi R, Kowal BP, Lindquist DM, Pitcock JA. Behavioral and neuroeconomics of drug addiction: competing neural systems and temporal discounting processes. Drug Alcohol Depend 2007 Sep;90 Suppl 1:S85-S91 [FREE Full text] [CrossRef] [Medline]
Bickel WK, Marsch LA. Toward a behavioral economic understanding of drug dependence: delay discounting processes. Addiction 2001 Jan;96(1):73-86. [CrossRef] [Medline]
Comer SD, Zacny JP, Dworkin RH, Turk DC, Bigelow GE, Foltin RW, et al. Core outcome measures for opioid abuse liability laboratory assessment studies in humans: IMMPACT recommendations. Pain 2012 Dec;153(12):2315-2324. [CrossRef] [Medline]
Bickel WK, Yi R, Mueller ET, Jones BA, Christensen DR. The behavioral economics of drug dependence: towards the consilience of economics and behavioral neuroscience. Curr Top Behav Neurosci 2010;3:319-341. [CrossRef] [Medline]
Comer SD, Bickel WK, Yi R, de Wit H, Higgins ST, Wenger GR, et al. Human behavioral pharmacology, past, present, and future: symposium presented at the 50th annual meeting of the Behavioral Pharmacology Society. Behav Pharmacol 2010 Jul;21(4):251-277 [FREE Full text] [CrossRef] [Medline]
Bretteville-Jensen AL. Addiction and discounting. J Health Econ 1999 Aug;18(4):393-407. [Medline]
National Drug Intelligence Center. National Prescription Drug Threat Assessment 2009. Washington, DC: US Department of Justice; 2009. URL: http://www.justice.gov/archive/ndic/pubs33/33775/33775p.pdf [WebCite Cache]
Inciardi JA, Surratt HL, Cicero TJ, Rosenblum A, Ahwah C, Bailey JE, et al. Prescription drugs purchased through the Internet: who are the end users? Drug Alcohol Depend 2010 Jul 1;110(1-2):21-29 [FREE Full text] [CrossRef] [Medline]
Good BM, Su AI. Crowdsourcing for bioinformatics. Bioinformatics 2013 Aug 15;29(16):1925-1933. [CrossRef] [Medline]
Yu B, Willis M, Sun P, Wang J. Crowdsourcing participatory evaluation of medical pictograms using Amazon Mechanical Turk. J Med Internet Res 2013;15(6):e108 [FREE Full text] [CrossRef] [Medline]
Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. J Med Internet Res 2012;14(3):e79 [FREE Full text] [CrossRef] [Medline]
Maclean DL, Heer J. Identifying medical terms in patient-authored text: a crowdsourcing-based approach. J Am Med Inform Assoc 2013 May 5:1. [CrossRef] [Medline]
Shelley W, Lawley R, Robinson DA. Crowd-sourced soil data for Europe. Nature 2013 Apr 18;496(7445):300. [CrossRef] [Medline]
Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. J Med Internet Res 2012;14(6):e167 [FREE Full text] [CrossRef] [Medline]
Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, Becker LB, et al. Crowdsourcing-harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med 2013 Jul 11:1. [CrossRef] [Medline]
Zhai H, Lingren T, Deleger L, Li Q, Kaiser M, Stoutenborough L, et al. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. J Med Internet Res 2013;15(4):e73 [FREE Full text] [CrossRef] [Medline]
Jarmolowicz DP, Bickel WK, Carter AE, Franck CT, Mueller ET. Using crowdsourcing to examine relations between delay and probability discounting. Behav Processes 2012 Nov;91(3):308-312. [CrossRef] [Medline]
Sonricker Hansen AL, Li A, Joly D, Mekaru S, Brownstein JS. Digital surveillance: a novel approach to monitoring the illegal wildlife trade. PLoS One 2012;7(12):e51156 [FREE Full text] [CrossRef] [Medline]
Chunara R, Andrews JR, Brownstein JS. Social and news media enable estimation of epidemiological patterns early in the 2010 Haitian cholera outbreak. Am J Trop Med Hyg 2012 Jan;86(1):39-45 [FREE Full text] [CrossRef] [Medline]
Chan EH, Brewer TF, Madoff LC, Pollack MP, Sonricker AL, Keller M, et al. Global capacity for emerging infectious disease detection. Proc Natl Acad Sci U S A 2010 Dec 14;107(50):21701-21706 [FREE Full text] [CrossRef] [Medline]
StreetRx. URL: http://www.streetrx.com/ [accessed 2013-08-13] [WebCite Cache]
Inciardi JA, Surratt HL, Cicero TJ, Kurtz SP, Martin SS, Parrino MW. The "black box" of prescription drug diversion. J Addict Dis 2009 Oct;28(4):332-347 [FREE Full text] [CrossRef] [Medline]
Dart RC, Cicero TJ, Surratt HL, Rosenblum A, Bartelson BB, Adams EH. Assessment of the abuse of tapentadol immediate release: the first 24 months. J Opioid Manag 2012;8(6):395-402. [CrossRef] [Medline]
Silk Road. URL: http://silkroadvb5piz3r.onion [accessed 2013-08-13] [WebCite Cache]
Christin N. Traveling the Silk Road: A measurement analysis of a large anonymous online marketplace. 2012. URL: http://arxiv.org/abs/1207.7139 [accessed 2013-08-13] [WebCite Cache]
Mount Gox. URL: https://www.mtgox.com/ [accessed 2013-08-13] [WebCite Cache]
Dasgupta N, Kramer ED, Zalman MA, Carino S, Smith MY, Haddox JD, et al. Association between non-medical and prescriptive usage of opioids. Drug Alcohol Depend 2006 Apr 28;82(2):135-142. [CrossRef] [Medline]
Caulkins J, Renter P. What price data tell us about drug markets. Journal of Drug Issues 1998;28(3):593-613.
Caulkins JP. What is the average price of an illicit drug? Addiction 1994 Jul;89(7):815-819. [Medline]
Caulkins JP. Price and purity analysis for illicit drug: data and conceptual issues. Drug Alcohol Depend 2007 Sep;90 Suppl 1:S61-S68. [CrossRef] [Medline]
Bach PB, Lantos J. Methadone dosing, heroin affordability, and the severity of addiction. Am J Public Health 1999 May;89(5):662-665. [Medline]
Wood E, Tyndall MW, Spittal PM, Li K, Anis AH, Hogg RS, et al. Impact of supply-side policies for control of illicit drugs in the face of the AIDS and overdose epidemics: investigation of a massive heroin seizure. CMAJ 2003 Jan 21;168(2):165-169 [FREE Full text] [Medline]
Caulkins J, Feichtinger G, Haunschmied J, Tragler G. Quality cycles and the strategic manipulation of value. Operations Research 2006;54(4):666-677.
Sajan A, Corneil T, Grzybowski S. The street value of prescription drugs. CMAJ 1998 Jul 28;159(2):139-142 [FREE Full text] [Medline]
Jonas AB, Young AM, Oser CB, Leukefeld CG, Havens JR. OxyContin as currency: OxyContin use and increased social capital among rural Appalachian drug users. Soc Sci Med 2012 May;74(10):1602-1609. [CrossRef] [Medline]
Cicero TJ, Shores CN, Paradis AG, Ellis MS. Source of drugs for prescription opioid analgesic abusers: a role for the Internet? Pain Med 2008 Sep;9(6):718-723. [CrossRef] [Medline]
Substance Abuse and Mental Health Services Administration. Results from the 2011 National Survey on Drug Use and Health: Volume I. In: Summary of National Findings (NSDUH Series H-38A, HHS Publication No. SMA Findings). Rockville, MD: Health and Human Services (Office of Applied Studies); 2011:12-4856.
Chen A. The Underground Website Where You Can Buy Any Drug Imaginable. 2011. URL: http://kotaku.com/5805928/the-underground-website-where-you-can-buy-any-drug-imaginable [accessed 2013-08-13] [WebCite Cache]
Brownstein JS, Freifeld CC, Madoff LC. Digital disease detection--harnessing the Web for public health surveillance. N Engl J Med 2009 May 21;360(21):2153-2157 [FREE Full text] [CrossRef] [Medline]
Haffey F, Brady RR, Maxwell S. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf 2013 Feb;36(2):111-117. [CrossRef] [Medline]
Goodman LS, Gilman A, Brunton LL. Goodman & Gilman's manual of pharmacology and therapeutics. New York: McGraw-Hill Medical; 2008.
IRB: Institutional Review Board
NDIC: National Drug Intelligence Center
RADARS: Researched Abuse, Diversion, and Addiction-Related Surveillance System
Edited by G Eysenbach; submitted 01.07.13; peer-reviewed by S Schnoll, Q Zhang, G Gilmore, D Ciccarone, J Zacny; comments to author 29.07.13; accepted 09.08.13; published 16.08.13
©Nabarun Dasgupta, Clark Freifeld, John S Brownstein, Christopher Mark Menone, Hilary L Surratt, Luke Poppish, Jody L Green, Eric J Lavonas, Richard C Dart. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 16.08.2013.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Dasgupta N, Freifeld C, Brownstein JS, Menone CM, Surratt HL, Poppish L, Green JL, Lavonas EJ, Dart RC
doi: 10.2196/jmir.2810 PMID: 23956042 PMCID: PMC3758048


Note: A postcript has been added at the bottom of this article with new and relevant informaiton.
In December 2012 I published a book How to Find a Black Cat in a Coal Cellar: The Truth About Sports Tipsters. One of the central themes in it was to show that, on average, sports betting tipsters of which many are selling their advice for money, actually offer little more than guesswork and luck. Now that Sports-Tipsters has finished verifying the advisory service network Betadvisor.com, possibly the best known sports tipping service on the Internet today, I thought it would be worth telling the story of their performance as I have seen it. Readers, as usual, should understand that this story reflects my opinion only, and not those of Betadvisor, who of course will have a completely different view of events.
Betadvisor.com is a multi-tipster platform allowing both sports and racing tipsters to sell their betting advice to punters wishing to buy it. By offering an outlet for tipsters who might otherwise not have the means or skills to manage and advertise their own website, Betadvisor takes a percentage of any sales from their subscriptions. From the 17th February 2010 until the 15th May 2011 Sports-Tipsters agreed to verify all sports tips (Sports-Tipsters was never responsible for verification of any of the racing picks) from all tipsters operating at Betadvisor, and analyse and publish their results as one complete aggregated record. After this time, Betadvisor had submitted 10,134 tips for verification, returning a yield of +0.76%. The time series of these tips is shown below. According to statistical analysis there was a 22% probability that this performance could have happened by chance alone. This is a long way removed from what I (and most statisticians) consider to be evidence that chance would probably not the cause (1%). The records of the participating individual tipsters were not analysed separately during this time.
It was then agreed that since such a record was essentially artificial (since no single subscription package for all tipsters existed), public verification of Betadvisor's tipsters as a single record would cease. In its place, Laurent Marty (rugby tipster) and Geert van Elsen (football tipster specialising in Belgian football), in Sports-Tipsters opinion Betadvisor's two best tipsters at that time, continued to be publically verified individually on Sports-Tipsters. In addition to this, Sports-Tipsters agreed to continue to receive and archive all tips e-mails privately for all BetAdvisor's sports tipsters, and did so until verification was terminated on the 19th December 2013. Both Laurent Marty and Geert van Elsen continued to demonstrate their obvious abilities as tipsters. As of the 19th December 2013 Laurent had a yield of +12.27% from a total of 1311 tips dating back to the 19th February 2010. Geert van Elsen had a yield of +12.48% from 401 tips dating back to the 7th April 2010. Their profit charts are show below.
On the 22nd October I came across a forum thread on VerifiedTipsters discussing Betadvsior's verification, and decided it was necessary to explain the situation. In my post I confirmed that Sports-Tipsters continued to be in receipt of all sports tips, but observed that Betadvisor's summary performance page of all tipsters presented a misleading picture of the genuine success of their tipsters during the time that Sports-Tipsters received their e-mails. At that time, there were 41 tipsters listed on that performance page and all of them were in profit. I was confident that based on what I had verified between February 2010 and May 2011 this was highly unlikely to properly reflect the actual performance of all sports tipsters that had passed through Betadvisor since that time, and decided to set about analysing the performance.
In doing so, it became apparent that individual reporting for each tipster's performance on Betadvisor was including both the tips that had been made available for subscription as well as those that were recorded in house during any trial period a tipster had undergone through Betadvsior's Tipster Academy. These Academy tips were never independently verified by Sports-Tipsters, and since they were neither ever publically available for sale, I took the view that they should not be included within the official statistical reporting. At the very least, if they were to be, then a clear distinction between the two periods should be made, with a date provided for when any tipster moved from the Tipster Academy to the main Betadvisor service. Consequently, in the interests of transparency I asked Betadvisor to make such a distinction clear.
On the 12th December Sports-Tipsters wrote to the managers of Betadvisor to explain that it was no longer willing to support it. Although some progress had been made to demonstrate the distinction between trial Tipster Academy tips and main Betadvisor tips on its certification page, I felt that it was still not enough. Importantly, it was still not possible to identify from individual tipster's records, which of their sports tips had been available for sale publically and independently verified (by Sports-Tipsters) and which had been trial tips (through Betadvisor's Tipster Academy), not been independently verified and never available for purchase. Seemingly, Betadvisor felt that implementing such a distinction on every tipster page presented a technical challenge that would take some time to organise. Given that Betadvisor utilises a highly sophisticated database to manage and report on its tipsters, I always felt that this was an unlikely excuse. Certainly, after 7 weeks of waiting it had little credibility.
The overall profit from the 17th February 2010 to 31st October 2013 from 37,813 picks (including voids) was just +0.44%. The probability that this had happened just by chance is 20%, again far, far removed from any serious measure of statistical significance that might demonstrate that something other than chance was involved. [Read an earlier article about how to separate betting luck and skill]. The time series is shown below.
Between the 15th May 2011 and 31st October 2013, only 2 of the 118 sports tipsters that had been operating during that period achieved what Sports-Tipsters considered to be a statistically significant level of profitability (at the 99% confidence limit) that might be regarded as some evidence of skill over chance: Laurent Marty and Andre Nitu. (Extending this period back to February 2010, Geert van Elsen was also seen to be demonstrating a statistically significant level of profitability, but because his rate of tipping was so low, his profitability did not actually reach the 99% confidence limit for the 15th May 2011 to 31st October 2013 period). 70 of the 118 tipsters (or 59%) between the 15th May 2011 and the 31st October 2013 actually lost money.
Betadvisor has stressed that losing tipsters have sold far fewer subscriptions, and so it is unfair to group all the tipsters together into one analysis. Indeed, this was precisely why aggregate-single-record reporting was stopped in May 2011. But this of course completely misses the point. Betadvisor claim to be offering the "best sports betting tips". It says as much quite clearly in its logo. Furthermore, on its FAQs page it confidently claims that all its analysts are "profitable in the long run" (see Do you recommend certain analysts over others?). I suppose if "long run" means infinity, then just possibly this will give enough time for Betadvisor's losing tipsters to turn things around. What is abundantly clear, however, is that Betadvisor itself does not grant this luxury to its underperforming tipsters. How else could a summary page show 41 profitable tipsters and 0 unprofitbale ones? Consequently, then, this assertion is manifestly untrue. What's more, quite how anybody is meant to know what will ultimately prove to be a losing tipster before he has become one and at the time a potential subscription is purchased is anybody's guess. Undeniably, based on these statistics, there will be plenty of Betadvisor subscribers who will have lost money. If that was not the case, it would rather beg the question why losing tipsters were ever allowed onto Betadvisor in the first place, if they weren't even selling anything much anyway.
Betadvisor has rightly made use of its Tipster Academy to trial and select sports tipsters for its main service. Unfortunately, it would appear that not nearly enough rigour is applied to the tipster selection process. Too many unskilled and "lucky" sports tipsters are wrongly being selected for forecasting talent that few of them actually possess. Between the 15th May 2011 and the 31st October 2013 70 out of 118 seemingly slipped through the net. In my book, that is not nearly good enough for a site that claims to be selling the best sports tips.
Without more effort made by Betadvisor to present this reality to potential customers, it is easy to see how the service can be viewed (as many have) as little more than slick marketing. Sports-Tipsters was always supportive of what Betadvisor was attempting to achieve, but given this perceived lack of recognition of its weaknesses, I was no longer prepared to put my name to the Betadvisor project. The sports tipping industry should be about selling forecasting skill to those punters who haven't got it themselves. If all we have, on average, is luck, we haven't actually got anything worthwhile to sell at all. One might challenge this view on the grounds that at least some of the tipsters proved to be skilful. 3 out of well over 100? Is that really good enough? Presumably if we recruit enough people we'll always find a small percentage of them that genuinely have an ability that's worth selling. But that's a message we can never really expect services like Betadvisor to promote; it simply wouldn't be good business, would it? Better to say that you offer the best sports tips and hide behind the law of large numbers. As a business, it's their right to advertise their product as they see fit, provided that advertising remains within the confines of the law. It's my right to challenge that marketing approach provided I am not intending to defame the brand by using information that is untrue. There is no intention on my part to defame, and all of the information I have presented here can be verified and confirmed by Betadvisor themselves if they so chose to make the data available.
Betadvisor, however, are not unusual. Indeed, it is the case that all multi-tipster services, and all multi-tipster samples of data I have analysed, fail to deliver anything much more than luck alone. Of course, my book reported on my 11 year verification of nearly 250 tipping histories that showed a yield of +1.13% from 161,564 tips. Between 2004 and 2012 the Punters Lounge Tipping competition saw 579 individual tip 160,094 betting picks for a total yield of -0.95%. The customers whom I have referred to Pinnacle Sports since 2003 (arguably more sophisticated than the typical punter betting at UK or European bookmakers) have shown a total profit over turnover of -0.25%, from 56,876 bets. Arguably only 3 or 4 of them (out of a total of 102 for which data exists) have shown a statistically significant level of profitability that might be evidence of skill.
Perhaps most incredibly of all, Oddsportal's community of tipsters has tipped over 3.3 million betting picks since it was first established for an overall yield of just +0.85%. Of course it's true to say that this is a profit. But did it happen, on average, by skill or by chance? Yes, as for all the other samples of tipsters I have talked about there is always a small few who genuinely appear to have a real forecasting ability. But remember, here we are talking about on the average. Oddsportal make it very easy for its users to find 100%+ payout lines. Whilst we have no data on theoretical payout figures for these 3.3 million tips, it might be reasonable to assume that the average figure would be close to 100%. Tipsters, after all, don't usually tip less than best prices, and most best prices are at or above 100% payout on Oddsportal. With a yield of just +0.85%, can we really say that the average Oddsportal tipster is doing anything more than finding a best price in a falling market? After all, using just that very strategy (to find 100% payouts in shortening markets) with just one bookmaker (Sportingbet) I was able to show a yield of +0.91% from 5,751 individual betting selections.
VerifedTipsters has reported a yield of +5.1% from 60,000 verified picks. This result appear to buck the trend that punters who are prepared to advise others what to bet on are not capable, on average, of performing anything better than could be expected from chance alone. For the most part, I've not been party to the tips that have been verified by VerifedTipsters, so am unable to comment on this sample of data. Perhaps VerifedTipsters have a means of attracting better than average quality betting forecasters. Or perhaps this sample of 60,000 is too small, in comparison to the 160,000 tips verified by Sports-Tipsters, the 160,000 tips posted for the Punters Lounge tipping competition or the 3.3 million tips advised by Oddsportal community users. This is not for me to judge. However, until I see some significant sample of data that contradicts everything I have reported on here, I am prepared to stand by my assertion that almost all so-called tipsters are anything but, and that most of what is being sold represents nothing more than slick marketing in an attempt to attract customers to products that in fact are not what they claim to be. For the most part, that is not because the sellers are trying to defraud the buyers, but simply because the sellers haven't yet realised that luck is not the same thing as skill.
Postcript Posted 26th September 2014: In September 2014 Betadvisor changed ownership and updated their FAQs page, removing reference to the suggestion that all its advisors were profitable in the long run. The new owners should be applauded for that.


Hatcher Pass and Independence Mine bring visitors back in time.
The Glenn Highway includes several quaint towns and a scenic drive.
Wasilla is home to the Iditarod headquarters, and is a bustling community located along the Parks Highway.
Talkeetna is known for its grand views of Denali, North America's tallest peak.


text:
#pragma once

#include "BasicMath.hpp"

namespace Diligent
{

class InputControllerAndroid : public InputControllerBase
{
public:
    void BeginDrag(float x, float y)
    {
        EndPinch();
        m_MouseState.ButtonFlags |= MouseState::BUTTON_FLAG_LEFT;
        m_MouseState.PosX = x;
        m_MouseState.PosY = y;
    }

    void EndDrag()
    {
        m_MouseState.ButtonFlags &= ~MouseState::BUTTON_FLAG_LEFT;
    }

    void DragMove(float x, float y)
    {
        m_MouseState.ButtonFlags |= MouseState::BUTTON_FLAG_LEFT;
        m_MouseState.PosX = x;
        m_MouseState.PosY = y;
    }

    void StartPinch(float x0, float y0, float x1, float y1);
    void PinchMove(float x0, float y0, float x1, float y1);
    void EndPinch();

private:
    float2 m_PrvePinchPoint0;
    float2 m_PrvePinchPoint1;
};

} // namespace Diligent

redpajama_set_name: RedPajamaGithub


Go now! The secret is out, but it's still a steal by European standards and there are so many amazing spots to explore.
Lufthansa from SFO/Frankfurt/Porto and back home from Lisbon/Frankfurt/SFO. We wanted to explore as much of the country as we could in 12 days, so rented a car for freedom and flexibility. We ditched our car while in Lisbon, then rented another for the drive south to The Algarve.
Residentas Apostolas: Beautiful, unique, apartments with fantastic location and gracious, helpful hosts.
Martinhal Lisbon Chiado: Lisbon outpost for the Martinhal Family Resorts. Huge apartments in central Chiado location. Kids club and in-hotel café geared towards children. Recommend for families with preschool aged children or families that may want hotel childcare during their stay.
There are a plethora of fabulous, high-end, Michelin starred restaurants in Lisbon. Traveling with school-aged children, we aimed for a slightly more casual feel, and although we didn't even scratch the surface, here are some of our favorites.
Á Cevicheria: Excellent, creative ceviche restaurant. They don't take reservations and the line to be seated can be long, but have one of their amazing Pisco Sours and know the wait is worth it.
Cervajeria Romero: Get the garlic shrimp and goose barnacles! The barnacles are ugly, and briny, but in a good way. Be brave! Old school, family-run restaurant with crazy, fresh fish. There's always a line, but its shorter for lunch.
Fourno d'Oro: Need a seafood break? Probably not, but if you do, Fourno D'Oro has insane pizza and a fun atmosphere.
Pastéis de Belém: Pastel de nata is a custard-filled pastry and the pride of Portugal. The most famous of these tarts are served at Pastéis de Belém, close to the Jeronimos Monastery from which they originally hail. Expect long lines and eat them while they are warm.
Gelateria Nannarella: Best gelato in Lisbon, and maybe all of Portugal.
Time Out Market Lisboa: Time Out Market Lisboa is a bustling, indoor food court featuring some of the very best Portuguese culinary vendors and restaurant outposts. Order from your preferred spot and coordinate with your family to snag a table, which can be especially challenging during dinner hours.
LX Factory: Formerly and industrial area, LX Factory is a cluster of shops, restaurants, galleries, and bookstores with an urban, hipster vibe. Unique boutiques and definitely a great place to for a few hours.
Alfama Neighborhood: Steep hills with narrow, cobblestone streets and tiny shops and restaurants make this neighborhood perfect for exploring.
Santa Justa Lift: Elevator built in the 19th century with sweeping view of Lisbon.
Castelo de São Jorge: Hike up the hill to this amazing castle overlooking Lisbon, then climb some more once inside the castle walls. Beautiful views of the city and beyond.
Belém: Great for a half day trip. Take a bus or Tuk-Tuk to this historical district a few miles from the town center. Don't miss the Torre de Belém, Jeronimos Monastery, and Jardím Botânica Tropical and remember to stop by the Pastéis de Belém for their famousePastel de nata.
So many great museums, but with kids, I'd recommend the National Coaches Museum and Musea Calouste Gulbenkian.
Go now! The secret is out, but it's still a steal by European standards and there are so many amazing spots to explore. Summer in the Algarve sounds amazing for a beach vacation, but we traveled in April and found crowds to be super manageable, and the weather great for exploring.
Would love to have visited chic Comporta, but didn't have time and summer probably a better time.
Tram 28 in Lisbon, is super crowded, foul smelling and there's a more than fair chance you'll be pickpocketed. Walk, or take a car or Tuk-Tuk.


text:
"""Notifications module for OpenStack Identity Service resources."""

import collections
import functools
import inspect
import socket

import flask
from oslo_log import log
import oslo_messaging
from oslo_utils import reflection
import pycadf
from pycadf import cadftaxonomy as taxonomy
from pycadf import cadftype
from pycadf import credential
from pycadf import eventfactory
from pycadf import host
from pycadf import reason
from pycadf import resource

from keystone.common import context
from keystone.common import provider_api
from keystone.common import utils
import keystone.conf
from keystone import exception
from keystone.i18n import _

_CATALOG_HELPER_OBJ = None

LOG = log.getLogger(__name__)
# NOTE(gyee): actions that can be notified. One must update this list whenever
# a new action is supported.
_ACTIONS = collections.namedtuple(
    'NotificationActions',
    'created, deleted, disabled, updated, internal')
ACTIONS = _ACTIONS(created='created', deleted='deleted', disabled='disabled',
                   updated='updated', internal='internal')
"""The actions on resources."""

CADF_TYPE_MAP = {
    'group': taxonomy.SECURITY_GROUP,
    'project': taxonomy.SECURITY_PROJECT,
    'role': taxonomy.SECURITY_ROLE,
    'user': taxonomy.SECURITY_ACCOUNT_USER,
    'domain': taxonomy.SECURITY_DOMAIN,
    'region': taxonomy.SECURITY_REGION,
    'endpoint': taxonomy.SECURITY_ENDPOINT,
    'service': taxonomy.SECURITY_SERVICE,
    'policy': taxonomy.SECURITY_POLICY,
    'OS-TRUST:trust': taxonomy.SECURITY_TRUST,
    'OS-OAUTH1:access_token': taxonomy.SECURITY_CREDENTIAL,
    'OS-OAUTH1:request_token': taxonomy.SECURITY_CREDENTIAL,
    'OS-OAUTH1:consumer': taxonomy.SECURITY_ACCOUNT,
    'application_credential': taxonomy.SECURITY_CREDENTIAL,
}

SAML_AUDIT_TYPE = 'http://docs.oasis-open.org/security/saml/v2.0'
# resource types that can be notified
_SUBSCRIBERS = {}
_notifier = None
SERVICE = 'identity'
PROVIDERS = provider_api.ProviderAPIs

ROOT_DOMAIN = '<<keystone.domain.root>>'

CONF = keystone.conf.CONF

# NOTE(morganfainberg): Special case notifications that are only used
# internally for handling token persistence token deletions
INVALIDATE_TOKEN_CACHE = 'invalidate_token_cache'  # nosec
PERSIST_REVOCATION_EVENT_FOR_USER = 'persist_revocation_event_for_user'
REMOVE_APP_CREDS_FOR_USER = 'remove_application_credentials_for_user'
DOMAIN_DELETED = 'domain_deleted'

def build_audit_initiator():
    """A pyCADF initiator describing the current authenticated context."""
    pycadf_host = host.Host(address=flask.request.remote_addr,
                            agent=str(flask.request.user_agent))
    initiator = resource.Resource(typeURI=taxonomy.ACCOUNT_USER,
                                  host=pycadf_host)
    oslo_context = flask.request.environ.get(context.REQUEST_CONTEXT_ENV)
    if oslo_context.user_id:
        initiator.id = utils.resource_uuid(oslo_context.user_id)
        initiator.user_id = oslo_context.user_id

    if oslo_context.project_id:
        initiator.project_id = oslo_context.project_id

    if oslo_context.domain_id:
        initiator.domain_id = oslo_context.domain_id

    initiator.request_id = oslo_context.request_id

    if oslo_context.global_request_id:
        initiator.global_request_id = oslo_context.global_request_id

    return initiator

class Audit(object):
    """Namespace for audit notification functions.

    This is a namespace object to contain all of the direct notification
    functions utilized for ``Manager`` methods.
    """

    @classmethod
    def _emit(cls, operation, resource_type, resource_id, initiator, public,
              actor_dict=None, reason=None):
        """Directly send an event notification.

        :param operation: one of the values from ACTIONS
        :param resource_type: type of resource being affected
        :param resource_id: ID of the resource affected
        :param initiator: CADF representation of the user that created the
                          request
        :param public: If True (default), the event will be sent to the
                       notifier API.  If False, the event will only be sent via
                       notify_event_callbacks to in process listeners
        :param actor_dict: dictionary of actor information in the event of
                           assignment notification
        :param reason: pycadf object containing the response code and
                       message description
        """
        # NOTE(stevemar): the _send_notification function is
        # overloaded, it's used to register callbacks and to actually
        # send the notification externally. Thus, we should check
        # the desired notification format in the function instead
        # of before it.
        _send_notification(
            operation,
            resource_type,
            resource_id,
            initiator=initiator,
            actor_dict=actor_dict,
            public=public)

        if CONF.notification_format == 'cadf' and public:
            outcome = taxonomy.OUTCOME_SUCCESS
            _create_cadf_payload(operation, resource_type, resource_id,
                                 outcome, initiator, reason)

    @classmethod
    def created(cls, resource_type, resource_id, initiator=None,
                public=True, reason=None):
        cls._emit(ACTIONS.created, resource_type, resource_id, initiator,
                  public, reason=reason)

    @classmethod
    def updated(cls, resource_type, resource_id, initiator=None,
                public=True, reason=None):
        cls._emit(ACTIONS.updated, resource_type, resource_id, initiator,
                  public, reason=reason)

    @classmethod
    def disabled(cls, resource_type, resource_id, initiator=None,
                 public=True, reason=None):
        cls._emit(ACTIONS.disabled, resource_type, resource_id, initiator,
                  public, reason=reason)

    @classmethod
    def deleted(cls, resource_type, resource_id, initiator=None,
                public=True, reason=None):
        cls._emit(ACTIONS.deleted, resource_type, resource_id, initiator,
                  public, reason=reason)

    @classmethod
    def added_to(cls, target_type, target_id, actor_type, actor_id,
                 initiator=None, public=True, reason=None):
        actor_dict = {'id': actor_id,
                      'type': actor_type,
                      'actor_operation': 'added'}
        cls._emit(ACTIONS.updated, target_type, target_id, initiator, public,
                  actor_dict=actor_dict, reason=reason)

    @classmethod
    def removed_from(cls, target_type, target_id, actor_type, actor_id,
                     initiator=None, public=True, reason=None):
        actor_dict = {'id': actor_id,
                      'type': actor_type,
                      'actor_operation': 'removed'}
        cls._emit(ACTIONS.updated, target_type, target_id, initiator, public,
                  actor_dict=actor_dict, reason=reason)

    @classmethod
    def internal(cls, resource_type, resource_id, reason=None):
        # NOTE(lbragstad): Internal notifications are never public and have
        # never used the initiator variable, but the _emit() method expects
        # them. Let's set them here but not expose them through the method
        # signature - that way someone can not do something like send an
        # internal notification publicly.
        initiator = None
        public = False
        cls._emit(ACTIONS.internal, resource_type, resource_id, initiator,
                  public, reason)

def invalidate_token_cache_notification(reason):
    """A specific notification for invalidating the token cache.

    :param reason: The specific reason why the token cache is being
                   invalidated.
    :type reason: string

    """
    # Since keystone does a lot of work in the authentication and validation
    # process to make sure the authorization context for the user is
    # update-to-date, invalidating the token cache is a somewhat common
    # operation. It's done across various subsystems when role assignments
    # change, users are disabled, identity providers deleted or disabled, etc..
    # This notification is meant to make the process of invalidating the token
    # cache DRY, instead of have each subsystem implement their own token cache
    # invalidation strategy or callbacks.
    LOG.debug(reason)
    resource_id = None
    initiator = None
    public = False
    Audit._emit(
        ACTIONS.internal, INVALIDATE_TOKEN_CACHE, resource_id, initiator,
        public, reason=reason
    )

def _get_callback_info(callback):
    """Return list containing callback's module and name.

    If the callback is a bound instance method also return the class name.

    :param callback: Function to call
    :type callback: function
    :returns: List containing parent module, (optional class,) function name
    :rtype: list
    """
    module_name = getattr(callback, '__module__', None)
    func_name = callback.__name__
    if inspect.ismethod(callback):
        class_name = reflection.get_class_name(callback.__self__,
                                               fully_qualified=False)
        return [module_name, class_name, func_name]
    else:
        return [module_name, func_name]

def register_event_callback(event, resource_type, callbacks):
    """Register each callback with the event.

    :param event: Action being registered
    :type event: keystone.notifications.ACTIONS
    :param resource_type: Type of resource being operated on
    :type resource_type: str
    :param callbacks: Callback items to be registered with event
    :type callbacks: list
    :raises ValueError: If event is not a valid ACTION
    :raises TypeError: If callback is not callable
    """
    if event not in ACTIONS:
        raise ValueError(_('%(event)s is not a valid notification event, must '
                           'be one of: %(actions)s') %
                         {'event': event, 'actions': ', '.join(ACTIONS)})

    if not hasattr(callbacks, '__iter__'):
        callbacks = [callbacks]

    for callback in callbacks:
        if not callable(callback):
            msg = 'Method not callable: %s' % callback
            tr_msg = _('Method not callable: %s') % callback
            LOG.error(msg)
            raise TypeError(tr_msg)
        _SUBSCRIBERS.setdefault(event, {}).setdefault(resource_type, set())
        _SUBSCRIBERS[event][resource_type].add(callback)

        if LOG.logger.getEffectiveLevel() <= log.DEBUG:
            # Do this only if its going to appear in the logs.
            msg = 'Callback: `%(callback)s` subscribed to event `%(event)s`.'
            callback_info = _get_callback_info(callback)
            callback_str = '.'.join(i for i in callback_info if i is not None)
            event_str = '.'.join(['identity', resource_type, event])
            LOG.debug(msg, {'callback': callback_str, 'event': event_str})

def listener(cls):
    """A class decorator to declare a class to be a notification listener.

    A notification listener must specify the event(s) it is interested in by
    defining a ``event_callbacks`` attribute or property. ``event_callbacks``
    is a dictionary where the key is the type of event and the value is a
    dictionary containing a mapping of resource types to callback(s).

    :data:`.ACTIONS` contains constants for the currently
    supported events. There is currently no single place to find constants for
    the resource types.

    Example::

        @listener
        class Something(object):

            def __init__(self):
                self.event_callbacks = {
                    notifications.ACTIONS.created: {
                        'user': self._user_created_callback,
                    },
                    notifications.ACTIONS.deleted: {
                        'project': [
                            self._project_deleted_callback,
                            self._do_cleanup,
                        ]
                    },
                }

    """
    def init_wrapper(init):
        @functools.wraps(init)
        def __new_init__(self, *args, **kwargs):
            init(self, *args, **kwargs)
            _register_event_callbacks(self)
        return __new_init__

    def _register_event_callbacks(self):
        for event, resource_types in self.event_callbacks.items():
            for resource_type, callbacks in resource_types.items():
                register_event_callback(event, resource_type, callbacks)

    cls.__init__ = init_wrapper(cls.__init__)
    return cls

def notify_event_callbacks(service, resource_type, operation, payload):
    """Send a notification to registered extensions."""
    if operation in _SUBSCRIBERS:
        if resource_type in _SUBSCRIBERS[operation]:
            for cb in _SUBSCRIBERS[operation][resource_type]:
                subst_dict = {'cb_name': cb.__name__,
                              'service': service,
                              'resource_type': resource_type,
                              'operation': operation,
                              'payload': payload}
                LOG.debug('Invoking callback %(cb_name)s for event '
                          '%(service)s %(resource_type)s %(operation)s for '
                          '%(payload)s', subst_dict)
                cb(service, resource_type, operation, payload)

def _get_notifier():
    """Return a notifier object.

    If _notifier is None it means that a notifier object has not been set.
    If _notifier is False it means that a notifier has previously failed to
    construct.
    Otherwise it is a constructed Notifier object.
    """
    global _notifier

    if _notifier is None:
        host = CONF.default_publisher_id or socket.gethostname()
        try:
            transport = oslo_messaging.get_notification_transport(CONF)
            _notifier = oslo_messaging.Notifier(transport,
                                                "identity.%s" % host)
        except Exception:
            LOG.exception("Failed to construct notifier")
            _notifier = False

    return _notifier

def clear_subscribers():
    """Empty subscribers dictionary.

    This effectively stops notifications since there will be no subscribers
    to publish to.
    """
    _SUBSCRIBERS.clear()

def reset_notifier():
    """Reset the notifications internal state.

    This is used only for testing purposes.

    """
    global _notifier
    _notifier = None

def _create_cadf_payload(operation, resource_type, resource_id,
                         outcome, initiator, reason=None):
    """Prepare data for CADF audit notifier.

    Transform the arguments into content to be consumed by the function that
    emits CADF events (_send_audit_notification). Specifically the
    ``resource_type`` (role, user, etc) must be transformed into a CADF
    keyword, such as: ``data/security/role``. The ``resource_id`` is added as a
    top level value for the ``resource_info`` key. Lastly, the ``operation`` is
    used to create the CADF ``action``, and the ``event_type`` name.

    As per the CADF specification, the ``action`` must start with create,
    update, delete, etc... i.e.: created.user or deleted.role

    However the ``event_type`` is an OpenStack-ism that is typically of the
    form project.resource.operation. i.e.: identity.project.updated

    :param operation: operation being performed (created, updated, or deleted)
    :param resource_type: type of resource being operated on (role, user, etc)
    :param resource_id: ID of resource being operated on
    :param outcome: outcomes of the operation (SUCCESS, FAILURE, etc)
    :param initiator: CADF representation of the user that created the request
    :param reason: pycadf object containing the response code and
                   message description
    """
    if resource_type not in CADF_TYPE_MAP:
        target_uri = taxonomy.UNKNOWN
    else:
        target_uri = CADF_TYPE_MAP.get(resource_type)

    # TODO(gagehugo): The root domain ID is typically hidden, there isn't a
    # reason to emit a notification for it. Once we expose the root domain
    # (and handle the CADF UUID), remove this.
    if resource_id == ROOT_DOMAIN:
        return

    target = resource.Resource(typeURI=target_uri,
                               id=resource_id)

    audit_kwargs = {'resource_info': resource_id}
    cadf_action = '%s.%s' % (operation, resource_type)
    event_type = '%s.%s.%s' % (SERVICE, resource_type, operation)

    _send_audit_notification(cadf_action, initiator, outcome,
                             target, event_type, reason=reason, **audit_kwargs)

def _send_notification(operation, resource_type, resource_id, initiator=None,
                       actor_dict=None, public=True):
    """Send notification to inform observers about the affected resource.

    This method doesn't raise an exception when sending the notification fails.

    :param operation: operation being performed (created, updated, or deleted)
    :param resource_type: type of resource being operated on
    :param resource_id: ID of resource being operated on
    :param initiator: representation of the user that created the request
    :param actor_dict: a dictionary containing the actor's ID and type
    :param public:  if True (default), the event will be sent
                    to the notifier API.
                    if False, the event will only be sent via
                    notify_event_callbacks to in process listeners.
    """
    payload = {'resource_info': resource_id}

    if actor_dict:
        payload['actor_id'] = actor_dict['id']
        payload['actor_type'] = actor_dict['type']
        payload['actor_operation'] = actor_dict['actor_operation']

    if initiator:
        payload['request_id'] = initiator.request_id
        global_request_id = getattr(initiator, 'global_request_id', None)
        if global_request_id:
            payload['global_request_id'] = global_request_id

    notify_event_callbacks(SERVICE, resource_type, operation, payload)

    # Only send this notification if the 'basic' format is used, otherwise
    # let the CADF functions handle sending the notification. But we check
    # here so as to not disrupt the notify_event_callbacks function.
    if public and CONF.notification_format == 'basic':
        notifier = _get_notifier()
        if notifier:
            context = {}
            event_type = '%(service)s.%(resource_type)s.%(operation)s' % {
                'service': SERVICE,
                'resource_type': resource_type,
                'operation': operation}
            if _check_notification_opt_out(event_type, outcome=None):
                return
            try:
                notifier.info(context, event_type, payload)
            except Exception:
                LOG.exception(
                    'Failed to send %(res_id)s %(event_type)s notification',
                    {'res_id': resource_id, 'event_type': event_type})

def _get_request_audit_info(context, user_id=None):
    """Collect audit information about the request used for CADF.

    :param context: Request context
    :param user_id: Optional user ID, alternatively collected from context
    :returns: Auditing data about the request
    :rtype: :class:`pycadf.Resource`
    """
    remote_addr = None
    http_user_agent = None
    project_id = None
    domain_id = None

    if context and 'environment' in context and context['environment']:
        environment = context['environment']
        remote_addr = environment.get('REMOTE_ADDR')
        http_user_agent = environment.get('HTTP_USER_AGENT')
        if not user_id:
            user_id = environment.get('KEYSTONE_AUTH_CONTEXT',
                                      {}).get('user_id')
        project_id = environment.get('KEYSTONE_AUTH_CONTEXT',
                                     {}).get('project_id')
        domain_id = environment.get('KEYSTONE_AUTH_CONTEXT',
                                    {}).get('domain_id')

    host = pycadf.host.Host(address=remote_addr, agent=http_user_agent)
    initiator = resource.Resource(typeURI=taxonomy.ACCOUNT_USER, host=host)

    if user_id:
        initiator.user_id = user_id
        initiator.id = utils.resource_uuid(user_id)
        initiator = _add_username_to_initiator(initiator)

    if project_id:
        initiator.project_id = project_id
    if domain_id:
        initiator.domain_id = domain_id

    return initiator

class CadfNotificationWrapper(object):
    """Send CADF event notifications for various methods.

    This function is only used for Authentication events. Its ``action`` and
    ``event_type`` are dictated below.

    - action: ``authenticate``
    - event_type: ``identity.authenticate``

    Sends CADF notifications for events such as whether an authentication was
    successful or not.

    :param operation: The authentication related action being performed

    """

    def __init__(self, operation):
        self.action = operation
        self.event_type = '%s.%s' % (SERVICE, operation)

    def __call__(self, f):
        @functools.wraps(f)
        def wrapper(wrapped_self, user_id, *args, **kwargs):
            """Will always send a notification."""
            target = resource.Resource(typeURI=taxonomy.ACCOUNT_USER)
            initiator = build_audit_initiator()
            initiator.user_id = user_id
            initiator = _add_username_to_initiator(initiator)
            initiator.id = utils.resource_uuid(user_id)
            try:
                result = f(wrapped_self, user_id, *args, **kwargs)
            except (exception.AccountLocked,
                    exception.PasswordExpired) as ex:
                # Send a CADF event with a reason for PCI-DSS related
                # authentication failures
                audit_reason = reason.Reason(str(ex), str(ex.code))
                _send_audit_notification(self.action, initiator,
                                         taxonomy.OUTCOME_FAILURE,
                                         target, self.event_type,
                                         reason=audit_reason)
                if isinstance(ex, exception.AccountLocked):
                    raise exception.Unauthorized
                raise
            except Exception:
                # For authentication failure send a CADF event as well
                _send_audit_notification(self.action, initiator,
                                         taxonomy.OUTCOME_FAILURE,
                                         target, self.event_type)
                raise
            else:
                _send_audit_notification(self.action, initiator,
                                         taxonomy.OUTCOME_SUCCESS,
                                         target, self.event_type)
                return result

        return wrapper

class CadfRoleAssignmentNotificationWrapper(object):
    """Send CADF notifications for ``role_assignment`` methods.

    This function is only used for role assignment events. Its ``action`` and
    ``event_type`` are dictated below.

    - action: ``created.role_assignment`` or ``deleted.role_assignment``
    - event_type: ``identity.role_assignment.created`` or
        ``identity.role_assignment.deleted``

    Sends a CADF notification if the wrapped method does not raise an
    :class:`Exception` (such as :class:`keystone.exception.NotFound`).

    :param operation: one of the values from ACTIONS (created or deleted)
    """

    ROLE_ASSIGNMENT = 'role_assignment'

    def __init__(self, operation):
        self.action = '%s.%s' % (operation, self.ROLE_ASSIGNMENT)
        self.event_type = '%s.%s.%s' % (SERVICE, self.ROLE_ASSIGNMENT,
                                        operation)

    def __call__(self, f):
        @functools.wraps(f)
        def wrapper(wrapped_self, role_id, *args, **kwargs):
            """Send a notification if the wrapped callable is successful.

            NOTE(stevemar): The reason we go through checking kwargs
            and args for possible target and actor values is because the
            create_grant() (and delete_grant()) method are called
            differently in various tests.
            Using named arguments, i.e.::

                create_grant(user_id=user['id'], domain_id=domain['id'],
                             role_id=role['id'])

            Or, using positional arguments, i.e.::

                create_grant(role_id['id'], user['id'], None,
                             domain_id=domain['id'], None)

            Or, both, i.e.::

                create_grant(role_id['id'], user_id=user['id'],
                             domain_id=domain['id'])

            Checking the values for kwargs is easy enough, since it comes
            in as a dictionary

            The actual method signature is

            ::

                create_grant(role_id, user_id=None, group_id=None,
                             domain_id=None, project_id=None,
                             inherited_to_projects=False)

            So, if the values of actor or target are still None after
            checking kwargs, we can check the positional arguments,
            based on the method signature.
            """
            call_args = inspect.getcallargs(
                f, wrapped_self, role_id, *args, **kwargs)
            inherited = call_args['inherited_to_projects']
            initiator = call_args.get('initiator', None)
            target = resource.Resource(typeURI=taxonomy.ACCOUNT_USER)

            audit_kwargs = {}
            if call_args['project_id']:
                audit_kwargs['project'] = call_args['project_id']
            elif call_args['domain_id']:
                audit_kwargs['domain'] = call_args['domain_id']

            if call_args['user_id']:
                audit_kwargs['user'] = call_args['user_id']
            elif call_args['group_id']:
                audit_kwargs['group'] = call_args['group_id']

            audit_kwargs['inherited_to_projects'] = inherited
            audit_kwargs['role'] = role_id

            try:
                result = f(wrapped_self, role_id, *args, **kwargs)
            except Exception:
                _send_audit_notification(self.action, initiator,
                                         taxonomy.OUTCOME_FAILURE,
                                         target, self.event_type,
                                         **audit_kwargs)
                raise
            else:
                _send_audit_notification(self.action, initiator,
                                         taxonomy.OUTCOME_SUCCESS,
                                         target, self.event_type,
                                         **audit_kwargs)
                return result

        return wrapper

def send_saml_audit_notification(action, user_id, group_ids,
                                 identity_provider, protocol, token_id,
                                 outcome):
    """Send notification to inform observers about SAML events.

    :param action: Action being audited
    :type action: str
    :param user_id: User ID from Keystone token
    :type user_id: str
    :param group_ids: List of Group IDs from Keystone token
    :type group_ids: list
    :param identity_provider: ID of the IdP from the Keystone token
    :type identity_provider: str or None
    :param protocol: Protocol ID for IdP from the Keystone token
    :type protocol: str
    :param token_id: audit_id from Keystone token
    :type token_id: str or None
    :param outcome: One of :class:`pycadf.cadftaxonomy`
    :type outcome: str
    """
    initiator = build_audit_initiator()
    target = resource.Resource(typeURI=taxonomy.ACCOUNT_USER)
    audit_type = SAML_AUDIT_TYPE
    user_id = user_id or taxonomy.UNKNOWN
    token_id = token_id or taxonomy.UNKNOWN
    group_ids = group_ids or []
    cred = credential.FederatedCredential(token=token_id, type=audit_type,
                                          identity_provider=identity_provider,
                                          user=user_id, groups=group_ids)
    initiator.credential = cred
    event_type = '%s.%s' % (SERVICE, action)
    _send_audit_notification(action, initiator, outcome, target, event_type)

class _CatalogHelperObj(provider_api.ProviderAPIMixin, object):
    """A helper object to allow lookups of identity service id."""

def _send_audit_notification(action, initiator, outcome, target,
                             event_type, reason=None, **kwargs):
    """Send CADF notification to inform observers about the affected resource.

    This method logs an exception when sending the notification fails.

    :param action: CADF action being audited (e.g., 'authenticate')
    :param initiator: CADF resource representing the initiator
    :param outcome: The CADF outcome (taxonomy.OUTCOME_PENDING,
        taxonomy.OUTCOME_SUCCESS, taxonomy.OUTCOME_FAILURE)
    :param target: CADF resource representing the target
    :param event_type: An OpenStack-ism, typically this is the meter name that
        Ceilometer uses to poll events.
    :param kwargs: Any additional arguments passed in will be added as
        key-value pairs to the CADF event.
    :param reason: Reason for the notification which contains the response
        code and message description
    """
    if _check_notification_opt_out(event_type, outcome):
        return

    global _CATALOG_HELPER_OBJ
    if _CATALOG_HELPER_OBJ is None:
        _CATALOG_HELPER_OBJ = _CatalogHelperObj()
    service_list = _CATALOG_HELPER_OBJ.catalog_api.list_services()
    service_id = None

    for i in service_list:
        if i['type'] == SERVICE:
            service_id = i['id']
            break

    initiator = _add_username_to_initiator(initiator)

    event = eventfactory.EventFactory().new_event(
        eventType=cadftype.EVENTTYPE_ACTIVITY,
        outcome=outcome,
        action=action,
        initiator=initiator,
        target=target,
        reason=reason,
        observer=resource.Resource(typeURI=taxonomy.SERVICE_SECURITY))

    if service_id is not None:
        event.observer.id = service_id

    for key, value in kwargs.items():
        setattr(event, key, value)

    context = {}
    payload = event.as_dict()
    notifier = _get_notifier()

    if notifier:
        try:
            notifier.info(context, event_type, payload)
        except Exception:
            # diaper defense: any exception that occurs while emitting the
            # notification should not interfere with the API request
            LOG.exception(
                'Failed to send %(action)s %(event_type)s notification',
                {'action': action, 'event_type': event_type})

def _check_notification_opt_out(event_type, outcome):
    """Check if a particular event_type has been opted-out of.

    This method checks to see if an event should be sent to the messaging
    service. Any event specified in the opt-out list will not be transmitted.

    :param event_type: This is the meter name that Ceilometer uses to poll
        events. For example: identity.user.created, or
        identity.authenticate.success, or identity.role_assignment.created
    :param outcome: The CADF outcome (taxonomy.OUTCOME_PENDING,
        taxonomy.OUTCOME_SUCCESS, taxonomy.OUTCOME_FAILURE)

    """
    # NOTE(stevemar): Special handling for authenticate, we look at the outcome
    # as well when evaluating. For authN events, event_type is just
    # identity.authenticate, which isn't fine enough to provide any opt-out
    # value, so we attach the outcome to re-create the meter name used in
    # ceilometer.
    if 'authenticate' in event_type:
        event_type = event_type + "." + outcome

    if event_type in CONF.notification_opt_out:
        return True

    return False

def _add_username_to_initiator(initiator):
    """Add the username to the initiator if missing."""
    if hasattr(initiator, 'username'):
        return initiator
    try:
        user_ref = PROVIDERS.identity_api.get_user(initiator.user_id)
        initiator.username = user_ref['name']
    except (exception.UserNotFound, AttributeError):
        # Either user not found or no user_id, move along
        pass

    return initiator

emit_event = CadfNotificationWrapper

role_assignment = CadfRoleAssignmentNotificationWrapper


text:
Q: HTTPS for intranet website My situation is, I have a server running inside a home network which can be connected from the outside through VPN (Zerotier). This network connected to the internet through NAT, thus there is no real public IP.
Now there are some services that requires HTTPS. Also using self-signed certificate would be quite troublesome as some other service is running inside container (and adding root CA for everyone of them can be a real pain). For example, I have drone running inside a container, and it needs to communicate to the gitea server through https protocol (or at least the document mentioned that).
So I need a valid certificate to run those services.
I do own a few domain name, and a couple of website operated by a hosting provider.
Now I'd like to know what are my options to get a certificate to run my intranet websites?

A: You can make public DNS records for your domain with your LAN addresses. So, drone.mybigcompany.com pointing to 192.168.1.1. You can then obtain the certificate in all kinds of ways. Let's Encrypt will be difficult, but you can also buy certificates and use CNAME verification. You can buy one for less than $10 per year. No need for a fancy one.

redpajama_set_name: RedPajamaStackExchange


This must be completed and approved before you can serve as a Watch D.O.G.S. dad. Please allow a minimum of 2 weeks for processing.
Sign-up for one or more days on the Hidden Lakes Elementary Volunteer Spot calendar.
The front office will receive a notification that you have signed up!
Arrive to HLES at 7:30am on the day of your assignment dressed in your pre-purchased Watch D.O.G.S. apparel. If you didn't get a chance to order a shirt...no worries, we're still happy to have you!
Check in with the front office. You will be given a name badge and a specific itinerary for your day which will include time in your student(s) classroom as well as during their recess and lunch.
At the end of the day, check back in with the office to complete a quick online survey and then head outside to help with dismissal.
After that, you're free to head home and nap. You'll likely need it!
If you have any questions regarding the program, please call the front office at (817) 744-5000.
Order your Watch D.O.G.S. merchandise here.


(CN) – A Georgia coroner has determined that a 22-year-old college student found dead in restraints in the Chatham County jail died from blunt-force injuries to his head and body.
Matthew Ajibade, a local artist and student with bipolar disorder, was found dead in a restraint chair at the jail on January 1, after he was booked on domestic violence and obstruction of justice charges.
According to published reports, deputies arrested Ajibade after responding to a domestic violence call in the early morning hours of New Year Day, and observing him holding a woman under a blanket whose face was bruised and bloodied.
The Georgia Bureau of Investigation said in January that Ajibade became combative during the booking process, and that force was used to restrain him. It said deputies were injured during the altercation, and Ajibade was confined to restraint chair.
The Georgia Bureau of Investigation and the Internal Affairs Division of the sheriff's office conducted investigations, and in May, nine deputies were fired in connection with the man's death.
Chatham County Coroner Wessinger ruled Ajibade's death a homicide in May. His determination became public on Thursday.


After missing their shot at a 16-0 playoff record, the Warriors returned to Oracle Arena and finally secured the clinching victory they couldn't pull off after winning three games against the Cleveland Cavaliers in last year's NBA Finals.
Durant - who swapped the Oklahoma City Thunder for the Warriors in search of an elusive title - scored 39 points on 14 of 20 shooting with seven rebounds and five assists. "We were really good tonight", he said. "Our perspective, being blessed to play on this stage three years in a row, it's for these fans, for our organization, for these families". "I'm just proud to be part of something special and I just want to do it again". "They beat us, but I'm pretty sure we'll be back and we'll be ready to battle again".
Paul George would fit the bill, as he helped IN put up the best showing against the Cleveland Cavaliers IN the playoffs leading up to the NBA Finals by averaging 28 points along with 8.8 rebounds, 7.3 assists, and 1.8 steals.
The reigning National Basketball Association coach of the year returned for Game 2 of the Finals after a six-week absence from the bench.
Johnson, a rookie with the Los Angeles Lakers in 1980, averaged 21.5 points, 11.2 rebounds, 8.7 assists and 2.7 steals in 42.7 minutes per game during that Finals.
The Warriors will roll through their schedule as long as they employ Stephen Curry, Kevin Durant, Draymond Green, and Klay Thompson together.
"I couldn't sleep for two days", said Durant.
Green had sat out Game 5 a year ago, suspended because of flagrant foul point accumulation after he swiped at James' groin in Game 4. Kyrie Irving added 26 points and J.R. Smith 25 in a losing cause.
With fans and pundits questioning the motivations behind his move last summer, Durant had simply said he came to an already star-studded Warriors lineup to win championships. "If KD was the consolation prize to lose, thanks for that loss, and we're champs this year".
But coach Steve Kerr, who only returned to the team in time for game two of the finals after a period of medical leave to deal with complications arising from back surgery, joked afterwards: "We have very little talent actually, it's mostly coaching!" Not only would the trade give Cleveland a much better chance of beating Golden State, it would allow IN to remain relevant without George, who is widely rumored to be leaving IN free agency at the end of next season. "We went through for lack of a better term, basketball hell, in that sense of just being so close to getting the job done and not realizing that goal and having to think about that for an entire year and compartmentalize and just try to keep the right perspective about this season and learn the lessons that we learned".