text:
Medication-Assisted Treatment (MAT) Deli...
Closed; Awards Announced
Medication-Assisted Treatment (MAT) Delivery for Pregnant Women with Substance Use Disorders Involving Prescription Opioids and/or Heroin - Cycle 2 2017
Cycle 2 2017
Funding Applications Encouraged Around Topic Themes
PCORnet®, The National Patient-Centered Clinical Research Network
Rapid-Cycle Research Studies and Projects
Funding Application Submission Guidelines
Tips for Submitting a Responsive LOI
Merit Review Process
Merit Review Criteria
Reviewer FAQs
Meet Our Merit Reviewers
Merit Review Mentor Program
PCORI seeks to fund large, randomized controlled trials (RCTs) or well-justified observational studies that compare the effectiveness for maternal and neonatal outcomes of different models for comprehensive Opioid Use Disorder (OUD) treatment delivery for pregnant and post-partum women with different levels of addiction severity. Treatment must include maintenance medication, and delivery components must be evidence based or currently in use. PCORI expects the treatment programs to be well characterized to facilitate replication and dissemination efforts. Proposed studies must address actual choices faced by decision makers in the field of treatment delivery to pregnant women with OUD. They must involve patient populations that represent the U.S. population; be large enough to address estimates of hypothesized effectiveness differences; and be large enough to support evaluation of potential differences in treatment delivery effectiveness between patient subgroups if such comparisons are planned.
For this solicitation, applicants should document that they have consulted with patients and other stakeholders to identify the important decisional dilemmas and evidence needs that will drive development of the research questions or reference previously documented decisional dilemmas. All studies should compare models or components of models that are ready for implementation on a large scale, should the study results be positive. Proposals should have strong endorsement and study input by patient organizations, professional organizations, payer or purchaser organizations, and other relevant stakeholder groups. Because close to 50 percent of U.S. births are covered by Medicaid, partnerships and input from state Medicaid agencies or Medicaid Medical Directors or other representatives are particularly important. Successful applicants may have to develop formal arrangements. PCORI may require a project Study Advisory Committee (SAC)[A] that comprises national or regional organizations that represent—at a minimum—patients and families with lived experience; relevant clinicians; payers; and health plans. PCORI may also recommend scientific and methodological experts for this oversight role. The SAC advises and assists the research team with refining the study questions, outcomes, and protocol. PCORI expects applications to follow the RCT or observational study designs specified for the research questions.
Note that this PFA does not support applications to conduct cost-effectiveness analysis; systematic reviews; or development and evaluation of shared decision making, decision-support tools, or clinical practice guidelines.
The proposed studies must address the priority research questions identified in the main body of this PFA.
Guidance for Preparing Applications
How To Submit an Application
Online System Opens
Letter of Intent Deadline
Funds and Project Period
Funds Available Up To
Up to $16 million
Total Direct Costs
Maximum Project Period
Summary of Program
The Patient-Centered Outcomes Research Institute (PCORI) is launching this funding initiative to support patient-centered comparative clinical effectiveness research (CER) on the delivery of medication-assisted treatment (MAT) for pregnant women with Opioid Use Disorder (OUD). The study designs that PCORI will consider include large randomized controlled trials (RCTs) or well-justified observational studies. Each treatment program being compared should include two or more delivery models or components of delivery models that are documented to be efficacious or in common use, and should also be well characterized to facilitate replication and dissemination efforts. Through this PCORI Funding Announcement (PFA), PCORI seeks to fund studies with sufficient sample sizes to address the priority research questions. This program's goal is to generate valid evidence that is readily generalizable to the broader population of pregnant women with OUD.
For this PFA, applicants should follow the definition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) for OUD: "a problematic pattern of opioid use leading to clinically significant impairment or distress."
There is concern that using the term "MAT" suggests that medication is merely an ancillary part of the treatment rather than a central component. This PFA uses "MAT" because it is well defined, broadly used, and recognized. This PFA also uses "MAT" to distinguish it from approaches that focus on abstinence. If studies do not include maintenance medication, PCORI will consider them nonresponsive.
PCORI is particularly interested in large studies that can investigate potential Heterogeneity of Treatment Effects (HTEs) with respect to important sociodemographic and clinical characteristics (e.g., addiction severity, socioeconomic status, comorbid mental health conditions, or other clinical or patient/demographic characteristics proposed by investigators with an accompanying strong rationale).
Applications must address one or both of the priority research questions described in this PFA. Proposed studies should include comparison of two or more treatment models or components of treatment models. Treatment models should offer comprehensive treatment, including prenatal care, maintenance medication, and psychosocial services (on-site or by referral). Based on a 2016 Agency for Healthcare Research and Quality (AHRQ) systematic review by Chou et al.,[1] models of interest may include education and outreach and additional components.
If applicants propose comparing models or components of models that are in common use but without clear evidence of efficacy, they must document the extensiveness of their use and demonstrate how they will interpret study results.
Applicants must adequately define proposed comparators and describe how they will be measured. Proposed models must be evidence based or in current use.
Outcomes should include maternal measures, including illicit drug use, relapse, treatment entry, treatment retention, continuation of OUD care following delivery, patient quality of life, and anxiety/depression measures as well as perinatal and neonatal measures including pregnancy complications, preterm birth, birthweight, neonatal complications, and Neonatal Abstinence Syndrome. Studies should conduct periodic outcome assessments, including a follow-up period of three months or longer post-partum, to allow for the assessment of continuity of OUD care for the mother.
The prevalence of OUD has increased dramatically among pregnant women in parallel with the current opioid epidemic. The largest increase in heroin use between 2002 and 2013 was in women, and the proportion of pregnant women who enter OUD treatment and report prescription opioids as their primary substance increased from 1 percent in 1992 to 19 percent in 2012. Incidence of Neonatal Abstinence Syndrome increased two-fold from 2009 to 2012.
OUD in pregnant women is associated with potentially serious maternal, fetal, and neonatal risks, including increased morbidity and mortality, decreased quality of life, complications during pregnancy, Neonatal Abstinence Syndrome in the baby, and potential longer-term cognitive and behavioral effects for the child.
MAT, which combines maintenance medication (methadone or buprenorphine) with psychosocial services, is an evidence-based, clinically effective treatment for pregnant women with OUD. Compared to methadone, buprenorphine has a more favorable safety profile and improved birth outcomes, including reduced incidence and severity of Neonatal Abstinence Syndrome.
Pregnancy might motivate women to seek treatment, but stigma such as treatment setting or methadone usage; lack of access to treatment; and, in some states, fear of legal consequences are important barriers. Unlike methadone, physicians might offer buprenorphine, and it does not have the associated stigma. However, buprenorphine is not as well-known and fewer than half the counties in the United States offer it; the need for treatment continues to exceed availability. Provider barriers include lack of expertise regarding the treatment (particularly in this patient population), lack of adequate support for its delivery, and lack of access to mental health providers or services necessary for evidence-based MAT.
There is evidence that pregnant women with OUD need comprehensive care, and some existing models deliver such care. These models are based on treatment models for the general population, and they can be characterized by the support offered to the provider who prescribes the maintenance medication.
Support can be provided in the care setting by integrating prenatal care, opioid treatment, and psychosocial services, or by co-locating prenatal care and opioid treatment with a collaboration with community psychosocial services. These models can be compared with the default option, "usual care," in which a clinician provides prenatal care and then refers the patient to an opioid treatment center, or an opioid treatment center or office-based clinician provides OUD care and the patient is referred for prenatal care.
Support to the provider can also be offered remotely. In the "hub-and-spoke" model, the hub is an integrated center with addiction expertise that provides support remotely to spokes—offices located at some geographical distance from the hub, where clinicians offer prenatal care and opioid treatment to patients who live too far from the hub to receive care there. Further provider support to these spokes might include induction taking place elsewhere (e.g., in a hospital), psychosocial services delivered off-site by referral, or non-physician staff taking care of the logistics of treatment delivery.
Evidence Gaps
AHRQ recently commissioned a systematic review on MAT models in primary care settings that identified 12 models of MAT delivery currently in use, including one focusing on integrated prenatal care and MAT. As its main conclusion, AHRQ recommends research "to clarify optimal MAT models of care and to understand effective strategies for overcoming barriers to implementation."
Comprehensive care for pregnant women with OUD includes prenatal care, maintenance medication for opioid addiction, and psychosocial services. Although there is a dearth of comprehensive treatment programs for pregnant women—according to recent data, only 19 states have treatment programs for pregnant women specifically—existing models do successfully treat pregnant women with OUD. These models vary by integration level and range from entirely separate care, in which the patient receives prenatal care and referrals to addiction care elsewhere or vice versa, to fully integrated, comprehensive prenatal and addiction care, in which a clinician or a team of clinicians provide prenatal and addiction care, and psychosocial services are on-site. In between nonintegrated and fully integrated care is co-located care, in which a prenatal care provider is co-located with a clinician providing Office-Based Opioid Treatment (OBOT), and the patient is referred to psychosocial services. There are also hub-and-spoke models, in which the hub is an addiction treatment center that provides support to clinicians offering OBOT at some distance from the center. These models vary by resource intensity for the spoke sites (e.g., induction in an addiction clinic or hospital versus in a provider's office; on-site psychosocial services versus referral). Although these models might successfully treat pregnant and post-partum women with OUD, there are no studies comparing the effectiveness of these different approaches for pregnant and post-partum women.
To qualify for the buprenorphine waiver, which is required to be able to prescribe buprenorphine, clinicians must be able to refer patients to counseling services. Providers might organize weekly group counseling sessions on-site or provide brief counseling themselves, or they might refer to off-site services. Psychosocial services and referrals might also include mental health screenings and assessments, hepatitis C screening, contingency management, psychiatric care, and other specialty care. Although counseling is considered a key component of MAT, the format, frequency, and focus of counseling might vary, and its comparative clinical effectiveness for different subgroups of pregnant and post-partum women is unknown.
Research Topic Prioritization
The Medicaid Medical Directors Network identified the treatment of pregnant women with OUD as a priority in 2015 and 2016. PCORI met individually with representatives from key stakeholder groups, including midwives and obstetricians working in addiction treatment settings, addiction specialists, researchers who have focused on treatment of pregnant women with OUD, and individual Medicaid Medical Directors. PCORI also consulted the Advisory Panel on Improving Healthcare Systems, which consists of patient representatives, clinicians, researchers, payers, and purchasers.
Priority Research Questions
Applications should propose RCTs or well-justified observational studies that address the priority research questions noted below. The study should take place in clinical settings where pregnant patients with OUD typically receive care. The studies must be sufficiently large to be able to demonstrate differences in comparative clinical effectiveness in the study arms and to allow adequate power to detect the potential differences in treatment responses by patient clinical characteristics of interest. In considering approaches to their study design, conduct, and analysis, applicants should explicitly consider the tradeoffs of each element on the continuum while ensuring the validity of comparative clinical effectiveness findings. Interventions require some degree of flexibility in their use but must be sufficiently defined to be replicable in their dissemination and implementation in U.S. health care.
The priority research questions are:
What is the comparative clinical effectiveness of alternative models for comprehensive OUD treatment delivery on maternal and neonatal outcomes in pregnant and post-partum women with different levels of addiction severity?
- Comprehensive care includes prenatal care, MAT for addiction, and psychosocial care.
What is the comparative clinical effectiveness, in terms of maternal and neonatal outcomes, of remotely supported OUD treatment delivery to pregnant women that includes more versus fewer resource-intense approaches to induction and psychosocial support for office-based opioid treatment?
Given the anticipated size and scope of proposed studies submitted under this funding initiative, applications should carefully consider and provide details supporting how the target sample size will be met across all study sites (e.g., expected eligible patients, recruitment capacity, integration of research and clinical workflow, etc.). Applicants should factor into target sample size calculations details such as anticipated attrition, informed by past studies by the investigative team and studies found in the extant literature. Similarly, recruitment timelines should include careful consideration of feasibility within the context of the proposed study. Furthermore, the proposed studies should have well-characterized interventions and comparators. Proposed treatment models or model components must address actual clinical choices faced by patients, caregivers, and clinicians in specific practice settings.
PCORI is interested in receiving applications that propose to conduct direct comparisons of models or model components that have a strong empirical or clinical rationale and/or are supported by published literature. Applicants should include sufficient details about the organization and components of treatment delivery. PCORI is always interested in studying populations with important disparities, important comorbidities, or difficult social conditions.
The characteristics that apply to these research questions include:
Population/Patient Problem: Pregnant women with OUD as defined by the DSM-5 and infants born to women with OUD. Although women with Medicaid insurance account for close to 50 percent of U.S. births, applications can include women with private insurance and uninsured women.
Intervention and Comparators:
The following treatment delivery models vary by spectrum of integration:
Integrated prenatal care, OUD care, addiction medicine, and psychosocial service
Co-located prenatal care, OBOT, and collaboration with community psychosocial services
Prenatal care by clinician, referral to OUD treatment, or opioid treatment and referral to prenatal care ("usual care")
Models might offer remote support for providers, including hub-and-spoke models, in which the hub is a center with on-site addiction treatment expertise and spokes are office-based clinicians at geographical distance from the hub. The spokes serve women who live too far away from the hub. Core components of the spokes include prenatal care and office-based opioid treatment by the clinicians. The spokes might be more or less resource intensive regarding treatment delivery. For example, patient induction and stabilization might occur on-site or in a methadone clinic or hospital, and psychosocial services might occur on-site or by referral to community services. Specific psychosocial services, their format, and duration are important comparators. For example, both on-site and off-site services might vary by format and duration (e.g., medical management versus group counseling sessions on-site; referral to group versus individual counseling sessions).
Compared models need to be meaningfully different from each other and may include "usual care." However, "usual care" is often ill defined, difficult to quantify, and subject to considerable geographic and temporal variations, which limits interpretability, applicability, and reproducibility. If an applicant proposes "usual care" as a rational and important comparator in the proposed study, then the applicant must described it in detail and include an explanation of how the care given in the "usual care" group will be measured in each patient (to the extent possible) and how appropriate inferences will be drawn from its inclusion.
Outcomes: Studies need to include addiction-specific outcomes (e.g., illicit drug use, relapse, treatment entry, treatment retention, post-partum treatment continuation, patient quality of life, and anxiety/depression) and pregnancy and neonatal outcomes (preterm birth, pregnancy complications, birthweight, neonatal complications, and Neonatal Abstinence Syndrome).
Time: Repeated assessments need to occur to measure maternal and neonatal outcomes during pregnancy. An assessment should also occur three months post-partum to assess continuity of care for the mother.
Setting: Focus on settings representative of locations where pregnant and post-partum women with OUD typically receive care.
Funds Available
PCORI has allotted up to $16 million in total costs under this PFA to fund high-impact studies related to treating pregnant and post-partum women with OUD. The proposed budget for all studies under this initiative may be up to $4 million in direct costs as appropriate, depending on the specific priority research questions the study proposes to address. The anticipated project period is three to four years; in no case should the project period exceed four years. For this solicitation, applicants should document that they have consulted with representatives from key stakeholder groups to identify the important decisional dilemmas and evidence needs that will drive development of the research questions or refer to previously documented decisional dilemmas. Because Medicaid covers close to 50 percent of U.S. births, partnerships and input from state Medicaid agencies, Medicaid Medical Directors, or other representatives is important. Successful applicants will collaborate with PCORI staff upon award of the proposed studies to establish a project Study Advisory Committee (SAC) or other appropriate engagement body that comprises national or regional organizations that represent—at a minimum—patients or families with lived experience; relevant clinicians; payers; and health plans. In collaboration with the applicant, PCORI might recommend other representation, including individual patients with lived experience and other relevant stakeholders, among them scientific and methodological experts. The SAC advises and assists the research team with further refining the study questions, outcomes, and protocol.
Given the significant treatment costs associated with some interventions, the applications must specifically address—in the context of the proposed studies—the support from payers, health plans, industry sponsors, or others in covering the study interventions and non-study, protocol-related clinical costs and services rendered in the care processes. This is important because the interventions should be immediately implementable on a large scale if the study results provide significant evidence. Of particular concern would be different levels of co-payment between two arms in a comparative study. Ideally, cost-sharing barriers will be eliminated in the study arms or equalized. If the study design does not allow for either option, the applicant should describe why and should also account for differences in co-payment costs in the analysis of the study's findings.
PCORI expects that project budgets and duration will vary substantially, depending on the approach selected, needs for recruitment or primary data collection, length of follow-up, and analytic complexity. PCORI seeks efficient studies—such as those that take advantage of large populations already under observation, registries, and the supportive involvement of delivery systems or health plans—to enhance recruitment, data collection, and coverage of treatment-related costs. A prolonged recruitment period is not an acceptable rationale for longer studies.
II. Guidance for Preparing Applications
The proposed study should strive to meet the following requirements:
Focus on the priority research questions, with consideration of what is important to patients and other decision makers.
Demonstrate consultation with patients and other stakeholders or their representative groups, or reference previously documented decisional dilemmas to determine if the study is answering a critical question—one that, if adequately answered, would substantially improve decision making.
Receive endorsement by relevant patient organizations, clinician organizations, payer or purchaser consortia, and Life Sciences industry representatives as enough to potentially answer a critical question—one that, if adequately answered, would substantially improve decision making.
Propose a sample size that is sufficiently large to allow for precise estimation of hypothesized effect sizes. The sample size must also support testing of a priori hypotheses related to potential differences in effectiveness, depending on sociodemographic and clinical characteristics (HTE). When determining anticipated effect sizes, pay attention to basing such estimates on efficacy results observed in prior and adequately conducted clinical studies.
Examine diverse populations receiving care in real-world settings.
Have strong interest and support from host delivery systems and clinical care settings.
Specify broad and simple eligibility criteria that will allow for wide generalization of results while attending appropriately to any ethical concerns of excess risk in some patient subgroups.
As applicable, compare interventions that can be implemented in real-world settings and are known to be efficacious, effective, or in common use.
Include patient-reported outcomes (PROs) as primary outcomes, when appropriate.
Provide preliminary evidence of the potential for efficient recruitment, high participation rates, and appropriate oversight by local or centralized Institutional Review Boards (IRBs), including plans for streamlining or waiving individual informed consent in cases of low-risk interventions (if applicable). PCORI believes that the intensity of oversight and the complexity of informed consent procedures should be closely related to the degree of risk from study participation. Applicants must address this issue and should present evidence that the study will not encounter significant recruitment or participation barriers.
Adhere to all applicable PCORI Methodology Standards. The full application will require the applicant to identify the standards appropriate to the proposed study and to describe how the study team plans to address each standard.
In the case of RCTs, also adhere to current best practices (standardized inclusion or exclusion criteria; proper randomization; techniques to minimize the potential for missing data; and appropriate safety monitoring, including establishing a Data and Safety Monitoring Board [DSMB] or indicating why such a board is unnecessary).
To carry out studies that allow for adoption of the findings in a real-world setting, and to maximize the efficient use of resources, take care to prevent these trials from becoming more complex and onerous than necessary. We encourage the applicant to be creative and consider the following innovative strategies, as appropriate and feasible:
Identify and engage with major patient and stakeholder organizations that would implement study findings, as well as with existing local communities of patients and care providers to refine the research questions and study protocol, help monitor progress, and disseminate the findings.
Consult with patients and other stakeholders on their decisional dilemma and evidence needs, or reference previously documented decisional dilemmas in preparation for submitting the Letter of Intent (LOI) and the full applications.
Carefully describe pertinent evidence gaps and why the project questions represent decisional dilemmas for stakeholders (i.e., patients, clinicians, and policy makers). Similarly, applicants should document why project outcomes are especially relevant and meaningful endpoints for patients and their families.
Justify why the interventions you are comparing are viable treatment alternatives worthy of a significant PCORI investment to elucidate the best treatment delivery options for subgroups of pregnant patients with opioid use disorder.
Minimize disruption to participants' daily routines (e.g., minimize participant visits intended for study-assessment purposes and capture PROs during office visits, electronically, or via phone).
Design the study so that you can conduct it using routine clinic or office operations.
Use efficient methods to obtain participant consent while still meeting ethical and legal requirements.
Capitalize on the existing electronic health records (EHRs) and other computerized information to identify and recruit eligible patients, monitor study conduct and patient safety, and collect study outcomes information.
If data standardization and interoperability across study sites have not already been accomplished, develop methods that will enhance the standardization of data that are accessed from different EHR systems.
Nonresponsiveness
Applications will be considered nonresponsive to this PFA if the proposed research:
Tests efficacy (or comparative efficacy) within a tight, protocol-controlled research setting (as opposed to a more real-world and pragmatic CER).
Conducts a cost-effectiveness analysis in the form of dollar-cost per quality-adjusted life-year to compare two or more alternatives.
Directly compares the costs of care between two or more alternative approaches.
Measures the relative costs of care of two or more alternative approaches as the primary criteria for choosing the preferred alternative.
Conducts studies of the natural history of disease, instrument development, pharmacodynamics, and fundamental science of biological mechanisms.
Evaluates new or existing decision-support tools. This includes developing and evaluating a decision-support or shared-decision tool or system for patients, clinicians, or both.
Develops clinical prediction or prognostication tools.
Applications that include studies of these issues may measure and report use of any or all health services, but may not employ cost-effectiveness analyses.
PCORI does have an interest, however, in studying conditions that lead to high costs to the individual or to society. Thus, PCORI is also interested in studies that:
Address cost-related issues, such as the resources needed to replicate or disseminate a successful intervention.
Address issues related to the cost burden on the patient (e.g., deductibles and co-payment).
Evaluate interventions to reduce health-system waste or increase health-system efficiency.
PCORI considers applications that include studies of these issues without using cost-effectiveness analyses or comparing the costs of alternatives to be responsive.
PCORI discourages applications in the following categories and is likely to deem them nonresponsive:
Study of the natural history of disease
Fundamental science or study of biological mechanisms
Establishing efficacy for a new clinical strategy
Pilot studies intended to inform larger efforts
Comparisons of patient characteristics rather than clinical strategy options
Features of Patient-Centered Outcomes Research (PCOR)
PCOR helps people and their caregivers communicate and make informed healthcare decisions, allowing their voices to be heard in assessing the value of healthcare options. This research:
Assesses the benefits and harms of preventive, diagnostic, therapeutic, and palliative care to inform decision making, highlighting the choices that matter to people
Is inclusive of an individual's preferences, autonomy, and needs, focusing on outcomes that people notice and care about (including survival, functioning, symptoms, and health-related quality of life)
Incorporates a wide variety of settings and diversity of participants to address individual differences and barriers to implementation and dissemination
Directly compares clinical interventions that are available in the clinical settings
Obtains stakeholder perspectives to address the burdens to individuals, availability of services, and requirements for technology and personnel
Leveraging Existing Resources
PCORI encourages investigators to propose studies that leverage existing resources, such as adding PCOR to an existing large clinical trial or analyzing existing large databases that contain valuable and relevant information that can be used to answer important CER questions. PCORI is interested in studies that leverage existing research networks or consortia that would facilitate the conduct of large, multi-site studies called for in this PFA.
Applicants proposing use of an existing research network infrastructure (e.g., the National Patient-Centered Clinical Research Network [PCORnet]); research consortia; or related data resources (e.g., electronic medical records data from healthcare delivery systems or administrative claims data from public or commercial insurers) should address the following in the Research Plan (as appropriate), with sufficient specificity:
Identify and justify all participating research network entities (e.g., health plans, consortia projects and members, etc.) or, in the case of PCORnet, identify the names of the participating Clinical Data Research Networks (CDRNs), Patient-Powered Research Networks (PPRNs), and PCORnet Collaborative Research Groups that will be collaborating on the project. Also identify the affiliated study performance sites.
Demonstrate that the data source can comprehensively capture the study variables needed to assess the interventions, covariates, and outcomes.
Describe how you will manage data across study sites within the research network or the proposed research consortia, and whether you will use any dedicated data-coordinating functions or facilities.
As applicable, demonstrate familiarity with the existing network governance policies or data-use restrictions. Provide a study management structure that identifies roles, responsibilities, and decision-making authority across the proposed research consortia.
As applicable, provide a timeline for establishing data-use agreements.
As applicable, describe the network infrastructure resource(s) used to conduct the study (i.e., core research-support facilities, streamlined IRBs, contracting, engagement and consenting processes, standardized data resources training, etc.). Indicate the percentage of sites that have previously used centralized versus localized IRBs.
As applicable, provide documentation supporting the involvement of network leadership throughout the study (e.g., detailed Letters of Support, budgets, and Budget Justifications that cover the costs of the network's efforts). You can obtain Letters of Support from PCORnet from the Coordinating Center by submitting a request via the PCORnet Front Door.
Preliminary Data and Use of Accepted Measures
PCORI encourages investigators to design their research using valid patient-centered outcomes measures. Include preliminary data that supports the proposed measures in the study population. We also encourage investigators to consider those measures described in the Patient-Reported Outcomes Measurement Information System (PROMIS).
Methodological Considerations
Regardless of study design, applications must adhere to all relevant PCORI Methodology Standards. These include 48 individual standards that fall into 12 categories. The first five categories are cross-cutting and are relevant to most PCOR studies. Researchers should refer to all of these standards when planning and conducting their research projects. These five categories are:
Standards for Formulating Research Questions
Standards Associated with Patient-Centeredness
Standards on Data Integrity and Rigorous Analyses
Standards for Preventing and Handling Missing Data
Standards for Heterogeneity of Treatment Effect (HTE)
In addition to these five sets of standards, the first standard listed under "Standards for Causal Inference Methods"—(CI-1)—is cross-cutting and applicable to all PCOR studies.
The seven other standards categories will be applicable to particular study designs and methods. Applicants should use the standards in each of these categories for guidance when they are relevant to a particular study. These categories are:
Standards for Data Registries
Standards for Data Networks as Research-Facilitating Infrastructures
Standards for Causal Inference Methods
Standards for Adaptive and Bayesian Trial Designs
Standards for Studies of Medical Tests
Standards for Systematic Reviews
Standards for Research Designs Using Clusters
Most of these standards are minimal. The PCORI Methodology Standards reflect practices that applicants should follow in all cases, and all deviations need to be explained and justified. Applicants should address additional best practices—including relevant guidelines for conducting clinical trials developed by other organizations—in the application for PCORI funding. To help reviewers quickly identify adherence to a particular standard, applicants must cite each relevant PCORI Methodology Standard within the Methodology Standards Checklist, following the instruction in the checklist itself and in the Application Guidelines. Program staff use the checklist to evaluate applications.
Applicants should specifically discuss their capacity to measure such factors as differential adherence to chosen treatments (or participation in intervention programs) that could create or explain apparent differences in the effectiveness of the alternative interventions being compared in clinical populations.
Patient and Stakeholder Engagement
PCORI encourages all applicants to outline how patients and other stakeholders will participate as partners in various phases of the proposed research. Before completing this section of the Research Strategy, we encourage applicants to review the Engagement Rubric, which can be found in the PCORI Funding Center. Applicants should also review the PCORI Methodology Standards Associated with Patient-Centeredness and PCORI's Sample Engagement Plans. The rubric and Sample Engagement Plans are not intended to be comprehensive or prescriptive; instead, they provide a variety of examples to incorporate engagement, where relevant, into the research process.
PCORI expects applicants to consult with patients and other stakeholders on their decisional dilemma and evidence needs, or to reference previously documented decisional dilemmas in preparation for submission. To describe the decisional dilemma, state the specific clinical decision(s) or treatment choice(s) the decision makers confront and explain how the findings from the proposed research will inform those decisions. State why this decision—such as choosing a specific medication, surgical approach, or care delivery strategy to treat a condition or manage a specific population—is important to patients. Document the uncertainty patients and other stakeholders face when making this decision. Identify the patients and other stakeholders you consulted when determining that the proposed study addresses their evidentiary needs for decision making, and indicate your commitment to continue engaging them actively in the study. Similarly, applicants should document how the project outcomes are especially relevant and meaningful endpoints for patients and other stakeholders.
For this PFA, applicants need not demonstrate that patients and other stakeholders are already engaged as research team members at the time of application. However, the Engagement Plan should outline how patients and other stakeholders will participate as partners in various phases of the proposed research, once awarded. Applicants should describe their plan to form a SAC, or other appropriate engagement body, to ensure that a broad spectrum of patients and other stakeholders advise and assist the research team with refining the study questions, outcomes, and protocols. These patients and other stakeholders must include national or regional organizations that represent—at a minimum—patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders. PCORI might recommend additional representation in collaboration with the applicant, including individual patients with lived experience and other relevant stakeholders, such as scientific and methodological experts. The SAC or other appropriate engagement body should meet in person at least two times per year, and the budget should account for these engagement costs.
PCORI understands that engagement structures and approaches vary widely. Other engagement approaches, such as forming stakeholder groups, panels, task forces, working groups, and other bodies or involving individual patient and other stakeholder partners in various ways, are also permissible to employ—either in addition to or instead of—the formation of the SAC. For clarification in your application materials and for merit review purposes, please indicate which body or structure is filling the SAC requirements, including the requirements for in-person meetings at least two times per year and appropriate budgeting.
Populations Studied
PCORI seeks to fund research that includes diverse populations with respect to age, gender, race, ethnicity, geography, or clinical status, so that possible differences in CER can be examined (otherwise known as HTE). PCORI recognizes that some proposed studies might represent important PCOR opportunities, even in the absence of a broadly diverse study population. However, the burden is on the applicant in such cases to justify the study's importance in the absence of diversity and to discuss which subgroups are most important and how they will analyze them, including whether there will be power to examine the question of effectiveness in subgroups. PCORI is interested in including previously understudied populations for whom effectiveness information is needed, such as hard-to-reach populations or patients with multiple conditions. Thus, comparisons should examine the impact of the strategies in various subpopulations, with attention to the possibility that the effects might differ across subpopulations. PCORI has developed the following list of priority populations to guide our research and engagement efforts:
Racial and ethnic minority groups
Low-income groups
Children (age 0–17 years)
Older adults (age 65 years and older)
Residents of rural areas
Individuals with special healthcare needs, including individuals with disabilities
Individuals with multiple chronic diseases
Individuals with rare diseases
Individuals whose genetic makeup affects their medical outcomes
Patients with low health literacy or numeracy and/or limited English proficiency
Lesbian, gay, bisexual, and transgender persons (LGBT)
Veterans and members of the Armed Forces and their families
Project Budget and Duration
PCORI has devoted up to $16 million in total costs to this PFA. Applicants may request up to $4 million in direct costs for a research project period not to exceed four years (not including peer review). The maximum budget includes all research and peer-review-related costs. (Please refer to the Application Guidelines for further details.) At the time of contract execution, PCORI sets aside all of the funds associated with an awarded project to be made available throughout the contract's period of performance. Obligated funding is available for the duration of the project period. Note that PCORI will not cover costs for interventions that are being compared in the proposed study. (See Appendix 2: Allowable and Unallowable Costs in the Application Guidelines for details.)
Applicants should propose realistic budgets, project durations, and associated timelines. For those rare circumstances in which the estimated direct cost exceeds the maximum direct costs outlined in this PFA, please provide a detailed justification in your LOI that ties the extra expense to the project's success. PCORI will not approve all requests for additional funds. We will deny any request for a project period longer than five years. For further information regarding PCORI's policies about allowable and unallowable costs, refer to Appendix 2 in the Application Guidelines. Note that although subcontractor indirect costs are included in the prime applicant's direct-cost budget, these costs are not considered when determining adherence to the PFA's direct-cost limit.
A contract is the funding mechanism for this program. Total project funding is contingent upon successful programmatic and budget performance (e.g., meeting recruitment targets). Milestones and targets—as well as possible pilot phases for the sole purpose of assessing recruitment feasibility—should be included in the budget and will be negotiated at the time of the award. PCORI will expect awardees to provide corroborating evidence to receive continual funding support. Some of the activities that will be considered during negotiations include:
Developing a study protocol and procedure manual for the intervention
Assigning roles and responsibilities to study team members for project implementation
Forming a SAC or other appropriate engagement body
Obtaining clearances from all institutional and community partners, including IRB approvals
Establishing a DSMB or providing a clear description of why one is unnecessary
Executing all subcontractor agreements
Agreeing on eligible patient populations for study recruitment
Identifying barriers to patient recruitment in the study and addressing these barriers effectively
Demonstrating successful recruitment during a pilot phase (if indicated)
Refer to the Application Guidelines for a list of additional PFA-specific project milestones.
PCORI is particularly interested in applications involving community and commercial organizations that can help researchers design, implement, disseminate, and sustain effective interventions. We encourage applications that include novel collaborations with accreditation organizations, credentialing bodies, educational enterprises, patient advocacy groups, industry, professional societies, and subspecialty societies.
This component (up to five pages) is included in the Research Plan Template. Describe the protection of human subjects involved in your proposed research. PCORI follows the Federal Policy for the Protection of Human Subjects (45 CFR part 46), including the Common Rule. For more detailed information, please see Section 5, titled "Human Subjects Research Policy" in the Supplemental Grant Application Instructions for All Competing Applications and Progress Reports, which is issued by the U.S. Department of Health and Human Services. PCORI does not require that applicants comply with sections of this policy referring to requirements for federal-wide assurance or to standards for including women, minorities, and children. Awardees must also comply with appropriate state, local, and institutional regulations and guidelines pertaining to the use of human subjects in research.
PCORI requires awardees to ensure that there is a Data and Safety Monitoring Plan, which might include the need to appoint a DSMB, as provided in the PCORI Policy on Data and Safety Monitoring Plans for PCORI-Funded Research.
PCORI merit reviewers will examine plans for protection of human subjects in all applications and may provide comments regarding the plans (see How To Evaluate Human Subjects Protections). Reviewers' comments on human subject research are not reflected in the overall application score, but PCORI staff might use them during potential funding negotiations. Final determinations about the adequacy of human subject protections rest with the IRB or international equivalent that has jurisdiction for the study.
The Awardee Institution, whether domestic or foreign, bears ultimate responsibility for safeguarding the rights and welfare of human subjects in PCORI-supported activities.
Required Education of Key Personnel on the Protection of Human Subject Participants
PCORI requires all applicants to adhere to the National Institutes of Health (NIH) policy on education in the protection of human subject participants in the conduct of research. This applies to all individuals listed as key personnel in the application. The policy and FAQs are available on the NIH website.
Data Management and Data-Sharing Plan
PCORI encourages openness in research and making research data available for purposes of replication and reproducibility. Although not required to be submitted as a component of the research application, if an award is made, the awardee must develop and maintain a plan addressing data management and data sharing of research project data in a manner that is appropriate for the nature of the research project and the types of research project data. This must be done in a manner that is appropriate for the research project, and in a manner consistent with applicable privacy, confidentiality, and other legal requirements.
Peer Review and Release of Research Findings
PCORI has a legislative mandate to ensure the scientific integrity of the primary research it supports and to make study findings widely available and useful to patients, clinicians, and the general public within a specific timeframe. Accordingly, the PCORI Board of Governors (Board) adopted the Process for Peer Review of Primary Research and Public Release of Research Findings.
In summary, Awardee Institutions are required to submit to PCORI for peer review a draft final research report that provides the methodological details, describes the main study results, and interprets the findings in clinical or other decisional contexts. Subject matter experts (SMEs); individuals with expertise on research methodology or biostatistics; and patients, caregivers, and other healthcare stakeholders will review the draft final research report. After Awardee Institutions have responded to reviewers' comments to PCORI's satisfaction, the report will be accepted and considered final. PCORI will then prepare a 500-word abstract summarizing the study results for patients and the general public, which the Awardee Institution will review and approve.
PCORI will post the following materials on its website no later than 90 days after the draft final research report is accepted: (1) a 500-word abstract for medical professionals; (2) a standardized summary of the study results for patients and the general public; (3) a link to the study record on ClinicalTrials.gov (as applicable); and (4) ancillary information, including conflict-of-interest disclosures. The final research report, along with anonymized reviewer comments, will be made publicly available on the PCORI website no later than 12 months after its acceptance, except by prior mutual agreement with the Awardee Institution.
III. How To Submit an Application
Letter of Intent (LOI)
Applicants should download the Cycle 2 2017 Medication-Assisted Treatment (MAT) Delivery LOI Template from the PCORI Funding Center. They must complete the document and convert it to a PDF with a three-page limit. PCORI suggests including all references as in-text citations using American Medical Association citation style, but we do accept other citation styles. Do not upload additional documents as part of your LOI, such as Letters of Endorsement or Support, because they are not requested at this stage. Their inclusion will result in LOI rejection without review. Please visit the PCORI Funding Center for additional applicant resources, including the FAQ and required templates.
Please answer all of the questions in the LOI Template. This includes the question on brief justification for the proposed cost of the study. Providing the answer "costs not to exceed $4 million" is not sufficient. Upload your document to PCORI Online. The deadline for LOI submission is July 25, 2017, by 5 p.m. (ET).
LOI Review
PCORI evaluates LOIs based on the following criteria:
Whether the proposed topic addresses the priority research questions identified in this PFA
Importance of the specific research questions (comparison), as evidenced by critical gaps identified by clinical guidelines developers or recent relevant systematic reviews
A size or scope sufficient enough to have a significant impact on patient outcomes or healthcare practice
Clarity and credibility of applicants' responses to the LOI questions, as well as their justification of the proposed study size, citing published estimates, including effect sizes, standard deviations, and the need for rigorous comparative analysis of important subgroups
Prior relevant experience
Programmatic fit and balance, considering whether the research study question and design comply with requirements in this PFA
Adherence to the administrative and formatting requirements listed in the Application Guidelines, specifically the three-page limit for the LOI
Only applicants whose LOIs are deemed most responsive to this PFA will be invited to submit a full application. Notification of denial or approval to submit an application will occur no later than August 22, 2017. Please refer to the Application Guidelines for information on how to submit your LOI via PCORI Online.
You are invited to submit an application based on the information provided in the LOI. Any changes to the following require PCORI approval:
Specific aims
Principal Investigator (PI) (Contact PI and PI #2)
You may email requests for changes or questions to [email protected].
Note: A PI can only submit one LOI per PFA. However, an individual listed as a PI on one LOI can be listed as and serve in another non-PI role (e.g., co-investigator or consultant) on other LOIs within the same PFA during the same cycle. A PI may submit multiple LOIs to different program PFAs in a cycle, but the PI must ensure that the research topics and projects are not similar. If a PI submits an LOI to multiple program PFAs, LOIs that exhibit scientific overlap or that appear to be duplicate submissions will be disqualified. PCORI will contact the PI and provide him or her with an opportunity to choose the PFA to which he or she would like to apply. This applies to single- and dual-PI submissions.
Submission Dates
LOIs and applications must be submitted in accordance with the published dates and times listed in the Overview section of this PFA and in the PCORI Funding Opportunities.
PCORI Online System
To submit an application, you must register with PCORI Online and submit both an LOI and an application for each cycle in which you are applying.
IV. Applicant Resources
Medication-Assisted Treatment Application FAQs (Cycle 2 2017)
Medication-Assisted Treatment FAQs (Cycle 2 2017)
Targeted PFA Application Guidelines (Cycle 2 2017)
V. Merit Review
PCORI's merit review process is designed to support the following goals:
Identify applications that have the strongest potential to help patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders make informed decisions to improve patient outcomes.
Implement a transparent, fair, objective, and consistent process to identify these applications.
Elicit high-quality feedback that reflects a diversity of perspectives to ensure that the PCORI-funded research reflects the interests and views of patients and other stakeholders and those who care for them, and that it meets the criteria for scientific rigor.
Fund projects that fill important evidence gaps and have strong implementation potential.
Regularly evaluate and continually improve the merit review process and policies in support of PCORI's mission.
PCORI merit review is a multiphase process that includes PFA development; staff evaluation of LOIs; the review panel's preliminary review of full applications; an in-person panel discussion of a subset of full applications (identified by PCORI's Research Priority Area Program staff and based on the preliminary review and program priorities); the Selection Committee's recommendation of applications for funding; and, finally, Board award approval.
Preliminary Review
PCORI conducts rigorous merit review of the full applications it receives. Note that PCORI may eliminate applications from the review process for administrative or scientific reasons (e.g., non-responsiveness). An application may be administratively withdrawn if it is incomplete; submitted past the stated due date and time; or does not meet the formatting criteria outlined in the Application Guidelines, in the PCORI templates, and in PCORI Online. An application can be scientifically withdrawn if it is not responsive to the guidelines described in this PFA, describes research that is not comparative, includes a cost-effectiveness analysis, or otherwise does not meet PCORI programmatic requirements.
PCORI Merit Review Officers (MROs) recruit each panel based on the number of and topic areas represented by invited LOIs. MROs recruit the Panel Chair, scientist reviewers who are SMEs, patient representatives, and representatives of other stakeholder groups. All panel members receive training during the review cycle to ensure that they understand the programmatic and organizational goals of review.
We designed the table below to help applicants understand how the PCORI merit review criteria align with criteria from other funding organizations with which applicants might be familiar (e.g., NIH). Though PCORI's criteria do map to most NIH criteria, there are areas where we ask for different information (i.e., PCORI does not include a criterion that tracks to NIH's innovation criterion, but does include criteria evaluating patient-centeredness and engagement), reflecting PCORI's unique approach.
Crosswalk of PCORI Merit Review Criteria with NIH Criteria
1. Potential for the study to fill critical gaps in evidence
2. Potential for the study findings to be adopted into clinical practice and improve delivery of care
3. Scientific merit (research design, analysis, and outcomes)
4. Investigator(s) and environment
PCORI-only Merit Review Criteria
PATIENT-CENTEREDNESS/ENGAGEMENT
5. Patient-centeredness
6. Patient and stakeholder engagement
Application Merit Review
Below are PCORI's merit review criteria. PCORI's merit review panels use these criteria during the preliminary and in-person review phases to evaluate and score all submitted applications and to ensure consistency and fairness in application evaluation.
Criterion 1. Potential for the study to fill critical gaps in evidence:
The application should address the following questions:
Does the application convincingly describe the clinical burden?
Does the application identify a critical gap in current knowledge as noted in systematic reviews, guideline development efforts, or previous research prioritizations?
Does the application identify a critical gap in current knowledge, evidenced by inconsistency in clinical practice and decision making?
Would research findings from the study have the potential to fill these evidence gaps?
Criterion 2. Potential for the study findings to be adopted into clinical practice and improve delivery of care
The application should describe how evidence generated from this study could be adopted into clinical practice and delivery of care by others. The application should also address the following questions:
Does the application identify who will make the decision (i.e., the decision maker) or use (i.e., the end-user) the study findings (not the intervention) this study produces, such as local and national stakeholders?
Does the application identify potential end-users of study findings—such as local and national stakeholders—and describe strategies to engage these end-users?
Does the application provide information that supports a demand for this kind of a study from end-users?
Would this study's research findings have the potential to inform decision making for key stakeholders? If so, provide an example. How likely is it that others could reproduce positive findings, resulting in improvements in practice and patient outcomes? Identify the potential barriers that could hinder others from adopting the intervention.
Does the application describe a plan for how to disseminate study findings beyond publication in peer-review journals and at national conferences?
Criterion 3. Scientific merit (research design, analysis, and outcomes)
The application should show sufficient technical merit in the research design to ensure that the study goals will be met. The application should also address the following questions:
Does the application describe a clear conceptual framework anchored in background literature which informs the design, key variables, and relationship between interventions and outcomes being tested?
Does the Research Plan describe rigorous methods that demonstrate adherence to the PCORI Methodology Standards?
Is the overall study design justified?
Are the patient population and study setting appropriate for the proposed research question?
Does the application provide justification that the outcome measures are validated and appropriate for the population?
Are each of the comparators (e.g., active intervention arm and comparator arm) described clearly justified? If "usual care" is one of the arms, is it adequately justified and will it be sufficiently measured?
Are the sample sizes and power estimates appropriate? Is the study design (e.g., cluster randomized design, RCT, or observational study) accounted for, and is the anticipated effect size adequately justified?
Is the study plan feasible? Is the project timeline realistic, including specific scientific and engagement milestones? Is the strategy for recruiting participants feasible? Are assumptions about participant attrition realistic, and are plans to address patient or site attrition adequate?
Criterion 4. Investigator(s) and environment
This criterion should assess the appropriateness (e.g., qualifications and experience) of the investigator(s)/team and the environment's capacity (e.g., resources, facilities, and equipment) to support the proposed project. It should not be an assessment of the institution's quality.
The application should also address the following questions:
How qualified are the PIs, collaborators, and other researchers to conduct the proposed activities? Is there evidence of sufficient clinical or statistical expertise (if applicable)?
Does the investigator or co-investigator have demonstrated experience conducting projects of a similar size, scope, and complexity?
If the project is collaborative or dual-PI, do the investigators have complementary and integrated expertise? Are the leadership, governance, and organizational structures appropriate for the project?
(Dual-PI Option Only) Does the Leadership Plan adequately describe and justify PI roles and areas of responsibility?
Is the level of effort for each team member appropriate for successfully conducting the proposed work?
Does the application describe adequate availability of and access to facilities and resources (including patient populations, samples, and collaborative arrangements) to carry out the proposed research?
Is the institutional support appropriate for the proposed research?
Criterion 5. Patient-centeredness
The application should demonstrate that the study focuses on improving patient-centered outcomes and employs a patient-centered research design (i.e., a design informed or endorsed by patients). (Note: The study can be patient-centered even if the end-user is not the patient, as long as patients will benefit from the information.)
Does the application include a thorough description about which outcomes (both benefits and harms) are important to patients, and are those outcomes included in the study plan?
Does the application provide information that indicates that closing the evidence gap is important to patients and other stakeholders?
Are the interventions being compared in the study available to patients now, and are they the best options for comparison (including whether patients and their healthcare providers would choose them for managing the condition being studied)?
Criterion 6. Patient and stakeholder engagement
The application should demonstrate the engagement of relevant patients and other stakeholders (e.g., patients, caregivers, clinicians, policy makers, hospitals and health systems, payers [insurance], purchasers [business], industry, researchers, and training institutions) in the conduct of the study. Quality of engagement should be evaluated based on scope, form, and frequency of patient and stakeholder involvement throughout the research process.
Does the application provide a well-justified description of how the research team incorporates stakeholder involvement? Does the study include the right individuals (e.g., researchers, patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders) to ensure that the projects will be carried out successfully?
Does the application show evidence of active engagement among scientists, patients, and other stakeholders throughout the research process (e.g., formulating questions, identifying outcomes, monitoring the study, disseminating, and implementing)? Are the frequency and level of patient and stakeholder involvement sufficient to support the study goals?
Is the proposed Engagement Plan appropriate and tailored to the study?
Are the roles and the decision-making authority of all study partners described clearly?
Are the organizational structure and resources appropriate to engage patients and stakeholders throughout the project?
In-Person Review
During preliminary review, all administratively and scientifically compliant applications are evaluated and scored based on PCORI's merit review criteria, including evaluating adherence to the PCORI Methodology Standards. After PCORI completes the preliminary review, PCORI program staff members evaluate panel scores and critiques to identify a subset of applications for merit reviewers to discuss at the in-person review meeting. Not all submitted applications move forward to in-person review.
During the in-person review, merit reviewers meet to discuss applications and to clarify the merits of the proposed research. They also identify areas for improvement. Each application is re-scored based on the content of discussion. The Panel Chair and PCORI MRO lead the in-person panel meeting and ensure that all applications receive a fair and thorough review according to the standards outlined in the PFA.
Post-Panel Review
After the in-person meeting, PCORI program staff evaluate final merit review panel scores and comments, identify duplication or synergy among funded projects, and consider the fit of applications within the programmatic vision. Program staff members then recommend projects to a Selection Committee, which includes members of the Board. The Selection Committee considers recommendations and works with staff to identify a slate of applications for possible funding based on merit review scores, programmatic balance and fit, and PCORI's strategic priorities. This slate then goes to the Board for consideration and approval.
In addition, PCORI evaluates applicant risk before issuing an award. Factors considered include financial stability, quality of management systems, audit findings, and past performance on PCORI awards (e.g., compliance with PCORI reporting requirements, conformance to PCORI terms and conditions on previous awards, and timely achievement of milestones). Based on the risk assessment, PCORI may impose special terms and conditions on awardees or withhold contract issuance until such business risks are mitigated. PCORI will not award new contracts to current awardees with overdue reports (progress, interim, final, etc.) until the awardees have submitted the overdue reports.
Summary Statements and Funding Recommendations
Applicants receive summary statements approximately two weeks before funding decisions are announced. If an application progresses to in-person discussion, the applicant will receive a summary statement that includes:
In-person panel discussion notes
Final average overall score
Preliminary reviewer critiques
Application quartile, which provides information for applicants to understand how they did relative to other discussed applications
Summary statements for applications that do not progress to in-person discussion include only the preliminary reviewer critiques.
PCORI makes funding recommendations by identifying meritorious applications that fit the programmatic needs and satisfactorily address the merit review criteria while adhering to the PCORI Methodology Standards. PCORI also considers the funds allotted for the current PFA when deciding which applications to recommend to the Board for approval. Applicants to this current cycle's PFA will receive summary statements and notification of the funding status of their application no later than May 2018.
[A] The intent of the SAC described in the PFA is to ensure that a broad spectrum of patients and other stakeholders advise and assist the research team with refining the study questions, outcomes, and protocols. These patients and other stakeholders must include national or regional organizations that represent—at a minimum—patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders. PCORI may recommend additional representation in collaboration with the applicant, including individual patients with lived experience and other relevant stakeholders, such as scientific and methodological experts. However, PCORI understands that engagement structures and approaches vary widely. Other engagement approaches, such as forming stakeholder groups, panels, task forces, working groups, and other bodies, or involving individual patient and other stakeholder partners in various ways are also permissible to employ—either in addition to or instead of—the formation of the SAC. The SAC provision is not meant to require that a separate governance or advisory entity be established beyond the study governance and advisory structure the awardee has planned, if an applicant already has an approach for including the relevant and required patient and other stakeholder partners. For clarification in your application materials and for merit review purposes, please indicate which body or structure is filling the SAC requirements, including the requirements for in-person meetings at least two times per year and appropriate budgeting.
[1] Chou R, Korthuis P, Weimer M, Bougatsos C, Blazina I, Zakher B, Grusing S, Devine B, McCarty D. Medication-Assisted Treatment Models of Care for Opioid Use Disorder in Primary Care Settings. Technical Brief No. 28 (Prepared by Pacific Northwest Evidence-based Practice Center under Contract No. 290-2015-00009-I). AHRQ Publication No. 16(17)-EHC039-EF. Rockville, MD: Agency for Healthcare Research and Quality: December 2016.


text:
.PHONY: check
check: local-tests basic-tests migrations-tests

.PHONY: local-tests
local-tests:
	./mzb_local_tests.py -v

.PHONY: basic-tests
basic-tests:
	./mzb_basic_tests.py -v --process-timeout=1200 --processes=10

.PHONY: migrations-tests
migrations-tests:
	./mzb_migrations_tests.py -v

.PHONY: quick-test
quick-test:
	./mzb_quick_test.py -v

.PHONY: clean
clean:
	-@rm -rf data.*
	-@rm -rf log.*

redpajama_set_name: RedPajamaGithub


HOUSING FOR VETERANS IN A MILITARY TOWN.
Despite being home to two military bases and the Airforce Academy, Colorado Springs had a long overdue need for permanent supportive housing for homeless veterans. When Freedom Springs opens in 2019, the community will have its first supportive housing development; one that will provide permanent housing, rental assistance, and supportive services for more than 25 formerly homeless veterans.


Westmont college orders only. Please use the coupon code given to you for your discount on the checkout page near the totals. The coupon will work on any Nike shoe that arenot on sale no matter what section it is in.


Spare Battery for the Plantronics Wireless HS CS510 & CS520, W710 & W720, w removal tool.
Spare Battery for the Plantronics Wireless HS CS510 & CS520, W710, W720, W410 & W420. comes with removal tool.
Plantronics APU-76 EHS Cable Connects CS500 Series Wireless Headsets to PC via USB.


A creature of the other world, an unnatural entity bent on chaos and carnage, has come to stalk the nighttime streets of the Galicien city of Davillon. There's never a good time for murder and panic, but for a community already in the midst of its own inner turmoil, this couldn't possibly have come at a worse one.
Not for Davillon, and not for a young thief who calls herself Widdershins.
It's been over half a year since the brutal murder of Archbishop William de Laurent during his pilgrimage to Davillon. And in all that time, Widdershins has truly tried her best. She has tried to take care of Genevieve's tavern and tried to make a semihonest living in a city slowly stagnating under the weight of an angry and disapproving Church. She has tried to keep out of trouble, away from the attentions of the Davillon Guard and above the secrets and schemes of the city's new bishop.
But she's in way over her head, with no idea which way to turn. The Guard doesn't trust her. The Church doesn't trust her. Her own Thieves' Guild doesn't trust her.
Too bad for everyone, then, that she and her personal god, Olgun, may be their only real weapon against a new evil like nothing the city has ever seen.
After reading Thief's Covenant, the first book in this series (my review), and absolutely loving it, I had very high hopes for False Covenant. I was already a loyal fan to the series, even having only read one book, but False Covenant changed my status from "loyal fan" to "die-hard, I need every book Ari Marmell has ever written, obsessed-fan-girl."
It's a major page turner! Literally, my eyes were glued to the pages, and there wasn't a single thing that could tear me away. I even sat at my kitchen table eating dinner while reading. I got some weird looks from my family, but it was totally worth it!
And there's a new, and much creepier antagonist in this book! Just reading about him gave me goosebumps. It amazed me how Ari Marmell managed to give even the bad guy a really interesting and dynamic personality, instead of just making him "the evil dude." I actually really enjoyed reading the scenes with him.
Overall, I give False Covenant a solid 5 out of 5 stars. This is a series not to be overlooked. It has everything a fan of fantasy could possibly want, and even those afraid to venture into high fantasy will love it. Anyone looking for a great story, check out the Widdershins Adventures novels.


text:
class Sys::RolesController < ApplicationController
  include Sys::BaseFilter
  include Sys::CrudFilter

  model Sys::Role

  menu_view "sys/crud/menu"

  private
    def set_crumbs
      @crumbs << [:"sys.role", sys_roles_path]
    end

  public
    def index
      raise "403" unless @model.allowed?(:edit, @cur_user)
      @items = @model.allow(:edit, @cur_user).
        order_by(name: 1).
        page(params[:page]).per(50)
    end
end


Trailtopia is a team of family, friends and fellow adventurers dedicated to providing outstanding, nutritious and convenient camp cuisine for your outdoor adventures. We are inspired by the great outdoors. We are fueled by our passion for exploration and innovation. We carefully source the finest ingredients and prepare them for you in our kitchen before shipping them your way. We believe it's easier to eat out of our bags than anything else out there. We love our products, and you will too.
This rich chocolate pudding is loaded with mini marshmallows and toasted slivered almonds. Yum! We blend our pudding from scratch with Premium Cocoa and a hint of Madagascar Bourbon Pure Vanilla.


text:
DC Releases Official 60+ Character Postcard to Celebrate FanDome
By Pierre Chanliau August 20, 2020
It certainly was a blow to fans having DC FanDome suddenly split in half just three days before the launch of the event. However, fans still have a lot to be excited about this coming Saturday, as shown in the latest trailer for the online convention .
Despite Saturday being a day dedicated mostly to the movies, the people behind the comics are just as excited as fans. Not only has Jim Lee drawn his own interpretation of Robert Pattinson's Batman , he has also collaborated with other artists to create something truly spectacular.
Posted by Mikhail Villarreal on Twitter , he shows three special postcards drawn by Jim Lee promoting DC FanDome, that then combine into one. Villarreal helpfully points out that the pencil work was, of course, done by Jim Lee, which was then inked by Scott Williams , and finally colored by Alex Sinclair .
Jim Lee as Penciler, Scott Williams as Inker, and Alex Sinclair as Colorist
Javicia Leslie Batwoman Art
It's impressive the amount of characters that Lee was able to include into this promotional piece of artwork for the event. He was able to include practically everyone from all the CW shows like The Flash, Legends of Tomorrow, Black Lightning, and even Stargirl.
Lee also managed to feature members of the recently revived Young Justice cartoon series, along with the more childish incarnations of the Teen Titans. Heck, Jim Lee even included both versions of Supergirl from the live-action series and the cartoon.
Mostly interestingly, right between the two of them at the bottom of the drawing, is what can only be Javicia Leslie dressed as Batwoman , which makes this technically the first image of Leslie in the costume. This could indicate that fans might be able to see an image of the actress in full costume at DC FanDome in September.
Wonder Woman Batman Batgirl Superman Supergirl Doom Patrol (Series) Teen Titans (Team) Cyborg DC FanDome DCU

redpajama_set_name: RedPajamaCommonCrawl


Rosewood Geranium Oatmeal bar soap is handcrafted from all natural ingredients. It is scented with a proprietary blend of essential oils. This is a vegan bath, body and face soap. It is made with a very gentle blend of emollient oils.
This soap lathers well and is very gentle on the skin. It is handcrafted by the cold process method of soap making. It contains pure essential oils, natural and organic plant based oils and other select natural ingredients.
The aroma is a fresh, sweet rosy bouquet with undertone hints of peppery roots. It is a very uplifting and enlivening scent with a balancing effect.
The color is a light-yellow green to soft beige, off white. And the bar is topped off with whole grain rolled oats.
Each Bar is created with the skin softening, emollient, gentle oils of organic virgin olive and organic virgin coconut. They also include sustainable organic palm fruit oil, organic hemp seed oil, organic shea nut butter and aloe butter. These ingredients combined result in a remarkable bar of soap. Then each bar is infused with mild skin loving aloe vera, colloidal oatmeal and plantain.
These soaps are handmade in small batches using the cold process method. These soaps are hand cut and weigh 5 oz.
Each bar measures about 3.25" wide by 3" high by 1" deep.
Ingredients: Organic Extra Virgin Olive Oil, Distilled Water, Organic Virgin Coconut Oil, Sustainable Organic Palm Fruit Oil, Sodium hydroxide, Organic Hemp Seed Oil, Organic Virgin Shea Nut Butter, Aloe Butter, Aloe Vera Gel Juice, *Fragrance, Colloidal Oatmeal Oat Flour, Grapefruit Seed Extract, Plantain, Whole Grain Rolled Oats.
Because our soaps are hand crafted and made from natural Ingredients, there will be variations between each bar. We have no control over the colors, scents or constituent properties of these natural ingredients. Each bar is truly unique. The photos are a representation of the current batch.
We use polyolefin film to wrap and protect our Soaps from handling, it is acid free, FDA approved for direct food contact and it is recyclable. Then the soap label is affixed. After removing soap please recycle packaging.


Christmas in a Scary World | Ron Hutchcraft Ministries, Inc.
In the Charlie Brown Christmas special, Linus comes on stage and quotes the angel's words from the Christmas story: "Fear not, for, behold, I bring you good tidings of great joy." Man, we could use some of that good news this Christmas!
With life-threatening terrorism and disease all over the news, that "fear not" that's repeated four times in the Christmas story is more than welcome. But why "fear not" when there's plenty to be afraid of? Because the Bible says about Jesus: "They will call Him Immanuel, which means, 'God with us'" (Matthew 1:23).
The God who rules the galaxies - who's bigger than anything that's bigger than us. Moving into your life. But it couldn't happen without Jesus coming - and dying for the sin that keeps us away from God. And this is no time to be without Him.
You know, we'd love to help you grab the life-saving, all-powerful hand of Jesus. Just text us at 888-NEED-HIM or ChatAboutJesus.com.
Because, as the Bible says, His "perfect love expels all fear" (1 John 4:18).


Improvements on refrigeration and air flow system.
Development of new features and functionalities.
Prepare experiment set up, plan a test plan, execute the test plan, evaluate the test results and report with required documentation.
Cost reduction, quality improvement, energy optimization and benchmarking activities.
Adaptation of existing products to new requirements of market demands.
Support to the project teams regarding lab relevant activities, as a project team member.
Responsible for remote access vending products with clouds.
Design and production of the first prototype of the products.
Analysis of problems in fluid mechanics, thermal and mixing.
Automate and improve the existing products.
Product concept, prepare schematic and maintain BOM.
Web site design and small business inventory management software.


Epinephrine administration is a hot topic in wilderness medicine and for programs responsible for the medical care of participants. As access to epinephrine becomes more common, it is important for program staff and individuals with exposure to epinephrine to be familiar with the variety of devices available. Longleaf Wilderness Medicine has produced a reference sheet detailing the variety of epinephrine auto-injectors on the market. In addition to knowing the products available, it is important to know your state's laws regarding administration of epinephrine.


text:
Home GOP
Key GOP Leaders React After Pres Trump Offers A Deal To Secure The Border & End Govt. Shutdown
Image Source: Video Screen Shots. The Freedom Times Compilation
Posted By: Lindsey Michelle January 19, 2019
On Saturday, President Trump addressed the nation and outlined his plan to address border security concerns and put an end to the partial government shutdown. The plan includes the temporary extension of legal protections for DACA recipients and funding for the wall.
Several Senators and Representatives praised the President for his leadership and willingness to negotiate and compromise with Democrats on the issue of wall funding.
As the President addressed the nation on Saturday, he introduced the main points of his proposed plan and said, "This is a common-sense compromise that both parties should embrace."
$800 million in urgent humanitarian assistance
$805 million in new drug detection technology
2750 new border agents and law enforcement professionals
75 new immigration judge teams
$5.7 billion for "strategic deployment of physical barriers" (a wall)
3 years of legal relief and protection from deportation for DACA recipients
3-year extension of protections for immigrants under Temporary Protected Status (TPS)
President Trump also announced that he expected his proposed plan to be taken up in the Senate by Majority Leader Mitch McConnell sometime in the next week.
The Commander in Chief concluded in his speech, "Whatever we do, I can promise you this, I will never forget that my first duty and ultimate loyalty is to you, the American people. Any reforms we make to our immigration system will be designed to improve your lives, make your community safer, and make our nation more prosperous and secure for generations to come."
White House press secretary Sarah Huckabee Sanders wrote, "President @realDonaldTrump demonstrated courageous leadership offering a fair deal to address the security and humanitarian crisis at our border. Let's hope Democrats care more about doing what's right than resistance."
Senate Majority Leader, Mitch McConnell tweeted, "I commend @POTUS for his leadership in proposing this bold solution to reopen the government, secure the border, and take bipartisan steps toward addressing current immigration issues." (Continued Below)
House Republican Leader Kevin McCarthy said, "President Trump has put forward a serious and reasonable offer to reform parts of our broken immigration system and reopen government. The moment now turns to Nancy Pelosi and Chuck Schumer. The country is watching."
House Republican Whip Steve Scalise said, "The contrast between @realDonaldTrump & Democrats could not be clearer today: people can see he's serious about ending the shutdown & securing our border, while Democrats have yet to make a single legitimate counteroffer throughout the last month the government has been shut down."
GOP Chairwoman, Ronna McDaniel said, "@realDonaldTrump has shown strong leadership with this bold plan to re-open the government and fund border security. What are Democrats offering? Nothing. Their baseless opposition has gone on long enough." (Continued Below)
Sen. Lindsey Graham (R-SC) wrote, "A fantastic proposal Mr. President! Let's get it done!"
Rep. Matt Gaetz (R-FL) tweeted, "I agree with @realDonaldTrump & @marcorubio. The President has presented a win-win. We get our wall, Dems save the Dreamers. Together, we open government & move forward. The alternative is a terrible lose-lose for the nation. Common Dems, let's work together!"
Sen. Marco Rubio (R-FL) wrote, "@POTUS offers to support two bills sponsored by Dems in exchange for Border Security & the instant reaction from Dem leaders is No. Because denying him a win on border matters more to them than paying fed workers or 3 yrs of certainty for TPS & DACA recipients."
He added, "Next week Senate will take up bill that: – Extends TPS & DACA – Funds disaster relief – Funds humanitarian aid for migrants – Ends #Shutdown But Dem leaders reject it because includes some Border Security money. This is completely irrational obstructionism." Rubio continued, "Bill will also – allow migrants to apply for asylum in home country instead of having to make the dangerous journey to try & cross the U.S. border – extend for another 3 years protection for those under TPS from Haiti, Honduras & Nicaragua."
(Continued Below)
Rep. Mark Meadows (R-NC) said, "This is the latest and most significant step yet of POTUS showing his willingness to negotiate and compromise with Democrats on the issue of wall funding. At this point, if Democrats refuse to come to the table, it will show they are not at all serious about solving this impasse."
Sen. Marsha Blackburn (R-TN) wrote, "@realDonaldTrump is correct. The problems of open borders create problems that spread across the country, making every town a border town and every state a border state.
Sen. Rick Scott (R-FL) tweeted, "I applaud @realdonaldtrump for proposing a compromise that re-opens government, protects our borders and takes care of those brought to this country illegally through no fault of their own. Children should never be punished for the actions of their parents, and I've always supported protections for our DREAMers. Now it's time for Democrats to decide if they want to negotiate in good faith and accept a compromise that secures our borders and protects DREAMers or if they want to continue playing politics. I hope they choose the former so we can get to work fixing problems for the American people."
Former Gov. Mike Huckabee (R-AR) wrote, "We'll soon know if Pelosi, Chuck and the Dems really care about Dreamers and Gov't workers or not. @realDonaldTrump has offered MORE than they had even asked for. If they say "no," they reveal it's not about shutdown, but showdown. It's time to either poop or get off the pot!"
Sen. Mitt Romney (R-UT) said, "@POTUS has put forth a reasonable, good faith proposal that will reopen the government and help secure the border. I look forward to voting for it and will work to encourage my Republican and Democratic colleagues to do the same."
Rep. Lee Zeldin (R-NY) tweeted, "It's past time for @SpeakerPelosi to negotiate & compromise. She's admitted she won't even negotiate w/POTUS after the shutdown on increased $ to harden our border. Unacceptable. She doesn't have absolute power. No one does. Let's talk to each other & work it all out. No excuses!"
President Trump's Remarks on the Humanitarian Crisis at the southern border and the shutdown along with Senators and Representatives official tweets:
https://twitter.com/realDonaldTrump/status/1086736152059625473
https://twitter.com/PressSec/status/1086777967307603968
https://twitter.com/senatemajldr/status/1086736944137228288
https://twitter.com/GOPLeader/status/1086740274267729921
https://twitter.com/SteveScalise/status/1086737112056123394
https://twitter.com/LindseyGrahamSC/status/1086739205349408768
https://twitter.com/GOPChairwoman/status/1086738188062871552
https://twitter.com/mattgaetz/status/1086748424819654656
https://twitter.com/marcorubio/status/1086745404849483778
https://twitter.com/RepMarkMeadows/status/1086736389239201792
https://twitter.com/MarshaBlackburn/status/1086743434877124610
https://twitter.com/SenRickScott/status/1086759085356273665
https://twitter.com/GovMikeHuckabee/status/1086780676437889024
https://twitter.com/SenatorRomney/status/1086742109242179585
https://twitter.com/RepLeeZeldin/status/1086756357976588288
(Related Article: Key GOP Leaders React After Pres Trump Postpones Nancy Pelosi's Foreign Trip)
GOP, Politics, President Trump, Senate, Trump Administration
Sen. McConnell: "We All Know How It's Gonna End; There's No Chance The President's Going To Be Removed From Office"
Elections, Investigations, Politics, President Trump, Supreme Court
Sidney Powell Agrees That 'If One State Falls' In Pending SCOTUS Lawsuits 'They're All Going To Fall'
Entertainment, Hollywood, Immigration, National Security, Politics, President Trump
In A Fiery Statement, Actor Robert Davi Demands Schumer Ends The Wall Games: VOTE FOR THE WALL!
1st Amendment, Congress, Elections, GOP, Government, Health, Investigations, Law & Order, Military, Politics, President Trump, Science, Social Issues
Rep. Greene Discloses Details Of Phone Call With Her 'Favorite President Of All Time, Donald J. Trump'


Webb's Excavating focuses primarily on offering excavation services in commercial, residential and industrial sectors. We also do a wide a variety of jobs, from installation of underground utilities to land clearing, and from snow removal to landscaping. Regardless of the size of the project, we are committed to providing our customers with high quality work done on time and at competitive prices. We use high-quality equipment from brands such as Hitachi and New Holland.
Located in Olds, Webb's Excavating serves Central Alberta area. We can also provide our services to customers in Calgary and Edmonton. Contact us today to schedule a free estimate!
We use newer equipment and the latest technology, having the necessary knowledge and abilities to do so.
We see that projects follow safety standards and do not result in property damage or personal injury.
When you hire Webb's Excavating, you speed up your project, saving both time and money.