of studies would be down-weighted. Perhaps only evidence from “independent would be accepted. evaluations” Definition Requiring an is only meaningful if that term is defined “independent evaluation” and in some way limits how a study is run. In part, this corresponds to the of the “rules” accounting profession. Such should be crafted—as much as possible—so that “rules” evaluators would serve policy makers rather than the agency being evaluated. To consider how to do that, we return to the four strategies identified in the accounting literature--professionalism, reputation, liability, and owner control—but in a different order. For each strategy, we consider to what extent they are currently applicable and reforms that might make them even more effective. Oversight “Owner” Current evaluation practice has only The minimal “owner oversight.” “evaluation of the agency being evaluated usually runs the evaluation. That shop chooses the shop” evaluator—sometimes putting current or past staff of the operating agency on the proposal review committee—and reviews and approves the plans. It controls evaluator’s the flow of information—what and when—to the Technical Working Group (TWG). That shop reviews and approves the report and any public discussion of evaluation evaluator’s results—orally at conferences, and in writing in journal articles and position papers. As a consequence, simple refusal to release project reports is the most common form of agency interference. Personal experience and discussion with other evaluators suggests that the threat of refusal to release will often induce an evaluator to accede to all but the most egregious requests/demands.
of studies would be down-weighted. Perhaps only evidence from “independent evaluations” would be accepted. Definition Requiring an “independent evaluation” is only meaningful if that term is defined and in some way limits how a study is run. In part, this corresponds to the “rules” of the accounting profession. Such “rules” should be crafted—as much as possible—so that evaluators would serve policy makers rather than the agency being evaluated. To consider how to do that, we return to the four strategies identified in the accounting literature--professionalism, reputation, liability, and owner control—but in a different order. For each strategy, we consider to what extent they are currently applicable and reforms that might make them even more effective. “Owner” Oversight Current evaluation practice has only minimal “owner oversight.” The “evaluation shop” of the agency being evaluated usually runs the evaluation. That shop chooses the evaluator—sometimes putting current or past staff of the operating agency on the proposal review committee—and reviews and approves the evaluator’s plans. It controls the flow of information—what and when—to the Technical Working Group (TWG). That shop reviews and approves the evaluator’s report and any public discussion of evaluation results—orally at conferences, and in writing in journal articles and position papers. As a consequence, simple refusal to release project reports is the most common form of agency interference. Personal experience and discussion with other evaluators suggests that the threat of refusal to release will often induce an evaluator to accede to all but the most egregious requests/demands.
AB-JAK_20110531-jn.docx