Quotes: food for thought on evalaution

Purposes of evaluation

"The evaluation of public policies, programs, and practices seems to be an intrinsic part of democratic government for four reasons. It reports information about government performance that the public needs to know. It adds new data to the existing stock of knowledge required for government action. It develops and analytical capability within agencies that moves them away from territoriality and toward a culture of learning. And, more generally, its spirit of skepticism and willingness to embrace dissent help keep the government honest."
Eleanor Chelimsky
“The Purposes of Evaluation in a Democratic society”Chapter 1 in The Sage Handbook of Evaluation Ed by Ian Shaw, Jennifer Greene and Melvin Mark p33

Evaluation in modern society

"Modern society is defined by its attempt to replace tradition and prejudice by a belief in rationality and progress...Evaluation emerges at a historical point in time when modern society becomes reflective, as it acknowledges that not all of its attempts to shape its own future are automatically successful, and some have side effects."
Peter Dahler-Larsen
“Evaluation after disenchantment? Five issues shaping the role of evaluation in society.” Chapter 6 in The Sage Handbook of Evaluation Ed by Ian Shaw, Jennifer Greene and Melvin Mark p143

Intellectual contexts of evalaution

"Evaluation is a human enterprise, and in an important sense the intellectual contexts from which evaluation has emerged are embodied in individual theorist-practitioners. Thus, early figures in the field brought that own intellectual traditions with them as they engaged in research, writing, and teaching that constructed evaluation as a field."
John Stevenson and David Thomas
“Intellectual contexts” Chapter 9 in The Sage Handbook of Evaluation Ed by Ian Shaw, Jennifer Greene and Melvin Mark p 201

Epistemological challenges

Epistemology in its simplest terms is concerned with the nature of knowledge. When researchers and evaluators ask themselves “How do I know?” or “How certain am I about what has happened here?” or “Under what conditions might others find my findings inadequate?” they are asking epistemological questions about the attributes, scope, and sources of their knowledge. They are examining the manner in which they have gathered information, constructed and manipulated concepts, and used their thoughts, memories, sense, and emotions to analyse and justify what they are presenting as “knowledge about.” Evaluation like social science research is now rooted in a growing diversity of epistemologies.
J Bradley Cousins and Lyn M. Shulha
“ A comparative analysis of evaluation utilization and its cognate fields of inquiry: Current issues and trends”. Chapter 12 in The Sage Handbook of Evaluation Edited by Ian Shaw, Jennifer Greene and Melvin Mark p 266

Evidence based practice

"According to the evidence-based agenda, the prevailing view is that randomized experiments are required in order to generate an adequate body of scientific knowledge to inform policy and practice. However, as argued above, this ‘gold standard’ does not necessarily provide an adequate basis for professional practice in all contexts."
Alan Clarke
Evidence-based evaluation in different professional domains: Similarities, differences and challenges. Chapter 25 in The Sage Handbook of Evaluation Ed by Ian Shaw, Jennifer Greene and Melvin Mark p 577

Evidence based policy

"How does ‘evidence’ speak to ‘power’? What do you get if you cross ‘research’ with ‘realpolitic’? Where lies the ground between the ‘ivory tower’ and ‘corridors of power’? What hope is there for nuptials between ‘knowing’ and ‘doing’? The anser to all these questions has coalesced in a new millennium big idea called evidence-based policy, and, in the pages to follow, I am going to attempt to assess the state of the union between the realms of evidence and policy."
Ray Pawson
Evidence-based Policy A Realist Perspective Sage Publications p1

Cause and effect: the causation issue

No discussion of evaluation nuts and bolts is complete without some mention of the causation issue. Although this is a relatively simple concept to grasp in everyday life, causation is one of the most difficult and one of the most important issues in evaluation. Even if we observe changes that are consistent with the expectations or goals of a program or other evaluand, we cannot refer to these as “impacts” or “outcomes” unless we can demonstrate that the evaluand [program] was at least a primary cause of those changes....Although the causation issue is incredibly important, demonstrating causal links can seem like an impossible task, especially for evaluators with limited time and resources...Here is the good news: There is some practical light at the end of the causation tunnel, and the tunnel is not nearly as long and treacherous as legend has it.
E. Jane Davidson
Evaluation Methodology Basics The Nuts and bolts of Sound Evaluation Sage 2005 p 67-68

Certainty

'Suppose I say “I know for certain that I am standing up; it is absolutely certain that I am; there is not the smallest chance that I am not.” Many philosophers would say: “You are wrong: you do not know that you are standing up; it is not absolutely certain that you are; there is some chance, though perhaps only a very small one, that you are not.”'
G. E. Moore
Philosophical Papers (New York: Collier Books, 1962 p 240)

Politics and evaluation

"Politics and evaluation are related in a number of ways. On is that evaluations, if they are used at all, become part of a political decision process surrounding the program being evaluated. Evaluations consequently become participants in the political arena along with other political actors. A second way that politics and evaluation are related is that evaluations, by taking a position about how well a program is doing, are inherently and unavoidably political."
Dennis Palumbo
Politics and Evaluation. Chapter 1 in The Politics of Program Evaluation Volume 15 Sage Yearbooks in Politics and Public Policy, Ed Dennis J Palumbo 1987 p12

Truth in evaluation

"One of the major points of departure among evaluators as they approach their work is grounded in how they view the notion of “truth” in evaluation. I take a fairly practical position on this that goes roughly as follows. Evaluation findings are “demonstrably true” when a solid mix of evidence supports a conclusion at or above the level of certainty required in that decision-making context. Does that mean that I think something has been “proven’ to be a 100% level of certainty so that it can be called an absolute truth in the strictest sense of the word? Certainly not."
E. Jane Davidson
Evaluation Methodology Basics The Nuts and bolts of Sound Evaluation Sage 2005 p xv

Utilization-focused evaluation

"...the focus in utilization-focused evaluation is on intended use by intended users. Since no evaluation can be value-free, utilization-focused evaluation answers the question of whose values will frame the evaluation by working with clearly identified, primary intended users who have the responsibility to apply evaluation findings and implement recommendations. Utilization-focused evaluation is highly person and situational. The evaluation facilitator develops a working relationship with intended users to help them determine what kind of evaluation they need...Utilization evaluation does not advocate any particular evaluation content, model or method...."
Michael Quinn Patton
‘The Roots of Utilization Focused Evaluation’ Chapter 18 in Evaluation Roots Tracing Theorists’s Views and Influences edited by MArvin G. Alkin Sage 2004 p 227

Scriven - The Science of Valuing

"Most evaluators acknowledge that evaluation assigns merit or worth, but they rarely discuss values more deeply. Scriven is the most forceful and persistent advocate of this position. He wants evaluation to be the science of valuing: “Bad is bad and good is good and it is the job of evaluators to decide which is which”. He criticizes other evaluators for “defining evaluation, as for example, the provision of information to decision-makers. Evaluation is what it is, the determination of merit or worth, and what is it is used for is another matter.”"
William Shadish, Thomas Cook, Laura Leviton
Foundations of Program Evaluation Theories of Practice William R. Shadish, Jr Thomas D Cook Laura C Leviton Sage 1991 pp 74-75

Early Weiss on evaluation

"Weiss summarised her initial theory in two highly influential books. She started with the premised that “the purpose of evaluation research is to measure the effects of a program against the goals it set out to accomplish as a means of contributing to subsequent decision making about the program and improving future programming.” Her early theory stressed experimental methodology to assessment program goal achievement, since “systematic evidence on the effects of each project would show which programs worked well and should be expanded , which ones were failures and needed to be abandoned and which were marginal and needed to be modified"”
William Shadish, Thomas Cook, Laura Leviton
Foundations of Program Evaluation Theories of Practice William R. Shadish, Jr Thomas D Cook Laura C Leviton Sage 1991 p 181

Later Weiss: Improving social programs

"In view of all these difficulties, Weiss concluded that the traditional view of “evaluation itself is problematic...Its decline is not the worst of the alternative futures we can imagine”. She began to consider alternatives to this view: For the social scientist who wants to contribute to the improvement of social programming, there may be more effective routes at this point than through evaluation research. There may be greater potential in doing research on the processes that give rise to social problems, the institutional structures that contribute to their origin and persistence, the social arrangements that overwhelm efforts to eradicate them, and the points at which they are vulnerable to societal intervention."
William Shadish, Thomas Cook, Laura Leviton
Foundations of Program Evaluation Theories of Practice William R. Shadish, Jr Thomas D Cook Laura C Leviton Sage 1991 p 190