Evaluation is Not Applied Research

What is the difference between evaluation and research, especially applied research? For some, they are one and the same. Evaluation and research use the same methods, write the same types of reports, and come to the same conclusions. Evaluation is often described as applied research. For instance, here are some recent quotes describing what evaluation is: “Evaluation is applied research that aims to assess the worth of a service.” (Barker, Pistrang, & Elliott, 2016). “Program evaluation is applied research that asks practical questions and is performed in real-life situations.” (Hackbarth & Gall, 2005), and the current editor of the American Journal of Evaluation saying that “evaluation is applied research.” (Rallis, 2014). This is confusing for introductory evaluation students, particularly those coming from a research background or studying evaluation at a research institution.

Others claim the distinction between evaluation and (applied) research is too hard to define. I do not disagree with this point. The boundaries between evaluation and research are fuzzy in many regards. Take, for instance, evaluation methodology. Our designs and methods are largely derived from social science methodology. However, as Mathison (2008) notes in her article on the distinctions between evaluation and research, evaluation has gone much further in the types of designs and methods it uses such as significant change technique, photovoice, cluster evaluation, evaluability assessment, and success case method. Scriven and Davidson have begun discussing evaluation-specific methodology (i.e., the methods distinct to evaluation), including needs and values assessment, merit determination methods (e.g., rubrics), importance weighting methodologies, evaluative synthesis methodologies, and value-for-money analysis (Davidson, 2013). These methods show that, while we indeed incorporate social science methodology, we are more than that and have unique methods beyond that.

This is no better illustrated than by the hourglass analogy provided by John LaVelle. The differences between research and evaluation are clear at the beginning and end of each process, but when it comes to the middle (methods and analysis), they are quite similar. Thus, evaluation differs from research in a multitude of ways. The following table should be interpreted with a word of caution. The table suggests clear delineations between research and evaluation, but as Mathison notes, many of the distinctions offered (e.g., evaluation particularizes while research generalizes) are not “singularly true for either evaluation or research.” (p. 189, 2008).

Area of difference Research Evaluation
Purpose Seek to generate new knowledge to inform the research base Seek to generate knowledge for a particular program or client
Who decides Researchers Stakeholders
What questions are asked Researchers formulate their own hypotheses Evaluators answer questions that the program is concerned with
Value judgments Research is value neutral Evaluators provide a value judgment
Action setting Basic research takes place in controlled environments Evaluation takes place in an action setting where few things can be controlled
Utility Research emphasizes “production of knowledge and leaves its use to the natural processes of dissemination and application.” (Weiss, 1997) Evaluation is concerned with use from the beginning
Publication Basic research is published in journals Evaluation is rarely published, typically only stakeholders can view the reports

I want to conclude by saying that if we are to call ourselves a transdscipline or an alpha discipline, like Scriven would argue we are, then we should work hard to differentiate ourselves from other disciplines, particularly basic and applied research. This may be difficult, particularly between applied research and evaluation, but we need to make these differences as explicit as possible, partly to help incoming evaluators in the field understand the differences (see EvalTalk for this repetitive question since 1998; Mathison, 2008) and partly to separate ourselves from research (and research from evaluation).

How Can Evaluation Avoid Lemons?

I recently stumbled across a blog post by Dr. Simine Vazire, an associate professor in psychology at UC Davis, which discussed an economics article by Akerlof, “The market for “lemons”: Quality uncertainty and the market mechanism.” Here’s what he wrote:

In this article, Akerlof employs the used car market to illustrate how a lack of transparency (which he calls “information asymmetry”) destroys markets.  when a seller knows a lot more about a product than buyers do, there is little incentive for the seller to sell good products, because she can pass off shoddy products as good ones, and buyers can’t tell the difference.  the buyer eventually figures out that he can’t tell the difference between good and bad products (“quality uncertainty”), but that the average product is shoddy (because the cars fall apart soon after they’re sold). therefore, buyers come to lose trust in the entire market, refuse to buy any products, and the market falls apart. (Vazire, 2017, “looking under the hood”)

This is much similar to the replication crisis in psychology, and I worry that evaluation may come to many of these same issues. This worry was also expressed by Scriven (2015).1 He writes, “Also depressing was the discovery that the great classic disciplines, although they thought they had a quality control system, in fact the procedure that everyone immediately put forward as performing that function–peer review–turned out to have been hardly ever studied for simple but essential virtues like reliability and validity…” (p. 18).

What peer review system does evaluation have? Scriven put forth meta-evaluation, but the practice is rarely, if ever, done. Scriven says:

This is because our real world practice is largely in the role of consultant, and consultants’ work does not normally undergo peer review. We need to tighten up the trashy way peer review is done in other disciplines and use serious meta-evaluation to fill the gap in our own emerging discipline with respect to that job that we say (and can prove) taht peer review ought to be done in the other disciplines. (p. 19)

Scriven goes on to argue that evaluation has a duty to study how evaluation is conducted in other disciplines, leading evaluation to be the “alpha discipline.” But before we can consider evaluation as the alpha discipline, we have to do a “full analysis of the pragmatics” of evaluation, meaning we need to more clearly define evaluation so that evaluation is considered a key methodology of social science

“that must be mastered in order to do all applied (and some pure) social science. In that way, good evaluation research designs will be the exemplar for much of social science, instead of social science treating personnel or program evaluation as something they can do with their current resources, albeit conceding that there are some specialists in these sub-areas.” (p. 20)

Scriven’s solution of serious meta-evaluation done publicly aligns with the solution promoted by Akerlof: transparency. I further argue that there need to be more serious regulations applied to ensure this transparency, and one way to do this is through professionalization. Unfortunately, professionalization within the American Evaluation Association has met with serious resistance, but work by our colleagues up north (the Canadian Evaluation Society) and work on standards and competencies within AEA are steps forward. I think professionalization will help evaluators more clearly define who evaluators are and what evaluation is so that the field can move forward as the alpha discipline Scriven describes.