Why aren’t evaluators adapting their evaluations to the developmental context?

Overall, my study found that evaluators are less likely to be participatory—both in the overall evaluation process and in data collection methods—when the program beneficiaries are children than when they are adults. Why is this the case?

One possibility is that the evaluators in my study were not well-versed in working with youth. However, half of the evaluators were in the Youth Focused Evaluation TIG or the PreK-12 Educational Evaluation TIG, indicating they had some experience working with youth programs. Membership in these TIGs and self-reported developmental knowledge did not really relate to their evaluation practices.

Another possibility is that some other evaluation characteristic, such as their education level, their evaluation role (i.e., internal or external), or years of experience as an evaluator, could relate to their developmentally appropriate practice. Again, there were few differences between these evaluation characteristics in their evaluation practices.

Thus, the questions remain: which evaluators are more likely to have developmentally appropriate practice and what are the barriers to developmentally appropriate practice?

Some previous research suggests that experienced evaluators, even those experienced in working with youth, may need help in conducting developmentally appropriate evaluations. In a content analysis of youth program evaluations, Silvana Bialosiewicz (2013) found that few evaluations reported developmentally appropriate practices. That study was the impetus for the current study. A follow-up study involved interviewing youth evaluators and found many barriers to high quality youth program evaluation practice (Bialosiewicz, 2015). These barriers included cost and time needed and misconceptions from clients about good evaluation practice. Overall, this suggests that evaluators may need more training in developmentally appropriate practice or better resources for conducting developmentally appropriate youth program evaluations.

Next Steps

As with most research, I’m left with many more questions about developmentally appropriate evaluations than I was able to answer. I believe the results of the study suggest more need in examining youth participatory evaluation. However, I’m particularly interested in survey techniques with children and adolescents. I often see misunderstanding about survey methodology in general, and this is exacerbated when surveying children and adolescents. I am hoping to present at AEA 2017 on best practices in surveying children to help remedy this issue, but I also would like to further study this topic.

Developmental Appropriateness as Cultural Competence in Evaluation

Children and adults differ more than simply age; rather, they differ in culture as well.1 This recognition can be hard for evaluators: as we have all passed through childhood, it is easy to believe we have the same or greater knowledge of children’s culture than they do. Furthermore, our “spatial proximity to children may lead us to believe that we are closer to them than we really are—only different in that (adults claim) children are still growing up (‘developing’) and are often wrong (‘lack understanding’).”2

This points to a need for cultural competence, which the American Evaluation Association (AEA) describes as “critical for the profession and for the greater good of society.”3 Cultural competence practice in evaluation includes:

  • Acknowledging the complexity of cultural identity
  • Recognizing the dynamics of status and power (e.g., the differential power between adults and children)
  • Recognizing and eliminating bias in language
  • Employing culturally (i.e., developmentally) appropriate methods

In particular, culturally competent evaluations require inclusion of cultural expertise on the evaluation team.4 In the case of youth programs, this means inclusion of developmental expertise, which can involve developmental experts (i.e., psychologists, developmental scientists) but evaluators should also strive to include the youth themselves.

A youth participatory approach can reduce the harmful power imbalances between adult evaluators and youth participants,5, are more ethical for youth6, and offer many benefits for children and adolescents, including knowledge about the evaluation process and improvements in self-esteem, decision-making, and problem-solving skills.7

However, a youth participatory approach can vary by a range of levels.8 At the lowest level, participants are simply included as a data source, which can yet more vary by direct (i.e., surveys, interviews) and indirect (i.e., observations, archival data) data collection. Further up the participatory ladder is giving youth input on the evaluation process. The highest level of youth participation is youth actually leading the evaluation, much like they would in a traditional empowerment evaluation.

Inclusion of youth, or at least adult developmental experts, can improve the likelihood of a culturally competent evaluation for the first two bullet points mentioned above. However, evaluators still must make sure the evaluation design and methods are culturally, and therefore developmentally, appropriate. The next post will discuss how evaluators can promote cultural competence across the evaluation process in the context of youth programs.