Can evaluators remain unbiased?

Bias is the prejudice in favor of something, usually in a way considered unfair.

People have been talking about bias in evaluation—and research—since the beginning. It is the idea that if a person is favorable towards a program or perhaps wants to receive future contracts from that program, they are more likely to adjust their evaluations in a way that can lead to findings that are invalid, unreliable, and lack credibility.

Understandably, some people have thought this level of bias should be avoided at all costs, and propose maximizing the distance between the evaluator and the program to achieve this goal. These methods to maximize the distance include goal-free evaluation, non-participatory evaluations (e.g., goal-free evaluation) independent funding (rather than the program funding the evaluation), designs that minimize threats to internal validity (a la Campbell, including randomized control trials), and peer reviews and meta-evaluation.

However, some of these approaches to evaluation—where the program and its stakeholders should be minimized to avoid contaminating the evaluator—alienates many evaluators, particularly those who report being in an internal role. Scriven and Campbell, some of the major proponents of minimizing bias, may not have the pragmatic or constructive epistemologies that many evaluators have. Thus, Scriven and Campbell argue for controlling for bias rather than acknowledging and recognizing bias like a constructivist would argue for.

Furthermore, this approach to evaluation alienates our stakeholders. Proponents of collaborative, empowerment, and participatory evaluations argue, and have found much evidence for, the benefits of such an approach for stakeholders, the program, and the evaluation. These benefits include giving participants ownership over the evaluation and results; building capacity to understand, use, and conduct evaluations; and improving program performance, learning, and growth.

Minimizing or controlling for bias may alienate evaluators and stakeholders who want a more participatory approach. 

There is a level of bias that some seem to consider the worst type of bias: advocacy. As soon as evaluators put on their advocacy hat, they are no longer value neutral toward the program. Rakesh Mohan wrote a wonderful piece on Sheila Robinson’s website titled “Why are evaluators so tentative about the advocacy aspect of our profession?[1] In it, he argues that “it is the fear of politics that makes many evaluators tentative about advocacy.”

He further argues in his related AJE article[2] that advocacy while maintaining independence is a difficulty and risky endeavor. Credibility is important in our profession, particularly outside of Canada without any sort of credential or professionalization system. As such, “the loss of credibility could adversely affect the evaluator’s professional reputation among peers and could negatively affect his or her livelihood. It is the fear of losing one’s credibility that keeps many evaluators away from engaging in advocacy activities.” (Mohan, 2014, p. 400).

Many are fearful of losing credibility if they are viewed as biased in any regard.

I think this loss of credibility—in the eyes of peer evaluators, stakeholders, and the outside community—is what most people think of when they think of, and fear for, bias. And I am not saying it is wrong to be fearful of it or wrong to avoid it necessarily. However, I think we need to balance credibility with participatory (or empowerment or collaborative) evaluations whereby we advocate for our programs when it is ethical to do so. We are often working with impoverished or disenfranchised communities or programs that have immense political implications. Through advocacy—or even just by maintaining closeness to these communities in our evaluation—we can help raise their voices in this highly political world.

[1] Rakesh’s blog and article are both focused primarily on advocating for evaluation, not necessarily for the programs. However, I feel his arguments are relevant to both cases.

[2] Mohan, R. (2014). Evaluator advocacy: It is all in a day’s work. American Journal of Evaluation, 35(3), 397-403.

Tips for Implementing the GTD System in your Workflow

These are some tips that I use to maintain my productivity in my workflow.

1. Experiment

I think the only reason I do my GTD system well is because I am constantly trying new things.

This is something I learned from Sam Spurlin through his blog posts and podcasts. He would always give himself a little experiment: go purely analog for a month, use all Apple native software, etc.

I very much do the same thing, especially when there’s something about my current workflow that I don’t like. Getting too many newsletters? Google how to reduce newsletters and clutter in your email. I eventually settled on SaneBox. Don’t like your to-do list workflow? I tried an analog agenda and multiple apps before I eventually settled on Todoist, which I’ve used for a few years now.

This also means that what works for me may not work for you. I found my complete opposite in Dr. Raul Pacheco-Vega. I am a purely digital person (okay, I’ve started bringing a small moleskin to take notes in, but I always transfer it to my computer) and he’s a purely analog person (he has this awesome thing called the Everything Notebook that you should check out).

2. Create a system you can take everywhere…

This is one reason why I am a purely digital person: I go nowhere without my phone. It is my brain for dealing with tasks. Need to remember something? Use the quick-add widget and 20 seconds later it’s written down for me to remember later.

If you are more analog, the Everything Notebook mentioned previously would be helpful. I can’t imagine lugging that thing around all the time, though, so perhaps have a small moleskin notebook that could fit in your pocket.

3. … and capture everything

Do. Not. Rely. On. Your. Memory.

Was that clear enough? You will forget something if you try to rely on memory. As soon as a task comes to mind, you are to write it down. Period. This is the only way nothing will slip through the cracks!

4. Chunk it out!

If you find you are procrastinating on an important task, constantly postponing the day you will do it, then chances are the task is too big and daunting. “Work on thesis” is an example of a very poorly written task. I sometimes write these tasks down because it’s something that pops up in my memory that I need to do, but later I need to sit down and chunk it out. By “work on my thesis,” I really mean I need to fix the descriptive statistics in the second paragraph of the results section, expand the discussion section to add in literature discussed in the introduction, and write my limitations section. Those are easily manageable tasks. They have the added benefit of taking a short amount of time, so I could do it in between meetings if I have a 15-minute time period.

Want some more tips? I found this article on the Todoist blog to be absolutely wonderful.

Managing life using the Getting Things Done system

If you want to learn how to maintain productivity, then you need to read the book Getting Things Done by David Allen. This book is seriously a life-changer. I have used this system for the past few years and while I don’t adhere to his system 100%, the majority of his principles are extremely beneficial.

The Getting Things Done system has five steps: capture/collect, process, organize, review, and do. I’m going to discuss each step in turn.

1. Capture/collect

First, you need a place to capture things that comes your way. I have two places where I capture everything: my email and my to-do list. You need a system that can capture things wherever you are. This is why I use a to-do list app (shout out to Todoist) because it is available on all operating systems and is connected to so many other types of software, including Gmail and Slack. If you deal with a lot of papers and other analog data, then it’s recommended you have a physical inbox as well.

This system needs to be able to capture everything for you. This includes emails, tasks, articles, questions that pop up in your mind, bills, and notes. Your brain is not a suitable place to collect these things! Your brain is fallible and you will forget something. Don’t be that person!

2 and 3. Process and Organize

Next, you need to figure out what to do with everything that you collect. When you decide to process tasks (i.e., go through your task list, check your email, clear out your paper inbox), then for each item you go through the following steps. If the task isn’t actionable, meaning you can’t do something with it, then either chuck it, archive it (e.g., save the email into another folder or save the document into it’s appropriate folder), or mark it as a “someday” task. I like to have a “someday” project in my to-do list where I save thoughts and ideas for future projects (like my thesis and dissertation!).

If the project does have an action, then you have one of three options. If the task takes less than two minutes, then do it. I find that the two-minute rule most often applies to emails. When I’m checking emails, if I can respond to that email within two-minutes then I do it right then and there. If I can’t get it done in two minutes, can someone else do it for me? Sometimes someone asks me a question that I do not know the answer to, in which case I forward the email to the person who does know the answer. In all other cases, defer the action. Either add the event to your calendar or add the task to your to-do list.

4. Review

I think this is the area where most people fail the Getting Things Done system. They store their information away and then promptly forget about it because they are not engaging with their system enough. To properly review, you need to get clear, get current, and get creative. To get clear, make sure your system is cleared and ready to go. This means to gather all loose things that need to be collected and process them all. To get current, review your calendar, to-do list, waiting-for list, project lists, and any other checklists you might have. Prioritize tasks for the upcoming week alongside your calendar (i.e., don’t place to write for three hours on days that you have back-to-back meetings from 9-5). Lastly, to get creative, review your someday/maybe lists and see what projects you might be able to start.

I personally do each review chunk in different timelines. I get clear at the end of each day or at the beginning of a long work session. This frees up my mind to really focus on the tasks at hand. I get current on a weekly basis. On Sundays, I set down with my calendar and to-do list and make sure my days are evenly split and I’ve properly prioritized my tasks. I get creative monthly. This means checking my someday/maybe pile and checking the progress on my projects that are low on the totem pole but where my inspiration resides. When the summer hits, I’ll be able to get creative a little more frequently (yay!).

5. Do!

At this point, you should be able to engage with your task list and get things done! With your mind free from all the things you are trying to remember, your inbox down to zero, and your calendar and task list organized and ready to go, you can now focus on the important work at hand. You might need additional work in how to prioritize tasks or break down tasks into manageable, bite-sized chunks, so we’ll cover that topic next!