Privilege, beauty and evaluation

I’ve been for a beautiful walk this morning in the warm mist of a spring day in the highlands near Victoria. It was quiet but for the cacophony of bird song, and everything was wet with mist and dew. This is the greenest time of year on the west coast, and the mossy outcroppings and forest floor were verdant.

There is a beauty in what is, in any given moment.

I’ve been thinking about this as I have been struggling with watching people be evaluated in their work recently.  My daughter is a jazz musician, training her art in a university program where she is judged on her performance and where that number assigned to that moment in time affects much in her life.  My son laid out the papers he has been graded on, showing me a variety of marks that surprised him and made him proud of what he had accomplished. All of it a shallow judgement applied to a limited action in a tiny slice of time. Do these numbers take into consideration my daughter’s love of jazz or my son’s pride in the story he wrote or his ability to solve quadratic equations? Do they take into account how my kids approached this test, what it meant to them, what they were trying to do? How do these numbers track their changes, their growth, the affect that they are having on the world around them?

The evaluator’s job comes with enormous privilege.  The privilege is in determining the frame within which the noticing takes place. Poorly done evaluation happens when an evaluator reduces a complex outcome like “impact” into a few arbitrary indicators developed in isolation with a poorly articulated rationale and coherence with what is happening. When an evaluator walks into a process it is amazing how much gravity also enters the work.

At some point in our culture – and maybe it was always thus – evaluation became something of an investigation used to justify accountability pursued with a particular agenda in mind. Frameworks became both too narrow and too fuzzy. I have been in processes where evaluators wanted a single number on a scale from 1-5 to rate the effectiveness of an experience. And I have been in processes where evaluators are seeking to measure “impact” without every defining it, or only defining it on how a process has advanced their client’s singular needs and not the need of the whole ecosystem. I have never seen an evaluation that says to a client “these people are discovering some stuff that has nothing to do with what you funded them for, and therefore your assumptions about change are wrong.”

Done well however, evaluation contributes a tremendous amount of knowledge, awareness and confidence to a process. It allows us to make sense of our work, it opens our eyes to different questions we should be asking and it can put the tools of meaning making in the hands of people doing work. In complex environments, it can give us a new set of senses that help us see and hear and feel what is happening, and that open up promising new directions to nudge an effort.

When evaluation is part of the work it makes a huge difference. When evaluation is a separate project, laid on top of the work or done at a distance, it can bring the work to a standstill as everyone organizes around what the evaluator is looking for instead of where the project is at in its evolution or what the needs are.

Evaluations conducted with principles such as these ones are amazingly useful and empowering. They are deeply powerful influencers in the life of a project, and they need to be done with intense awareness of this power. We need to demand from our clients and funders and stakeholders, a more sophisticated standard of engagement around evaluation, and we need to hold evaluators to these principles too.

There is tremendous beauty in the moments of people working together, learning, creating, trying to improve the lives of others. Some days are rich with green and lush life and others are despairing failures. I would love to read an evaluation report that is as rich as Thoreau’s observations of life at Walden capturing the changes and the beauty, witnessing the growth all around, understanding its meaning and being open to the surprises that come with being immersed in an experience.

I’ll be writing more about this topic in the next little while. What are your longings for or experiences of great evaluation?