I’ve been for a beautiful walk this morning in the warm mist of a spring day in the highlands near Victoria. It was quiet but for the cacophony of bird song, and everything was wet with mist and dew. This is the greenest time of year on the west coast, and the mossy outcroppings and forest floor were verdant.
There is a beauty in what is, in any given moment.
I’ve been thinking about this as I have been struggling with watching people be evaluated in their work recently. My daughter is a jazz musician, training her art in a university program where she is judged on her performance and where that number assigned to that moment in time affects much in her life. My son laid out the papers he has been graded on, showing me a variety of marks that surprised him and made him proud of what he had accomplished. All of it a shallow judgement applied to a limited action in a tiny slice of time. Do these numbers take into consideration my daughter’s love of jazz or my son’s pride in the story he wrote or his ability to solve quadratic equations? Do they take into account how my kids approached this test, what it meant to them, what they were trying to do? How do these numbers track their changes, their growth, the affect that they are having on the world around them?
The evaluator’s job comes with enormous privilege. The privilege is in determining the frame within which the noticing takes place. Poorly done evaluation happens when an evaluator reduces a complex outcome like “impact” into a few arbitrary indicators developed in isolation with a poorly articulated rationale and coherence with what is happening. When an evaluator walks into a process it is amazing how much gravity also enters the work.
At some point in our culture – and maybe it was always thus – evaluation became something of an investigation used to justify accountability pursued with a particular agenda in mind. Frameworks became both too narrow and too fuzzy. I have been in processes where evaluators wanted a single number on a scale from 1-5 to rate the effectiveness of an experience. And I have been in processes where evaluators are seeking to measure “impact” without every defining it, or only defining it on how a process has advanced their client’s singular needs and not the need of the whole ecosystem. I have never seen an evaluation that says to a client “these people are discovering some stuff that has nothing to do with what you funded them for, and therefore your assumptions about change are wrong.”
Done well however, evaluation contributes a tremendous amount of knowledge, awareness and confidence to a process. It allows us to make sense of our work, it opens our eyes to different questions we should be asking and it can put the tools of meaning making in the hands of people doing work. In complex environments, it can give us a new set of senses that help us see and hear and feel what is happening, and that open up promising new directions to nudge an effort.
When evaluation is part of the work it makes a huge difference. When evaluation is a separate project, laid on top of the work or done at a distance, it can bring the work to a standstill as everyone organizes around what the evaluator is looking for instead of where the project is at in its evolution or what the needs are.
Evaluations conducted with principles such as these ones are amazingly useful and empowering. They are deeply powerful influencers in the life of a project, and they need to be done with intense awareness of this power. We need to demand from our clients and funders and stakeholders, a more sophisticated standard of engagement around evaluation, and we need to hold evaluators to these principles too.
There is tremendous beauty in the moments of people working together, learning, creating, trying to improve the lives of others. Some days are rich with green and lush life and others are despairing failures. I would love to read an evaluation report that is as rich as Thoreau’s observations of life at Walden capturing the changes and the beauty, witnessing the growth all around, understanding its meaning and being open to the surprises that come with being immersed in an experience.
I’ll be writing more about this topic in the next little while. What are your longings for or experiences of great evaluation?
Last weekend I took a ramble across Bowen Island, where I live, with a friend and colleague, Annemarie Travers. Annemarie and I have been teaching the Leadership 2020 program for a number of years now and we both love walking: she on the long pilgrimages of the Camino and Shikoku and me in the mountains of southern British Columbia. We are also both interested in managing in complexity.
My friends over at the Social Labs Revolution website have been fielding questions about the prototyping phase of labwork and today published a nice compilation of prototyping resources. It’s worth a visit. It got me thinking this morning about some of the tools I use for planning these days.
As Bronagh Gallagher and I have been musing about our offering on complexity, facilitation and social justice, we have been discussing the shift in activism from ideology to evolutionary. Ideological movements try to coalesce activities and people along a line towards a fixed end state. Evolutionary movements start with intentions, principles and move outward in multiple directions along vectors. They adjust and learn as they go, and they both respond to and change their context.
This nice post from Network Centered Advocacy capgtues what I’m talking about by first looking at how a lacrosse player’s artistry evolves in changing contexts and then concludes with these important paragraphs:
Being labeled a “movement” is a reflection of evolutionary status. One person or organization does not qualify as a movement, yet there is no set size of a movement. Movements are messy, complex and organic. The movement label is shorthand, an inclusive term of many independent leaders and supporters, their support structures, all that they can tap into, as well as their capacity to disagree as often as they align on work.
Movements are a reflection of self-directed, adaptive, resilient, self-sacrificing, supported and persistent initiatives to work on complex problems. There are no movement structures, but instead a movement is a mass migration of people, organizations, businesses and communities unified in common story, driving to shift culture, policy, behavior and norms. Successful movements build and transform the landscape as they progress providing a base for further progress. A quick scan of the first few pages of google news for” movements” produces a snapshot of the current movements that come to mind, including the movement against fracking, the climate change movement, the tea party movement, Occupy, #blacklivesmatter, the anti-austerity movement, the dump-Trump movement, the maker-movement, the LGBTQ movement–the list goes on.
A key evolution point in a movement’s trajectory is the transition away from any single point of failure, to be loosely structured and resilient enough to absorb setbacks. The agility and adaptive characteristics of movements are fueled not only by personal stakes, individualism, driven leadership, passion and local control, but also by unpredictable solidarity and a distributed organizing approach that resists centralization. The difference between an organization, coalition, centralized campaign and a genuine movement is the way each fuels smart local initiatives and the ways leaders align power.
Building a movement is actually more aptly perceived as unleashing a movement, creating new spaces that help the movement surge in wider, expansive and still supportive directions. As a movement gains organizing momentum, strategies shift to broadly unfold and push a wide set of actions that draw opposition thin rather than clustering and making defense easy. This distributed layout requires a shift in thinking and strategy.
The key thing to notice here is that culture is changed by evolving movements, not linear programs. Movements are not led TOWARDS a goal, but rather emanate from a set of connected and coherent stories, actions and intentions, and self-correct, fail and adapt as they go. This is true whether the venue of action is organizational or societal. Cultures are complex and require complexity to change them. Diving more into the examples given in the quote will give you more insight into how these movements have become a part of, and transformative agents within, the cultures they are aiming to change.
Sometimes people see that I’m a dialogue practitioner and the assume that I am not a fan of quantitative measurement. I think this has to do with the fact that the dialogue practitioner community has been a kind of antithesis to the “measure and manage” world of empirical scientific management.
In any endeavour both qualitative and quantitative measurements are important. The issue isn’t whether or not numbers are to be more trusted than meaning making; the issue is whether we are measuring thing properly.
The issue is whether or not we use measurements as targets or gauges.
Again, this is helpful in understanding the distinction between summative and developmental evaluation and sensemaking. In a linear system, you are aiming for certain end states and targets. In a complex and non-linear system you are aiming to keep to vectors. So using technology to increase production by 5% and decrease expense by 15% can be achieved and you can look back and see how well you achieved that target. You can also do tests and host conversations with workers and customers to discuss the quality of your product, aiming for a general score of “happy” which in turn might be reflected in numbers like sales, returns, recommendations and so on.
In a complex system, lilke an organization’s culture however, you are not managing for a target, but rather you are managing a kind of balance and a direction. You get to choose that direction from your own moral and ethical sense of what is right to do. For example, maintaining an organizational culture of openness, respect, creativity and support requires monitoring your culture in real time, a lot, and noticing how things are shifting and changing. Dialogic methods play an important role here, especially in perceiving patterns and making decisions about what to do, as well as engaging people in the endless negotiation about what those values look like on a daily basis. As a management tool, developing skillful dialogue tools allow you to manage the day to day issues with departures from your preferred set of values, beliefs or practices. Being complex, things like organizational cultures won’t always act they way you want them too, and so good leaders do two things well: they help resolve the inevitable violations of standards and practices in a manner that reflects the preferred way, and they gather together people over time to discuss what everyone is learning about the way the culture is working.
It’s not good enough to convene an annual meeting about the organization’s values and culture. That simply gives you a snapshot in time and tells you nothing about how an organization is evolving and changing, nor does it provide information about promising practices. To monitor over time, you can use a tool like CultureScan or a series of other regular ways of documenting the small observations of daily life that together help provide a picture of what the organization is doing.