Putting VALUE in Evaluation

The following is a summary of key learnings and resources from a session held as part of the ArtsEngage Learning Community in mid-October, 2020. The focus of this session was Evaluation.

Evaluation often gets a bad name. We associate evaluation with counting heads and filling out reports. We want to change that thinking!

So we started our session with establishing a shared understanding of what evaluation can be. A report is not an evaluation: it’s the summation of an evaluative process. A report is right at the end of the project; evaluation starts at the conception stage and should be woven throughout the life of the project. Thinking about evaluation right from the start not only allows you to better measure the impact of your project, but can help you to clarify your values and goals, and therefore shape the project itself. 

What matters?

We first prompted participants to “pitch” their project to each other starting with “I believe this project matters to my community because…”. This was a quick warm-up to start thinking in terms of value, what matters, what drives us to act, what hope for change we continue to believe in.

“I am really pleased to look at the value beyond numbers and monetary value. I like going deeper for true feedback from the participants.”

We then invited everyone to go one step further in their processing by focusing on the values they expressed or heard in this exchange. What stood out in your conversation? What values are underpinning the projects you talked and heard about?

“I like this idea of measurement as ‘opening’ and the project as a ‘continuum’ - what does it open up for the future?”

During this session, we collected the following “project intentions”:

• Keeping the arts alive in our communities
• Safety
• Encouragement
• Relevance
• Validating and dramatizing children’s stories from this time
• Connection; not just responding to but creating, the “new normal”
• Sharing experience with broader community
• Participants are artists already and they have the tools to create in them now
• Invention
• Art as an instrument of change
• Value of public space
• Bringing the community together through the expression of dance, music, rhythm and creativity.
• Inclusivity: showing the town as an inclusive place
• Bridging historical divides
• Connection
• Listening" allowing artists to encourage difficult conversations that the municipality WANTS to have
• Reconciliation: bringing marginalized sectors of the community together to communicate and collaborate.
• Opening the participants to the fact that you can do whatever you put your mind to: self confidence, self esteem, listening, connecting both sides of the brain
• Exploring the entry point of Art to the community: how do we get in there?

During our first Meet and Greet, participants offered one word on what community means to them.

Having values and intentions established right at the start of a project provides a lens to observe changes in this particular area. If we know what we’re looking for - for example, increased confidence - then we can ask the right questions to participants and ourselves. What confidence looks or feels like will be different for everyone, so evaluation is also about making sense of plurality and complexity.

Asking the wrong questions and failing to recognize complexity can alter the shape of a project and even create negative consequences. For example, in the book Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed, James Scott gives the example of a forestry project that ultimately killed the forest by focusing only on the question of profit (“how much wood can we get from this forest?”).

Read more about this parable

Putting Ideas Into Practice:

Using JamBoard, we tried out taking a few ArtsEngage projects as case studies to guide evaluation using a 5W framework:

• WHAT is the change you want to see?
• HOW can you measure this change?
• WHO is concerned?
• WHEN should you ask questions?
• WHY does it matter? What is the “true” question? 

Click the button below and duplicate our Jamboard to try this exercise yourself!

Try this exercise

“This feels useful to do with all the stakeholders/collaborators to put us on same page”

“This has really made me think - love that the approach doesn’t have to be scientific”

Want to learn more? We've created a handy two-pager with further evaluation resources, and even suggested a few based on how much time you have!

Explore further resources

Our thanks to the Department of Canadian Heritage for their support of the ArtsEngage Learning Community!