Archives: Resources

By Fanny Martin

As we’re slowly coming back to live performances, orchestras have a fantastic opportunity to embed evaluation into the way they work to engage, observe, listen and spot the seeds for change that will make them more resilient, responsive and relevant. 

Art of Festivals is a creative company specialised in strategic production to bring more art into everyday life and more life into the arts sector. We’ve worked with many professional ensembles, music training programmes, community music organisations and participatory projects in Canada and Europe, and we contributed to Orchestras Canada’s Resilient Orchestras Ontario programme by creating bespoke tools and frameworks adapted to the needs of regional & community-led orchestras. 

We’re sharing here a simple evaluation methodology to collect evidence about audience behaviour at events and performances, help you support new directions in programming and marketing, engage deeper with all stakeholders and amplify your shared core values. 

 

A. What is evaluation? 

Evaluation is a creative practice that feeds into all dimensions of an arts organisation: artistic programming, operations, governance, stakeholder engagement, financial management. 

It opens opportunities for dialogue with artists, audiences, partners and funders; and generates information to support essential organizational functions such as programming, advocacy and strategic planning.   

BetterEvaluation—a website dedicated to improving evaluation practice and theory—defines evaluation, in its broadest sense, as “any systematic process to judge merit, worth or significance by combining evidence and values”. 

Evaluation specialists have developed many methods and processes adapted to different scales, purposes and resources. This array of choices—sometimes heavily theoretical—can feel intimidating, especially if we’re already short on time and people. We may also get worried about unearthing conflict, disappointment or shortcomings that could compromise our relationship with funders and partners. 

To get beyond the jargon and confusion, it’s useful to reframe evaluation as, simply, a process of reflective observation that’s only truly effective when it’s designed and implemented at your own scale and pace. So if you’re starting your evaluation journey, you just need to keep three principles in mind: 

• start small: by using simple methods, you’ll have more flexibility to adapt them to circumstances and improve them as you learn;  
• be curious: ask open questions, listen actively and observe keenly to get behind the obvious and understand what people really get—or want to get—from your offering;  

• and keep going: this is the “systematic” part of the evaluation process, which simply means that you should follow the same method from start to end, again and again, to get the compounded evidence you need to base sound decisions upon. 

 

B. What are we evaluating?   

In “evaluation”, there is “value”—and what we’re really talking about here is assessing the presence, quality or intensity of values.

Values, as defined by UK-based social change consultancy Common Cause, are 

“…  the principles or standards that we each carry through our lives and that guide and inform our thoughts, attitudes and actions. They influence, and are influenced by, our experience of the society in which we each live. Our values help determine what is important to us and shape how we interact with other people and the more-than-human world.

When you’re thinking about programming, marketing, interacting with players, audiences and partners, which principles are guiding your decisions? What do you want people to do, think and feel as a result of what you’re offering to them?”

 

In the Evaluation Grid template provided below, we worked on the following values: 

CONNECTION → Are we creating community? Are people feeling closer to each other?

CURIOSITY → Are people stimulated? Are they learning, discovering new things, open to new people? 

CONTENTMENT → Are people satisfied? Are they getting what they expected—and perhaps even more? 

 

C. Which values do you want to amplify?   

In Emergent Strategy, social justice facilitator and writer adrienne maree brown states as one of her guiding principles: “what we pay attention to grows”. Just like a magnifying glass focuses the sun’s energy to create heat and light a fire, evaluation can amplify the shared values we want to activate. 

Another key principle from Emergent Strategy is the notion of scale: “how we are at the small scale is how we are at the large scale”. The evaluation process itself can contribute to this amplification: the core values—such as community, participation, beauty, curiosity, courage…—that you want to observe in your stakeholders’ actions and reactions can also guide the way questions are developed, how observers are trained, and how results are analysed. In other words, everything matters in your process. Values are amplified by making time and space for people to co-create a meaningful framework, by ensuring that all interactions are positive and intentional, by being attentive to tension, conflict, resistance… 

 

D. How to develop and apply your own Evaluation Grid? 

To develop your evaluation framework, you will need to: 

• Establish when and where the evaluation will take place 
• Confirm who will carry out the evaluation (and train them) 
• Define why you are conducting this evaluation by co-developing your core values

The template Evaluation Guide below can be customised as a reference document for your staff, board and volunteers, with pointers on what to pay attention to and which questions to ask to elicit the responses that will feed into your analysis. 

A key principle in evaluation is to only collect information that you can use. Instead of lengthy and intrusive on-site questionnaires or online surveys, this simple tool will: 

• Actively engage your players, volunteers, board members, youth musicians’ parents… into the life of the orchestra; 
• Create a platform for interacting with audience members that observers can use in the ways they are most comfortable with;  
• Enable you to collect evidence linked to the shared values you want to amplify. 

 

E. How to analyse and utilise results? 

To collect responses, you could ask observers to hand in their printed Evaluation Grid after each event (making sure to write in a legible manner!), or to fill in an online form as soon as possible.  

Once all responses are collated, they could form the basis for a facilitated discussion. A few questions to get started could be: 

• What surprises you most? 
• What can be addressed immediately? 
• What needs to be tackled on a longer-term scale? 
• What should the next observation pay more attention to? 
• How can the Grid or the observation methods be improved? 

→ Evaluation – or reflective observation – is an effective tool to enhance change, understand resistance and amplify your core values. Start small, make it fun and participatory, celebrate results and keep going!

Templates:

Observation Form

Observation Guide