Measurable outcomes are an integral part of successful planning and evaluation. To know if we have been successful in achieving our objectives, we must be able to understand and assess the change we seek. However, the articulation of the value of cultural development activities, including the arts, is considered particularly challenging.
This schema has been developed to assist in measuring outcomes of cultural engagement. The Measurable Cultural Outcomes offered here ensure that planning and evaluation practices in cultural development can take an evidence-based, outcome-focussed approach. This will enable the contributions of cultural engagement, often considered intangible and therefore immeasurable, to be assessed using a systemised approach. This, in turn, will enable organisations to understand how effective they are being in achieving their objectives.
The problem being addressed
Historically, arts and cultural organizations have often counted only outputs, (the amount of activity undertaken, as measured by number of arts products created, event participants or tickets sold) as metrics of success. While recording the amount of activity that occurs is important, it does not tell us anything about whether that is beneficial, and does not consider the possibility of neutral or negative outcomes, or relative benefit to costs of options for cultural participation. As yet, there are no nationally agreed standards in local government for collecting cultural data outputs.
CDN has been working with the National Local Government Cultural Forum over the last two years to establish an agreed set of measures for inputs and outputs of local government’s cultural activity. This may also be applicable to other levels of government and other cultural organisations. The data schema of inputs and outputs is now available here.
Emerging evaluation methods focus on measuring audiences’ perceptions of artwork made or experienced. While these tools assess the quality of the product, they do not measure outcomes – what difference the work ultimately makes – to those who receive it. A causal relationship might be expected, with the quality of arts products related to their impact, but this cannot be assumed.
When outcomes of engagement in cultural activity have been considered, they have often been categorised as either intrinsic or instrumental, where intrinsic is seen to be directly related to the cultural experience itself, and instrumental occurs outside, or in addition to, the cultural experience. Evaluation processes have often use social and economic outcomes (those often identified as instrumental) as proxies of cultural value, partly because outcome measures are better established in those domains, and also because economic outcomes are often prioritised above others. Assessment of intrinsic (i.e. cultural) value has been perceived as problematic, because culture is considered to be intangible and therefore difficult to measure. Thus, cultural outcomes of cultural activities have largely not been measured. More on this topic.
How the outcome schema was developed
The impetus for establishing this outcome schema came from CDN’s work with the National Local Government Cultural Forum to develop two on-line tools, a Framework for Cultural Development Planning and a program logic tool and recording system for cultural development projects. The Planning Framework offers a step by step guide for cultural development staff to create strategic plans, while the program logic tool assists with outcome-focussed planning of cultural activities. However, for these tools to be useful, a means of measuring outcomes of cultural activity was required.
The schema was developed through reference to theoretical material and input from practice experts. The process began with a wide and deep survey of the literature, encompassing theoretical pieces on the nature and function of cultural practices, policy documents from international bodies advising cultural policy-makers and outcome studies. Ideas about, and evidence of, outcomes of cultural activity were extracted from this literature. Findings from this material were sorted first into policy domain areas of cultural, social, civic, economic and environmental. For example, findings from studies indicating that cultural activity can contribute to employment and business success was grouped in the economic domain. The domain definitions as offered by Community Indicators Victoria were utilised for this step, as they offer clear distinctions between the domains, as well as clear outcomes in each domain (desired future) that are categorically different. Thus, the cultural domain is distinct from the social domain, because the desired outcome in the cultural domain is cultural richness and vibrancy, whereas outcomes in the social domain are health, safety and inclusion.
Then material gathered within each policy domain was further distilled into a set of outcomes that are considered to be categorically different from each other, while still being related and contributing to the desired future in each domain. Each domain has four or five outcomes that the literature indicates can be directly achieved or impacted by cultural activity.
These outcomes and a set of associated measures were improved by discussions with experts and peers over several years. This included presentations to more than 800 researchers, bureaucrats and practitioners from fields such as cultural development, local government, arts management, evaluation and regional development, in Australia, Asia and Europe between 2012 and 2017. Further refinement was enabled by focus group discussions and informal Delphi-style consultations (Rowe & Wright, 2001) with arts managers and decision-makers at federal, state and local government levels across Australasia. Consistently positive responses were received, indicating the potential of the framework and related measures to address evaluation needs of these professionals. The measures were endorsed by the Australian National Local Government Cultural Forum in April 2016, and are being trialled in 2017 by Forum members in different settings to validate their use and efficacy. Details of consultations re the measures.
The outcomes schema
This schema of measurable outcomes of cultural engagement offered here is based on the premise that cultural products and activities do not hold intrinsic value in and of themselves. Value is generated or experienced as humans engage with the artwork or experience, with different individuals perceiving or receiving this value in different ways. Therefore, the outcomes are not assessing ‘quality’ or ‘excellence’ of the cultural experience, but the impact on the person who engages with it.
This schema posits five domains (cultural, social, civic, economic and environmental), of public policy and activity. All activity can be considered as initiating in one or other of those domains, and all outcomes can also be categorised within them. Activities are assumed to have outcome/s in the domain in which they are sited, but they may also have outcomes in other domains, depending on their focus. For example, cultural activities obviously have cultural outcomes, but may also have economic and social outcomes.
In this approach, it is assumed that outcomes occur at an individual level, which, when aggregated, can become community, society or population level change. This is distinct from many other evaluation approaches that posit different types of change at different levels (e.g. Brown 2006; O’Hagan 2016). Thus, data can be elicited by understanding the experience of individual participants, either by direct questioning, participant observation or other methods.
Each of the Outcomes in the schema is presented with a handle, a one word heading that encapsulates the concept, supported by short and detailed Descriptions. These headings are supported by measures expressed such that change can be assessed, both in amount and direction (positive or negative). This addresses the challenge that positive change cannot be assumed to result from any activity. Evaluation questions are provided that could be used to elicit data about each outcome. These questions can be used for direct probing or as a stimulus for other types of data collection.
Evaluation methods that could be used are offered in this detailed resource. Examples of activities that could be supported by policy makers and funders to contribute towards each outcome are listed. These outcomes and measures are mapped to other measurement schema, and supported by Theory and Evidence from outcome studies, along with Processes documented as contributing to this outcome.
Evaluating outcomes of cultural development activity
This section is a growing resource to support local government and other organisations to evaluate outcomes of their cultural development activity. Information provided here is regularly updated as the outcomes and methods to measure them are trialled in different communities and for different events.
This site is a developing resource. As changes are made, this log is updated to allow all partners to track its development. The cultural outcomes are the best developed set as yet. Experts in other policy domains are currently being consulted to assist with confirmation of outcomes in the other four domains. This page documents the development of the outcomes framework, indicating when changes were made and the reasons.