The development of the Community Services Outcomes Tree
The Community Services Outcomes Tree (CSOT) has been developed by the Centre for Social Impact (CSI) at Swinburne University of Technology. It aims to resource community services organisations with some of the tools needed to underpin the measurement of outcomes experienced by consumers of their services.
The context
The demand for measurement of program and service outcomes has been steadily increasing in the community service sector. This includes community service organisations, not-for-profit organisations and social enterprises. Research undertaken by the CSI node at the University of Western Australia has shown that community services are often poorly resourced to undertake outcomes measurement. Given this, we aimed to design an outcomes framework – and data collection approach – that was relevant and useable for community sector organisations who want to measure outcomes for individuals.
An outcomes framework is a collection of outcomes that can be measured and are often organised according to key domains that may apply to the work an organisation undertakes. However, a multitude of frameworks exist which can be confusing for organisations attempting outcomes measurement. Many frameworks have been developed to be applicable to only specific initiatives (such as education or employment programmes), or they may be designed for a specific sector (such as disability or aged care), while very few have a broader approach that can be applied to all sectors.
The development of the Community Services Outcomes Tree was guided by a requirement to encourage a ‘whole of life approach’ and recognise the ways in which life domains interrelate, and that services – whatever their specific focus – are often achieving outcomes across this landscape of life domains. While our outcomes approach may also have some utility for community level outcomes, our focus was at the individual level – what has happened for users of community services (individuals and families).
Method of development
Step 1: Developing a draft framework
We began with a review of the outcomes literature using purposive sampling to identify approximately 20 core outcomes frameworks that comprise a broad ‘whole-of life-approach’ (e.g. NSW-Human-Services-Outcomes-Framework 2017, Vic Govt. Department of Health and Human Services Outcomes Framework 2019). We then coded and thematised these according to domains and outcomes (and/or indicators within these). A major focus was to separate out discrete outcome concepts, moving away from outcome definitions that amalgamated different concepts into single ‘catch all’ outcome statements. This approach aimed to increase the clarity and singularity of each outcome focus thereby making each more easily understood. Services could then select only those outcomes relevant to them or that best matched those required by their funder.
Step 2: Extensive literature review and analysis
We continued to review other government, not-for-profit and academic literature related to outcomes measurement, both in Australia and internationally. This collection of more than 200 documents, included outcomes frameworks, data collection instruments, policy documents, quality standards and funding criteria documents. Many of the identified outcomes frameworks and other relevant indices of outcome attainment were focused on just one domain or life area (e.g. education) or were at the population level (i.e. how many people have certain attributes such as completing year 12 secondary education) which we translated into a community service context for individual outcomes. Throughout the research process we collated this information and continued to confirm and/or change themes until saturation was reached where the outcomes discovered had been previously identified from other sources. This led to a variety of domain areas with numerous outcomes and indicators that we then set about constructing into the CSOT.
Step 3: Emerging structure
As discussed above, we thematically analysed the outcomes under each domain, grouping outcomes with similar themes together and seeking to name the central construct the outcome reflects. In each outcome area, the various sub themes within the outcome are reflected in the outcome description, provided on the website. The structure that emerged was:
- Domain
- Outcome
- Descriptor for outcome (based on outcome subthemes).
Throughout the construction of the CSOT we undertook a system of checking by three researchers who worked to ensure consistency across domains and outcomes, making decisions about the shape of the framework as it developed.
Step 4: Testing
The evolving framework of domains and outcomes were continually tested for face validity through application to various service contexts, partner organisations, and research projects with which the Centre for Social Impact (CSI) was engaged. The draft framework was also tested with Uniting Vic Tas staff and with service-users at several stages of development and trialled within one service evaluation in 2021. In 2022, piloting will be undertaken in a diverse range of community services within Uniting Vic Tas.
Also part of our design and testing was to ensure that the CSOT aligns with the core outcome frameworks used in Australia, many of which guided the initial design (step 1). We intended that our framework could be incorporated into other frameworks and approaches. This is beneficial as services have many funders and may have to report their outcomes according to a variety of frameworks. CSOT aims to allow data to be realigned to other frameworks, given the breadth of domains and outcomes incorporated in the Tree. To ensure this is possible, we undertook the development of a ‘translation matrix’ (called ‘recipe’ on the website, under the Resources tab) for many of the main Australian government and sector frameworks.
Step 5: Development of data collection method
In the main, outcomes frameworks do not have tools attached to assist in the collecting of data and measurement against the framework outcomes. In developing the CSOT, we elected to consider how outcomes could be collected by services that often do not have the resources or skills to undertake this task. There are many ways to collect data and our approach was to design a data collection method that could be adapted by organisations that do not have the time to undertake complex data collection and do not wish to over-burden service-users with multiple data collection tasks. We adopted a ‘minimum data set’ approach, using a lean data collection method for each outcome based on service user self-report at a single time point following service provision.
We examined many outcomes instruments, focusing on the types of data collected and language used. We decided that measuring the amount of change and the level of contribution the service has made to the change would enable meaningful data that would be useful to understanding outcomes for individuals in relation to the service. An under-pinning principle was to enable a self-report approach for service-users, both in recognition of the greater validity of this method (as contrasted with proxy report) and to value the lived experience of consumers.
Validity and limitations
Constructing an outcome framework and measurement approach is challenging and complex – both because of the volume of outcomes frameworks available and because of our recognition that creating frameworks and measuring outcomes can never be a precise and exact undertaking given that human life is messy and complex. The countless outcomes that can be achieved cannot be neatly categorised and measured in a way that is clear-cut and unambiguous. We were guided throughout by critical thinking that recognises that human social life and knowledge is often contradictory and contested, that there are a multitude of perspectives and disagreements over categorizations.
With this in mind, we have been guided by the desire to create a rigorous and valid outcomes framework and survey measurement method based on wide reading linked to practical testing and feedback. The outcomes described can be understood as having construct validity in that they have been identified and built from validated concept descriptions or those provided in both peer reviewed and grey literature. Further piloting will add data about the utility and relevance of the framework and data collection methods.
Further development
Importantly, we recognise that there is a need to continually enhance the CSOT. To this end, we are undertaking piloting of the data collection methods and the framework with Uniting Vic Tas services, testing the extent to which they are ‘fit-for-purpose’. We also seek feedback (which can be done via this website) from those who engage with the framework, ensuring that the CSOT is adapted and updated as appropriate. In this way, we see the CSOT as continuing to evolve and reflect new understandings and demonstrated need.
Overall, outcome frameworks serve as a guide, tools to enhance practice and understanding of service impact – they are not an end in themselves and we need to recognise both their value and limitations. Any data collection will always be partial in the story it tells given the complexity of service provision and service users’ lives but, while recognising this limitation, outcomes measurement offers an opportunity to capture data that can serve a useful purpose to understanding service provision in relation to outcomes. The CSOT provides one way of furthering that understanding.
For further information on how to use the Community Services Outcomes Tree and/or the survey template, see Community services outcomes tree: an introduction (available on APO).