Evaluate

In an outcomes-based commissioning environment, evaluation is critical to ensure policies and programs are meeting the needs of clients. Organisations need to think strategically about the data required to demonstrate how well their initiative is working on the ground.

We design evaluations and reviews that are rigorous, ethical and focused on real-world outcomes. We also work with our clients to build their capacity for evaluation so they can continuously improve their policies and programs.

Our services to assist clients to discover what impact they’re having include:

  • evaluation toolkits and guidelines
  • collaborative develop of program logic models and theory of change
  • research into social programs
  • summative evaluation when a program is completed
  • outcomes evaluation
  • social impact assessment
  • social return on investment
  • economic evaluation, including costing, cost-effectiveness and cost-benefit analyses
  • meta-evaluation to assess the quality of internal evaluation processes
  • synthesis of multiple evaluations
  • tailored professional development in evaluation
  • mentoring for evaluation.

Project examples

Click on any of the following projects for more information.

Evaluation of Resilient Families Service

(NSW Treasury, 2016–2019)

Evaluation of Resilient Families Service

(NSW Treasury, 2016–2019)

Resilient Families is an intensive family preservation service using evidence-based practices for building resilience within families where there are concerns about the safety and well-being of children. The Benevolent Society (TBS) has been delivering Resilient Families since October 2013, under one of the two Social Benefit Bond (SBB) pilots that are currently underway in NSW. 

In November 2013, ARTD was engaged by NSW Treasury to complete the Stage 1 evaluation of Resilient Families to assess the implementation, effectiveness and cost-effectiveness in achieving benefits for families. The evaluation compared outcomes for children and families receiving the service, with a matched control cohort of children (established as part of the SBB pilot) who receive a business as usual child protection response. The difference in performance between the two cohorts is used by an independent assessor to calculate the bond payments. Stage 1 was completed in May 2016 with the Interim Report, which described the implementation and early outcomes of the service, including a cost analysis, and made recommendations to improve the matching and referral processes and the reliability of outcome measures used to calculate performance under the SBB. 

In August 2016, NSW Treasury engaged ARTD to complete Stage 2 of the evaluation. Informed by what we learnt in Stage 1, our approach will involve monitoring the progress of Resilient Families; examining benefits for clients; conducting a cost analysis and cost-effectiveness analysis (comparing the services with similar funded programs and with FACS’ business as usual response); and determining whether the measures used for SBB payments have been an adequate indicator of the intended benefits for children and families. This will include assessing the impact of revisions made to the matching, referral and bond measurement process following our Stage 1 recommendations. In addition to methods used in Stage 1—client and staff interviews, staff surveys, analysis of administrative and service data from TBS systems, and analysis of child protection data for the matched and control groups—Stage 2 will also involve a literature scan and a review of FACS case files for the matched control group to better understand the business as usual child protection response. Stage 2 will be completed in February 2019. Both stages have involved ethics approval from the University of Sydney Human Research Ethics Committee to conduct client interviews and analyse child protection and resilience data.

Evaluation of the Young Mob Leaders Program

(World Vision Australia, 2017–2019)

Evaluation of the Young Mob Leaders Program

(World Vision Australia, 2017–2019)

The Young Mob Leadership program is a culturally centred Positive Youth Development program for Aboriginal and Torres Strait Islander youth. World Vision Australia (WVA) has engaged ARTD as an independent and qualified evaluator to partner with Young Mob and WVA staff to build the evidence base of Young Mob and support its scale-up and access to ongoing government funding. ARTD will provide monitoring, evaluation and evidence development support during 2017 and 2018 and conduct an impact evaluation of the 2017-2019 period of Young Mob in 2019.  The focus of the 2017 project work plan is to review the Young Mob M&E systems and analyse relevant evidence to strengthen the evidence base and ensure appropriate data is being collected to support future impact evaluation and enable its future development.

Evaluation of the beyondblue Support Service

(Medibank Health Solutions, 2017–18)

Evaluation of the beyondblue Support Service

(Medibank Health Solutions, 2017–18)

Medibank Health Solutions is the contracted Service Provider for the beyondblue Support Service. The beyondblue Support Service aims to provide the Australian population with access to high quality information about depression and/or anxiety, as well as assistance in identifying appropriate immediate, short-term, solutions focused support and referral services via a telephone, webchat, email and online critical risk management service. ARTD has been engaged to determine the immediate and short-term (1 month) behaviour changes and mental health benefits that result from the provision of single session brief psychological support and referral via telephone and webchat for Support Service users. ARTD will conduct two online surveys of services users on their experience with the service and whether they have acted on the advice of and/or referrals provided. The findings from this project are intended to be used by beyondblue and Medibank Health Solutions to better understand the impact of the Support Service and inform how it can be improved.

Evaluation of the Comcare claims management pilot

(Department of Employment, 2016–2017)

Evaluation of the Comcare claims management pilot

(Department of Employment, 2016–2017)

The Comcare claims management pilot was conducted as part of a Commonwealth Department of Finance Efficiency through Contestability Programme. The pilot private sector insurance companies managing over a thousand claims and millions of dollars of workers compensation costs for the ATO and DHS. The first part of the project was the design of an evaluation framework. This was challenging given that decisions about the future of claims management were to be made before conclusive data on the effectiveness of the pilot could be generated. It was also challenging as the pilot was conducted in different ways in the two largest agencies each with a specific history and culture. This limited the ability to determine the likelihood of the experiences under the pilot being replicable with other agencies. To address these challenges we designed and conducted a theory-driven, empirically validated evaluation in partnership with ANU and Uncommon Approach. We used mixed methods, including administrative data, surveys, interviews and workshops. We used a realist lens and incorporated expert advice in workers compensation to assist with interpretation of patterns in the data and determine the value of the pilot. The pilot was very successful and the evaluation was able to identify the causal mechanisms and contexts by which the pilot worked to provide implications for the continuation and/ or extension of the pilot to other agencies.

Evaluation of the Modified Wage System Trial in Australian Disability Enterprises

(Department of Social Services, 2016)

Evaluation of the Modified Wage System Trial in Australian Disability Enterprises

(Department of Social Services, 2016)

The Department has worked with Australian Disability Enterprises (ADEs), unions and representatives of service providers, and people with disability to identify an alternative wage assessment tool through a Fair Work Commission conciliation process. On 15 October 2015, conciliation parties agreed a subgroup would trial a modified version of the Supported Wage System (SWS). ARTD was engaged to evaluate the Modified SWS Trial in 20 ADEs around Australia. This included designing the participant selection process and gaining formal ethics approval. The mixed-method evaluation was designed in consultation with the Steering Committee. Data collection involved visits to all 20 participating ADEs around Australia to interview supported employees (n=159), managers (n=20), staff collecting timings (n=35) and support workers not involved in collecting timings (n=22) to understand their perspectives on how the process worked and the resulting assessment outcomes. We also interviewed all assessors (n=15) involved in the Trial to understand how the approach worked from their perspective. Finally, we analysed the Trial data to assess consistency of implementation and productivity and wage outcomes, comparative to existing wage tools. The findings informed ongoing discussion of the options for wage assessment in ADEs.

NGO Benchmarking Model Stage One Evaluation

(NSW Department of Finance Services and Innovation, 2017)

NGO Benchmarking Model Stage One Evaluation

(NSW Department of Finance Services and Innovation, 2017)

The NSW NGO Benchmarking Model enables NGOs to self-assess their performance against 10 standards, related to customer-focused service delivery and organisational management, using an online platform. The Model is part of the NSW Government’s commitment to NGO capability building and strengthened human services outcomes. ARTD was engaged to conduct a formal evaluation of stage one of the Model. First, we conducted a series of workshops with government agencies and NGOs to develop an intervention logic describing how the model is intended to work and confirming the evaluation approach to ensure it can answer the key questions of all stakeholder groups. Next, in the data collection phase, we will work with a third party verifier to test the reliability of the model and seek feedback from participating NGOs and relevant government agencies through a survey and interviews. Finally, we will synthesise the findings to produce a report that will contribute to the iterative development of the Model.