Evaluate

In an outcomes-based commissioning environment, evaluation is critical to ensure policies and programs are meeting the needs of clients. Organisations need to think strategically about the data required to demonstrate how well their initiative is working on the ground.

We design evaluations and reviews that are rigorous, ethical and focused on real-world outcomes. We also work with our clients to build their capacity for evaluation so they can continuously improve their policies and programs.

Our services to assist clients to discover what impact they’re having include:

  • evaluation toolkits and guidelines
  • collaborative develop of program logic models and theory of change
  • research into social programs
  • summative evaluation when a program is completed
  • outcomes evaluation
  • social impact assessment
  • social return on investment
  • economic evaluation, including costing, cost-effectiveness and cost-benefit analyses
  • meta-evaluation to assess the quality of internal evaluation processes
  • synthesis of multiple evaluations
  • tailored professional development in evaluation
  • mentoring for evaluation.

Project examples

Click on any of the following projects for more information.

Tirkandi evidence building partnership

(Department of Family and Community Services, 2018−19)

Tirkandi evidence building partnership

(Department of Family and Community Services, 2018−19)

Their Futures Matter reform has funding allocated to build an evidence base of promising programs that are seen to be improving outcomes for Aboriginal children, young people, families and community. Tirkandi Inaburra, a remote cultural and development centre for Aboriginal boys 12-15, selected ARTD as an evidence building partner to help the organisation to demonstrate outcomes being achieved through their work. ARTD facilitated workshops with key stakeholders from the program, the Department of Family and Community Services and the Department of Education, to develop a program logic and outcomes matrix that aligned with the NSW Human Service Outcome Framework. We then developed a data matrix to document existing data collection processes, identified data gaps and new data collection opportunities and options. We are working closely with Tirkandi to develop and refine data tools and build a monitoring system and embed evidence-collecting in the program.

Legal Aid Refugee Service Evaluation Framework and Plan

(Legal Aid NSW, 2017)

Legal Aid Refugee Service Evaluation Framework and Plan

(Legal Aid NSW, 2017)

Legal Aid NSW is delivering a new dedicated Refugee Service, providing direct legal information, advice, assistance and community education to refugees settling in NSW, including additional refugee arrivals from Syria and Iraq, and expanding current legal services to clients from culturally and linguistically diverse backgrounds. ARTD worked with Legal Aid to develop an evaluation framework for the Refugee Service. We began with a program logic to ensure Legal Aid is collecting relevant and sufficient data to guide the evaluation of the service. Our methods included reviewing program documentation and a workshop with key Legal Aid staff in order to make the framework fit for purpose. The evaluation framework and plan will enable Legal Aid to monitor their service and have suitable data to evaluate its effectiveness.

Evaluation of the Natural Disaster Resilience Program

(Office of Emergency Management, 2018–19)

Evaluation of the Natural Disaster Resilience Program

(Office of Emergency Management, 2018–19)

The Office of Emergency Management in the Department of Justice is responsible for the management of the delivery of the Natural Disaster Resilience Program in NSW. Over the term of the 2013–15 partnership agreement with the Federal Government, the Office of Emergency Management (in conjunction with the Rural Fire Service, Office of Environment and Heritage, and the State Emergency Service) delivered six funding programs which aimed to support disaster resilience priorities at a national and state level. ARTD are evaluating the outcomes of these programs to understand the degree to which the programs have addressed these priorities. The evaluation is being overseen by an advisory panel of program managers and experts in disaster resilience from the academic and private sectors. The project commenced with the development of program logics and outcomes matrices for each of the programs and for the Natural Disaster Resilience Program as a whole, and the development of a rubric for the assessment of each of the programs in terms of addressing key priorities. To assess the programs, a mixed methods approach is being applied which combines analysis of documents and data from program activities, interviews with program managers and funding recipients, and case studies of selected projects and communities which aim to illustrate the delivery of the Natural Disaster Resilience Program funding and its impacts on supporting disaster resilience in communities. The evaluation is expected to be completed by March 2019 for delivery to relevant State and Federal ministers.

Talking Together evaluation

(NSW Fair Trading, 2018)

Talking Together evaluation

(NSW Fair Trading, 2018)

The NSW Department of Fair Trading was funded under the Information Linkages and Capacity building grants to co-deliver peer-facilitated information and education sessions to people with disability, their families and carers about their rights as consumers. This follows an initial Consumer Awareness Protection Initiative. ARTD was first engaged to develop a monitoring and evaluation framework that Fair Trading could continue to apply to consumer engagement and, subsequently, to evaluate both programs.

For the Talking Together evaluation, we worked with peer researchers to interview Fair Trading, peer facilitators and partner organisations, as well as observe education sessions and conduct focus groups with participants. Our approach was tailored to the needs of culturally and linguistically diverse communities, Aboriginal communities, people with intellectual disability and people with psychosocial disability. We held a discussion forum with Fair Trading, peer facilitators and partner organisations to ensure the findings reflected diverse stakeholder experiences and perspectives. The final report provides Fair Trading with an assessment of the effectiveness and value of co-design and co-delivery that can inform decision-making about future use of these approaches.

Dementia Australia monitoring and evaluation framework

(Dementia Australia, 2018)

Dementia Australia monitoring and evaluation framework

(Dementia Australia, 2018)

Dementia Australia formed as national organisation in 2018 and contracted ARTD to develop a Client Services Monitoring and Evaluation Framework and Strategy to ensure a consistent understanding of client services activities and intended outcomes among Dementia Australia staff. We used a capacity building approach to develop the Framework and Strategy to ensure knowledge could be transferred for use in future monitoring and evaluation activities. We held two workshops in which we worked collaboratively with the Client Services Leadership Team to develop and refine a logic model for client services, an outcomes matrix and the Strategy. We also developed client survey questions and an applicable response scale, that could be used consistently across Australia and all programs. The Framework and Strategy will assist Dementia Australia to provide consistent and effective client services to people with dementia, their families and carers across Australia.

Evaluation of Resilient Families Service

(NSW Treasury, 2016–2019)

Evaluation of Resilient Families Service

(NSW Treasury, 2016–2019)

Resilient Families is an intensive family preservation service using evidence-based practices for building resilience within families where there are concerns about the safety and well-being of children. The Benevolent Society (TBS) has been delivering Resilient Families since October 2013, under one of the two Social Benefit Bond (SBB) pilots that are currently underway in NSW. 

In November 2013, ARTD was engaged by NSW Treasury to complete the Stage 1 evaluation of Resilient Families to assess the implementation, effectiveness and cost-effectiveness in achieving benefits for families. The evaluation compared outcomes for children and families receiving the service, with a matched control cohort of children (established as part of the SBB pilot) who receive a business as usual child protection response. The difference in performance between the two cohorts is used by an independent assessor to calculate the bond payments. Stage 1 was completed in May 2016 with the Interim Report, which described the implementation and early outcomes of the service, including a cost analysis, and made recommendations to improve the matching and referral processes and the reliability of outcome measures used to calculate performance under the SBB. 

In August 2016, NSW Treasury engaged ARTD to complete Stage 2 of the evaluation. Informed by what we learnt in Stage 1, our approach will involve monitoring the progress of Resilient Families; examining benefits for clients; conducting a cost analysis and cost-effectiveness analysis (comparing the services with similar funded programs and with FACS’ business as usual response); and determining whether the measures used for SBB payments have been an adequate indicator of the intended benefits for children and families. This will include assessing the impact of revisions made to the matching, referral and bond measurement process following our Stage 1 recommendations. In addition to methods used in Stage 1—client and staff interviews, staff surveys, analysis of administrative and service data from TBS systems, and analysis of child protection data for the matched and control groups—Stage 2 will also involve a literature scan and a review of FACS case files for the matched control group to better understand the business as usual child protection response. Stage 2 will be completed in February 2019. Both stages have involved ethics approval from the University of Sydney Human Research Ethics Committee to conduct client interviews and analyse child protection and resilience data.

Evaluation of the Young Mob Leaders Program

(World Vision Australia, 2017–2019)

Evaluation of the Young Mob Leaders Program

(World Vision Australia, 2017–2019)

The Young Mob Leaders Program is a culturally centred positive youth development program for Aboriginal youth delivered in participating schools by World Vision Australia. ARTD is conducting a two-stage evaluation of the Program over a three-year period. ARTD undertook a comprehensive literature review to establish a theory of change and program logic. We then worked closely with World Vision staff to build an evidence base for the Program through refining monitoring data collection tools and processes and developing a data collection system. Monitoring data will be supplemented by the collection of qualitative data through school visits and interviews with program staff and stakeholders. We are developing two case studies of school the Program, which will be finalised at the beginning of 2018. Our final evaluation report will be delivered in March 2020. The Young Mob evaluation has been iterative in nature and ARTD is progressively feeding results and findings back to World Vision to support ongoing program improvement. Our findings will support the Program’s improvement, scale-up and ongoing funding.

Evaluation of the beyondblue Support Service

(Medibank Health Solutions, 2017–18)

Evaluation of the beyondblue Support Service

(Medibank Health Solutions, 2017–18)

Medibank Health Solutions is the contracted Service Provider for the beyondblue Support Service. The beyondblue Support Service aims to provide the Australian population with access to high quality information about depression and/or anxiety, as well as assistance in identifying appropriate immediate, short-term, solutions focused support and referral services via a telephone, webchat, email and online critical risk management service. ARTD has been engaged to determine the immediate and short-term (1 month) behaviour changes and mental health benefits that result from the provision of single session brief psychological support and referral via telephone and webchat for Support Service users. ARTD will conduct two online surveys of services users on their experience with the service and whether they have acted on the advice of and/or referrals provided. The findings from this project are intended to be used by beyondblue and Medibank Health Solutions to better understand the impact of the Support Service and inform how it can be improved.

Evaluation of the Comcare claims management pilot

(Department of Employment, 2016–2017)

Evaluation of the Comcare claims management pilot

(Department of Employment, 2016–2017)

The Comcare claims management pilot was conducted as part of a Commonwealth Department of Finance Efficiency through Contestability Programme. The pilot private sector insurance companies managing over a thousand claims and millions of dollars of workers compensation costs for the ATO and DHS. The first part of the project was the design of an evaluation framework. This was challenging given that decisions about the future of claims management were to be made before conclusive data on the effectiveness of the pilot could be generated. It was also challenging as the pilot was conducted in different ways in the two largest agencies each with a specific history and culture. This limited the ability to determine the likelihood of the experiences under the pilot being replicable with other agencies. To address these challenges we designed and conducted a theory-driven, empirically validated evaluation in partnership with ANU and Uncommon Approach. We used mixed methods, including administrative data, surveys, interviews and workshops. We used a realist lens and incorporated expert advice in workers compensation to assist with interpretation of patterns in the data and determine the value of the pilot. The pilot was very successful and the evaluation was able to identify the causal mechanisms and contexts by which the pilot worked to provide implications for the continuation and/ or extension of the pilot to other agencies.

Evaluation of the Modified Wage System Trial in Australian Disability Enterprises

(Department of Social Services, 2016)

Evaluation of the Modified Wage System Trial in Australian Disability Enterprises

(Department of Social Services, 2016)

The Department has worked with Australian Disability Enterprises (ADEs), unions and representatives of service providers, and people with disability to identify an alternative wage assessment tool through a Fair Work Commission conciliation process. On 15 October 2015, conciliation parties agreed a subgroup would trial a modified version of the Supported Wage System (SWS). ARTD was engaged to evaluate the Modified SWS Trial in 20 ADEs around Australia. This included designing the participant selection process and gaining formal ethics approval. The mixed-method evaluation was designed in consultation with the Steering Committee. Data collection involved visits to all 20 participating ADEs around Australia to interview supported employees (n=159), managers (n=20), staff collecting timings (n=35) and support workers not involved in collecting timings (n=22) to understand their perspectives on how the process worked and the resulting assessment outcomes. We also interviewed all assessors (n=15) involved in the Trial to understand how the approach worked from their perspective. Finally, we analysed the Trial data to assess consistency of implementation and productivity and wage outcomes, comparative to existing wage tools. The findings informed ongoing discussion of the options for wage assessment in ADEs.

NGO Benchmarking Model Stage One Evaluation

(NSW Department of Finance Services and Innovation, 2017)

NGO Benchmarking Model Stage One Evaluation

(NSW Department of Finance Services and Innovation, 2017)

The NSW NGO Benchmarking Model enables NGOs to self-assess their performance against 10 standards, related to customer-focused service delivery and organisational management, using an online platform. The Model is part of the NSW Government’s commitment to NGO capability building and strengthened human services outcomes. ARTD was engaged to conduct a formal evaluation of stage one of the Model. First, we conducted a series of workshops with government agencies and NGOs to develop an intervention logic describing how the model is intended to work and confirming the evaluation approach to ensure it can answer the key questions of all stakeholder groups. Next, in the data collection phase, we will work with a third party verifier to test the reliability of the model and seek feedback from participating NGOs and relevant government agencies through a survey and interviews. Finally, we will synthesise the findings to produce a report that will contribute to the iterative development of the Model.