Evaluate

In an outcomes-based commissioning environment, evaluation is critical to ensure policies and programs are meeting the needs of clients. Organisations need to think strategically about the data required to demonstrate how well their initiative is working on the ground.

We design evaluations and reviews that are rigorous, ethical and focused on real-world outcomes. We also work with our clients to build their capacity for evaluation so they can continuously improve their policies and programs.

Our services to assist clients to discover what impact they’re having include:

  • evaluation toolkits and guidelines
  • collaborative develop of program logic models and theory of change
  • research into social programs
  • summative evaluation when a program is completed
  • outcomes evaluation
  • social impact assessment
  • social return on investment
  • economic evaluation, including costing, cost-effectiveness and cost-benefit analyses
  • meta-evaluation to assess the quality of internal evaluation processes
  • synthesis of multiple evaluations
  • tailored professional development in evaluation
  • mentoring for evaluation.

Project examples

Click on any of the following projects for more information.

Peer review of outcomes of an evaluation conducted by NSW Treasury

(NSW Treasury, 2015)

Peer review of outcomes of an evaluation conducted by NSW Treasury

(NSW Treasury, 2015)

Peer review is often conducted to provide an expert independent appraisal of the quality of an evaluation. This peer review was commissioned by, and conducted for the NSW Centre for Program Evaluation and considered their evaluation of the Community Justice Program. This used a quasi-experimental propensity score matching method for measuring program outcomes. The review identified some issues with statistical power and provided advice around statistical analysis of complex interventions into complex systems and the latest advice on reporting of statistical data provided by major international journals. The review led to the Centre considering the need for bringing in evaluation peer review early in the evaluation process so that suggestions may be addressed prior to the finalisation of data collection protocols. 

Doorways Review

(The Salvation Army Australia, 2015)

Doorways Review

(The Salvation Army Australia, 2015)

Doorways is the model the Salvation Army – Australia Eastern Territory  (TSA – AUE) began to distribute from 1 April 2015 through  the Department of Social Emergency Relief (ER)  to provide financial assistance to those facing short and long term financial hardship. ARTD was commissioned to complete a review of the Doorways model to identify its strengths and weaknesses, capture internal stakeholder’s feedback, and strengthen the service delivery to ensure that the intended outcomes were being met.

The review of the six-month rollout was completed between September and October 2015 using analysis off SAMIS data pre-and post-Doorways, online surveys, completed by staff, interviews were done with randomly selected clients and staff and, a program logic workshop was facilitated by ARTD.   At the completion of the report ARTD found that the core principals of the Doorways model were generally supported, however attitudes of staff were split due to the result of poor communication and a perception of fewer clients being seen at SalvosConnect. ARTD make recommendations including providing extra training for staff, and to develop and a comprehensive monitoring and evaluating framework to monitor the implementation of the program ensuring that the organisation could track the progress and impact of Doorways on staff and clients. 

Review of the BreastScreen NSW

(Cancer Institute NSW, 2015)

Review of the BreastScreen NSW

(Cancer Institute NSW, 2015)

ARTD undertook a review of the BreastScreen NSW systems for managing invitation and bookings for breast screening, to consider the feasibility efficiency and effectiveness of centralising the booking system and make recommendations for the future. Methods included analysis of breast screening data, review of booking systems in other jurisdictions, a literature review, consultation with current system providers, collection of cost data, review of process changes, identification of issues and constraints, and cost benefit analysis of possible outsourcing arrangements and options. The review will be used by the NSW Cancer Institute to support changes to their current invitation and booking systems.

Evaluation of the Resilient Families Service

(NSW Treasury, 2013-16)

Evaluation of the Resilient Families Service

(NSW Treasury, 2013-16)

In November 2013, ARTD was engaged by NSW Treasury to evaluate Resilient Families, an intensive support service delivered to families in greater Sydney by The Benevolent Society as part of the NSW Government’s Social Benefit Bond scheme (TBS)

The purpose of the evaluation is to assess the implementation and outcomes of the service over its first three years of operation from 2013 and assess the appropriateness of the measures place for the purpose of bond payments. Throughout the three year study the service has moved to a more established stage of implementation, providing a flexible service, responsive to client needs.  ARTD’s recommendations have focussed on further development of the service to optimise outcomes for families. The Preliminary and Mid-term evaluation reports can now be viewed on the NSW Department of Premier and Cabinet’s website.

Aboriginal Women Leaving Custody Strategy

(NSW Department of Family and Community Services, 2015)

Aboriginal Women Leaving Custody Strategy

(NSW Department of Family and Community Services, 2015)

The AWLC Strategy is a multi-agency approach to reducing homelessness and reoffending, and improving wellbeing among Aboriginal women leaving custody on short-notice release. The Strategy consists of two service components (housing support and legal advocacy) and a strategic component that focuses on strengthening service partnerships and progressing policy improvements through agency coordination. 

Our evaluation will describe the implementation of these components, assess outcomes for clients, and highlight what can be learnt about how well agencies work together to identify and/or address operational and systemic policy barriers that make it difficult for Aboriginal women on short-notice release to find safe and secure housing.

We are using a mixed-methods approach that combines quantitative analysis of administrative and monitoring data with a detailed qualitative study. We are speaking to clients to understand their post-release journey.  We are also consulting with service providers and government agencies, and reviewing the documentation produced through multi-agency collaboration (e.g. minutes from local coordination group and steering group meetings).

Barriers to accessing Housing Pathways

(NSW Department of Family and Community Services, 2015)

Barriers to accessing Housing Pathways

(NSW Department of Family and Community Services, 2015)

This project extends ARTD's recent evaluation of Housing Pathways, which found that older people from culturally and linguistically diverse backgrounds and Aboriginal people have difficulty accessing social housing. The current work will explore barriers to access, and identify practical suggestions for improving access within the current system. It includes an online survey of all Housing NSW and community housing provider staff responsible for access, as well as in-depth consultation with these staff in selected regions. Aboriginal and CALD advocates will also be consulted.

Home Power Savings Program

(Office of Environment and Heritage, 2014)

Home Power Savings Program

(Office of Environment and Heritage, 2014)

The Home Power Savings Program, established in 2010, supported 225,214 low-income households in NSW to reduce their power usage and save on energy bills by providing participants with a home power assessment, a tailored action plan, and a kit of energy-saving items. Based on the recommendations of ARTD’s interim evaluation, the Office of Environment and Heritage introduced a behaviour change trial to the program in June 2013 to test methods that could support greater outcomes for participants. The trial, designed with BehaviourWorks at Monash University and the NSW Department of Premier and Cabinet’s  Behavioural Insights Team (seconded from the UK Cabinet Office), used three key triggers identified in the behaviour change literature—commitment, social norms and loss aversion—to try to improve participants’ implementation of tips in their power saving action plans. New participants were randomly allocated to one of three types of follow-up—second home visit, phone call, SMS—or no follow-up. 

ARTD’s evaluation of the trial drew on ratings of assessors’ use of behaviour change techniques, an after-treatment survey (administered through computer-assisted telephone interviewing) of 2,207 participants, 8 focus groups, 8 case studies, participant data analysis and costing analysis. The evaluation provided a robust assessment of the impact of different types of follow-up compared with no follow-up and the use of behaviour change techniques. The findings inform the design of similar programs with a behaviour change component.

First stage family case management project

(NSW Department of Premier and Cabinet, 2012)

First stage family case management project

(NSW Department of Premier and Cabinet, 2012)

This twelve-month integrated case management intervention for families, identified by government and non-government agencies as high-level service users, was a Keep Them Safe strategy. ARTD conducted a process, outcomes and economic evaluation of the pilot and an analysis to estimate the size and geographic distribution of the potential target group, to inform decisions about further rollout. The evaluation introduced the North Carolina Family Assessment Scale–General to support effective case management practice and enable assessment of changes in family functioning achieved. The evaluation drew on multiple sources of evidence, including administrative data—child protection reports, accident and emergency presentations, school non-attendance, and public housing arrears and complaints—to inform a cost-benefit analysis. The interim report informed ongoing management of the pilot and the final report contributed to the evidence base about integrated service models and outcomes.

NSW Youth Mentoring Program

(Office of Communities, 2014)

NSW Youth Mentoring Program

(Office of Communities, 2014)

The YWCA NSW delivered 4 youth mentoring models in 29 locations across NSW. To evaluate the program, we reviewed the national and international peer-reviewed and grey literature; assessed the models against the Australian Youth Mentoring Network Benchmarks; explored how the models were implemented in different contexts; and identified the strengths and weaknesses of each approach to develop a best practice model. This has since been scaled up at the state level.

The relationship between the Department and migration agents

(Department of Immigration and Border Protection, 2013)

The relationship between the Department and migration agents

(Department of Immigration and Border Protection, 2013)

The Department is faced with an increasing caseload of visa applicants, a substantial proportion represented by migration agents. One of the avenues the Department identified to deal with the increasing caseload is to improve the application process and interactions with migration agents. ARTD was engaged to conduct research into the relationship between the Department and registered migration agents for three visa categories: employer-sponsored, student, and first stage partner visas. Our research used a statistical analysis of applications data for three financial years and an online survey of 236 Department service delivery network officers, focused on their interactions with migration agents. The research identified a range of areas for improvement in how the Department and migration agents interact during lodgement and assessment of applications.

Fostering NSW Recruitment and Retention Project

(NSW Department of Family and Community Services, 2014)

Fostering NSW Recruitment and Retention Project

(NSW Department of Family and Community Services, 2014)

Community Services, NSW Department of Family and Community Services funded the Association of Children’s Welfare Agencies (ACWA) to develop and deliver the Foster Carer Recruitment and Retention Project between 2012 and 2014. ARTD evaluated the implementation and outcomes of the Project. The design used mixed methods, drawing on existing administrative data (phone and web enquiries) and campaign reach data, and collecting new data from non-government providers of out-of-home care through a survey and interviews. The results are being used to inform further recruitment and retention strategies.

The employment experience of people with disability in the NSW public sector

(NSW Department of Family and Community Services, 2012)

The employment experience of people with disability in the NSW public sector

(NSW Department of Family and Community Services, 2012)

EmployABILITY: A strategy to increase the employment opportunities for people with a disability in the NSW public sector 2010-2013 sets out the NSW Government’s strategic intent to increase employment opportunities for people with a disability in the NSW public sector. ARTD conducted research with employees with disability, their managers and HR staff to provide a better understanding of the experience of people with disability employed in the NSW public sector. We used a case study approach, recruiting participants from six units within six NSW Government agencies. To protect their privacy, employees with a disability were able to register their interest in being interviewed for the research through an online system and decide whether or not we could interview their manager. This research was part of a broader project the Public Service Commission and Ageing, Disability and Home Care used to inform policies and practices for people with a disability in the NSW public sector.

Private Rental Brokerage Service

(NSW Department of Family and Community Services, 2014)

Private Rental Brokerage Service

(NSW Department of Family and Community Services, 2014)

The Private Rental Brokerage Service operates by working intensively with eligible clients to develop and enhance their capacity to access the private rental market. The service focuses on brokering the establishment of individual private rental tenancies with local real estate agents and landlords, while ensuring support arrangements are in place to sustain the tenancy. This strategic review evaluated the delivery of the service and the outcomes achieved. It considered the service in the context of the Private Rental Assistance Strategic Framework. The review included an options paper for FACS executives to consider the future delivery of the service in the current policy context.

Sport Recruitment Programs

(Sport and Recreation, Office of Communities, 2014)

Sport Recruitment Programs

(Sport and Recreation, Office of Communities, 2014)

The Australian Rugby Union delivers two programs that aim to recruit young people to rugby. ARTD evaluated these programs, supported by our expert advisor Dr Louise Hardy from the University of Sydney. Using a pre- and post-design, we surveyed parents, conducted focus groups with children and reviewed existing data and documents to review program delivery and outcomes and assess what led to their success. The evaluation has been used by Sport and Recreation to better understand how their sport recruitment programs work. 

ADHC-CS MOU

(NSW Department of Family and Community Services, 2013)

ADHC-CS MOU

(NSW Department of Family and Community Services, 2013)

The Memorandum of Understanding (MoU) between ADHC and CS establishes an agreed position on the roles of ADHC and CS for working together to deliver services to children and young people with a disability in NSW. ARTD undertook the mid-term review of the ADHC–CS MoU. The purpose of the review was to understand how it was working and to inform the future development of the agreement in light of recent changes to the policy and operational context. It involved analysis of relevant client data collected in ADHC and CS databases and interviews with senior government stakeholders.

Graduate Diploma in Indigenous Health Promotion Alumni Project

(University of Sydney, 2014)

Graduate Diploma in Indigenous Health Promotion Alumni Project

(University of Sydney, 2014)

Established in the School of Public Health, University of Sydney, in 1998, the Graduate Diploma in Indigenous Health Promotion aims to provide a pathway to higher education for Indigenous health workers; a globally recognised qualification; and knowledge and skills specifically for advanced health promotion in urban, rural and remote Indigenous communities. The diploma program received funding to enhance students’ learning experience through alumni returning to contribute to the block teaching program. ARTD was invited to support an evaluation of the Alumni Project and worked collaboratively with staff and students to measure the outcomes of the program through surveys and interviews. Positive findings resulted in further funding being sought to continue the Alumni Project. A paper on the findings has been prepared for publication in an appropriate academic journal.

Links to Learning for youth at risk

(NSW Department of Education and Communities, 2013)

Links to Learning for youth at risk

(NSW Department of Education and Communities, 2013)

Links to Learning is targeted at students in Years 7 to 12 who are at risk of disengaging from education, and young people who have left school before completing Year 12 and are not enrolled in Vocational Education and Training. The Department funds non-government organisations and local government authorities to deliver innovative projects designed to engage these students in education. ARTD reviewed the program’s effectiveness, efficiency and appropriateness. We used a mixed-methods design, drawing on a desktop analysis and literature review, online survey of program providers, key informant interviews, and stakeholder forums. The findings were used to develop a more strategic approach to the program and identify priorities for resourcing the new model.

Four long-term housing and support programs

(NSW Department of Family and Community Services, 2013)

Four long-term housing and support programs

(NSW Department of Family and Community Services, 2013)

The NSW Homelessness Action Plan 2009–14 set the direction for state-wide reform of the homelessness service system to achieve better outcomes for people who are homeless or at risk of homelessness. It funded a range of projects using evidence-based models. ARTD was engaged to evaluate four of the funded long-term housing and support projects. We used a mixed-method design: scanning the literature, analysing project self-evaluation reports and HAP client outcomes data, assessing costs, and collecting new data through online stakeholder surveys and in-depth interviews with stakeholders and clients. We provided individual reports for each project and an overarching report on the model, which are now available on the Housing NSW website. The evaluations—along with others on different models—were used to inform future homelessness planning.

Cultural immersion training program for teachers

(NSW Department of Education and Communities, 2013)

Cultural immersion training program for teachers

(NSW Department of Education and Communities, 2013)

Connecting to Country is an innovative new program designed to develop teachers’ and principals’ willingness and ability to establish relationships with Aboriginal students through learning about their cultural, linguistic and family background. The program aims to promote quality teaching, and improve engagement and educational outcomes for Aboriginal students. ARTD’s evaluation focused on understanding and describing the impact of the program on principals, teachers, and Aboriginal students, their families and local communities. To get an in-depth understanding, we used mainly qualitative methods: observation of training sessions and interviews with principals, teachers, Aboriginal students and community representatives. Our report informed ongoing implementation of the program in NSW, and the Department submitted it to the Commonwealth Department of Education as part of the evidence about the value of the program.

One Social Housing System

(Queensland Department of Housing and Public Works, 2013)

One Social Housing System

(Queensland Department of Housing and Public Works, 2013)

The One Social Housing System was a broad suite of reforms designed to simplify tenants’ access to social housing. ARTD worked with the Evaluation Reference Group to design a framework for evaluating the reform at the system level. Data was drawn from secondary sources, including the Housing Register and National Survey of Public Housing, and new data collection included a tenant survey and key stakeholder interviews at two case study sites. Our work was the first systematic exploration of the new policy reforms, giving the new Queensland Government insight into issues affecting the social housing sector.

National package for children with autism

(Department of Social Services, 2009-12)

National package for children with autism

(Department of Social Services, 2009-12)

The Helping Children with Autism Package is a $190 million Commonwealth Government funded initiative designed to improve support, education and access to best practice early intervention services for children with autism and their families. In 2008, we worked with Department stakeholders to design a logic model for the package components they manage. This provided the basis for the evaluation framework.

In 2009, we were commissioned to deliver the evaluation in three phases—initial process evaluation, mid-term review, and outcomes evaluation—over three years. This approach enabled the evaluation to inform the development of the package during key stages of growth. All three phases of the evaluation included surveys and interviews with key stakeholder groups (families with children registered for the package, approved early intervention providers, autism advisors and diagnosticians) and an analysis of monitoring data.

The first phase focused on assessing design and initial implementation. It included consultation with representatives of professional associations and a scan of the literature on early intervention for children with autism to assess the quality and scope of available research, identify major gaps in the evidence for interventions, and identify areas for future research (e.g. on treatment mediators and moderators and participant variables) and methodological considerations.

The final evaluation focused on outcomes. It included research with families identified as facing additional barriers to access in previous phases—those from Aboriginal and culturally and linguistically diverse backgrounds and those in regional and remote areas.

The final report drew together data from all stakeholder groups across the three years to assess the package’s impact on access to best practice early intervention and contribution to improved outcomes for children and families. It also provided recommendations for improving delivery. ARTD workshopped recommendations related to their role with autism advisors to define next steps. The Department has responded to other recommendations.  ARTD and the Department presented on the findings and management response to recommendations at the 2012 PsychDD conference in Sydney.

U-Turns for Youth

(Bankstown Council, 2012)

U-Turns for Youth

(Bankstown Council, 2012)

The council’s U-Turns for Youth program provides local youth at risk of involvement in motor vehicle crime with access to support services and automotive and life skills workshops to support their engagement in education, training and employment. ARTD developed an evaluation framework, plan and data collection tools for the council’s internal evaluation of the project. We then analysed the data collected by council staff and provided a final report about the program’s contribution to the council’s goals for this sector and for the broader community.