Partnerships

Social problems are often complex and interconnected, so cross-sectoral initiatives and partnerships are increasingly important to government program delivery. This means evaluators need to work collaboratively with multiple stakeholders and develop effective means of evaluating programs delivered through partnerships.

Methods to evaluate partnerships have proliferated, but tend to focus only on particular aspects of partnerships. None alone provide a comprehensive picture of how a partnership is working. So we have pioneered an approach that combines a partnership survey (adapted from the Nuffield Partnership Assessment Tool), an integration measure (based on the Human Service Integration Measure developed by Brown and colleagues in Canada), Social Network Analysis (using UCINET), and qualitative data collection. This approach provides a clear picture of how a partnership is working and the outcomes it has achieved. 


Project examples

Click on any of the following projects for more information.

National partnership for and investment in preventive health evaluation

(Australian National Preventive Health Agency, 2014)

National partnership for and investment in preventive health evaluation

(Australian National Preventive Health Agency, 2014)

The Partnership Agreement was a $973 million COAG funding reform to improve the health of Australians through healthy lifestyle programs, social marketing and enabling infrastructure. In 2012, ARTD, in partnership with the Prevention Research Collaboration, University of Sydney, developed an evaluation strategy and framework for the Agreement. The framework and strategy were developed through three rounds of consultation with preventive health managers in the Commonwealth and State and Territory health departments, and peak stakeholders. We started with a logic model, which lead to an evaluation framework and strategy. All parties endorsed the framework and strategy and it provided the structure and methods for the three-year national evaluation.

In 2013 ARTD, in partnership with the Prevention Research Collaboration, were commissioned to undertake the national evaluation. The primary focus was the systems and processes of the partnership approach, within the broader context of the overall delivery of the Agreement. It had three main stages: a formative evaluation in 2013, with plans for a preliminary evaluation in 2014, and a summative evaluation to be completed by June 2015. The formative evaluation was successfully completed in 2013. It was designed to improve the partnership and to guide future investment. The evaluation synthesised reports on implementation of initiatives by the jurisdictions, held an evaluation forum, and collected primary data through interviews and online methods to assess partnership and cooperation between jurisdictions, using tools for partnership assessment, cooperation analysis and social network analysis. The NPAPH was terminated in the 2014 Budget. 

Evaluation of four long-term housing and support programs

(NSW Department of Family and Community Services, 2013)

Evaluation of four long-term housing and support programs

(NSW Department of Family and Community Services, 2013)

The NSW Homelessness Action Plan (HAP) 2009–14 set the direction for state-wide reform of the homelessness service system to achieve better outcomes for people who are homeless or at risk of homelessness. It funded a range of projects using evidence-based models. ARTD was engaged to evaluate four of the funded long-term housing and support projects. We used a mixed-method design: scanning the literature, analysing project self-evaluation reports and HAP client outcomes data, assessing costs, and collecting new data through online stakeholder surveys and in-depth interviews with stakeholders and clients. We provided individual reports for each project and an overarching report on the model, which are now available on the Housing NSW website. The evaluations—along with others on different models—were used to inform future homelessness planning.

Review of ADHC-CS MOU

(NSW Department of Family and Community Services, 2013)

Review of ADHC-CS MOU

(NSW Department of Family and Community Services, 2013)

The Memorandum of Understanding (MoU) between ADHC and CS establishes an agreed position on the roles of ADHC and CS for working together to deliver services to children and young people with a disability in NSW. ARTD undertook the mid-term review of the ADHC–CS MoU. The purpose of the review was to understand how it was working and to inform the future development of the agreement in light of recent changes to the policy and operational context. It involved analysis of relevant client data collected in ADHC and CS databases and interviews with senior government stakeholders.

Evaluation of first stage family case management project

(NSW Department of Premier and Cabinet, 2012)

Evaluation of first stage family case management project

(NSW Department of Premier and Cabinet, 2012)

This twelve-month integrated case management intervention for families, identified by government and non-government agencies as high-level service users, was a Keep Them Safe strategy. ARTD conducted a process, outcomes and economic evaluation of the pilot and an analysis to estimate the size and geographic distribution of the potential target group to inform decisions about further rollout. The evaluation introduced the North Carolina Family Assessment Scale–General to support effective case management practice and enable assessment of changes in family functioning achieved. The evaluation drew on multiple sources of evidence, including administrative data—child protection reports, accident and emergency presentations, school non-attendance, and public housing arrears and complaints—to inform a cost-benefit analysis. The interim report informed ongoing management of the pilot and the final report contributed to the evidence base about integrated service models and outcomes.

Cross-Agency Risk Assessment and Management Framework pilot for domestic and family violence evaluation

(NSW Health Primary Health and Community Partnerships, 2011)

Cross-Agency Risk Assessment and Management Framework pilot for domestic and family violence evaluation

(NSW Health Primary Health and Community Partnerships, 2011)

The pilot for key agencies aimed to test a consistent approach to assessment and risk management of NSW domestic violence cases. The trials in two regions in 2010 involved four agencies assessing risks (Police, Health, Courts, Community Services) and making referrals to NGO domestic violence services. ARTD evaluated each of the trials as case studies. The process evaluation covered referrals, case management and interagency coordination, using referral data from agency systems, surveys of program stakeholders, observations, and a partnership assessment tool. The outcomes evaluation involved a quantitative analysis of 670 cases, and interviews with a sample of clients. The evaluation report identified substantial challenges for agencies with very different responsibilities and methods, when working under a common risk management framework.

Evaluation of the Shared Access trials

(NSW Department of Family and Community Services, 2011)

Evaluation of the Shared Access trials

(NSW Department of Family and Community Services, 2011)

Shared access was a partnership involving one or more government and non-government agencies entering into a formal agreement with Housing NSW to implement locally-responsive joint assessment and allocation arrangements to deliver supportive social housing for clients with complex housing needs. It aimed to improve pathways to housing and support, build shared responsibilities and improve quality of life for clients. This evaluation provided an assessment of Shared Access based on thirteen of the fifteen trial projects established since the Shared Access Trial Schedule was signed in 2006. The evaluation used mixed methods, including a review of the literature, surveys and consultation with stakeholders.

Evaluation of pilot program for young people with challenging behaviour

(Prevention and Pathways Directorate, 2011)

Evaluation of pilot program for young people with challenging behaviour

(Prevention and Pathways Directorate, 2011)

Stay Connected, a case management program for young people with autism or intellectual disability and challenging behaviours at risk of suspension or expulsion from school, was piloted at two service locations in NSW. ARTD’s evaluation explored processes (including how the partnership between two government agencies and a non-government service provider had worked) and outcomes for young people. We scanned the literature, surveyed partner organisations using a partnership assessment tool, analysed program data and interviewed reference group members and school staff. We also interviewed parents/ carers and developed case studies from these. Our report identified critical success factors and barriers for effective case management within the partnership model to inform future implementation of the program.

Evaluation of cross-agency homelessness intervention project

(NSW Department of Premier and Cabinet, 2010)

Evaluation of cross-agency homelessness intervention project

(NSW Department of Premier and Cabinet, 2010)

The Homelessness Intervention Project comprised the Homeless Intervention Team and the Nepean Youth Homelessness Service, two cross-agency projects that provide clients with long-term accommodation and support. ARTD evaluated the project, with a focus on identifying enablers and barriers to successful implementation, assessing the extent to which the projects met their objectives, and understanding systemic issues affecting homelessness intervention. Our methods included analysis of administrative and client data, a desktop review of project reports and interviews with project staff and key stakeholders. We also analysed the cost-effectiveness of the projects. Our report outlined implications relevant to future NSW Government policies to address homelessness.