Design

Policies and program designs will be more appropriate if they are designed with the people they affect. So government agencies and non-government organisations need appropriate ways to give clients, staff and communities a voice in designing policies and programs.

We work with our clients to engage with their communities and stakeholders, and give a voice to those traditionally disempowered in consultation processes. We help develop communications strategies, clearly outlining target audiences, key messages, and appropriate channels to enable a civic and corporate conversation. We often use partnership approaches when working with Indigenous communities, people from cultural and linguistic diversity and people with disability.

Our services to inform policy and program design include:

  • rapid scans of academic and credible grey literature
  • translation of research for practice and practice-based evidence
  • co-design processes for policies, programs and regulation
  • sensitive and creative facilitation
  • multi-modal communications strategies
  • face-to-face public consultation forums made inclusive through accessible venues, hearing loops, translations and appropriate processes
  • online surveys, discussion forums and websites to elicit feedback
  • in-depth workshops with local communities
  • collaborative development of program logic models and theories of change
  • evaluation frameworks and strategies
  • discussion papers
  • design specifications.

Project examples

Click on any of the following projects for more information.

Stakeholder Consultation for the Central Coast LHD Community Health Centre

(The Agency for Clinical Innovation, 2015)

Stakeholder Consultation for the Central Coast LHD Community Health Centre

(The Agency for Clinical Innovation, 2015)

ARTD was initially commissioned by the Agency for Clinical Innovation to provide stakeholder consultation for a project they funded on the Central Coast. The successes of the first stage lead to a suite of high-level evaluation advice, data analysis and formative evaluation projects to inform a broader review of reforms to improve the efficiency of the Central Coast Community Health Service. We commenced with a baseline analysis of quantitative and qualitative data (surveys and focus groups) on stakeholder (clients, community health staff, allied health professionals and GPs) perceptions about referral pathways, communication systems, patient outcomes, barriers and success factors in the local service system.

The second project was to analyse Community Health Occasions of Service and staffing levels across seven community health service types, and modelling and projections for Potentially Avoidable Hospitalisations using Activity Based Funding data. The third project, identified by the evaluation steering committee, was a formative evaluation of the introduction of a major IT reform designed to increase efficiency and effectiveness of the system. This included face-to-face interviews and site visits with all affected frontline staff and separate interviews with their managers. Post-intervention stakeholder consultations are planned for 2015. ARTD has worked closely with ACI on this project, providing consolidated reports to assist ACI to perform its broader monitoring and evaluation role for the Central Coast reform project. 

ACI BMT Survivors Model of Care Evaluation Plan

(The Agency for Clinical Innovation, 2015)

ACI BMT Survivors Model of Care Evaluation Plan

(The Agency for Clinical Innovation, 2015)

The ACI commissioned ARTD to develop an evaluation framework and plan for a new model of care Long-Term Follow-Up for Allogeneic Blood & Marrow Transplant (BMT) Survivors for ACI BMT Survivors Model of Care Evaluation Plan. ARTD spoke with key stakeholders, including service directors and clinicians within BMT units across NSW, to develop the evaluation framework and plan. Workshops were held to test a draft evaluation framework and plan with stakeholders. The framework and plan will be used to support implementation of an evaluation of the new Model of Care, once the MoC is finalised and implemented across NSW.

Knowledge Hubs Evaluation Framework

(NSW Department of Industry, 2015)

Knowledge Hubs Evaluation Framework

(NSW Department of Industry, 2015)

Knowledge Hubs are industry-led opportunities to improve the competitiveness of key industry sectors in NSW by increased innovation through collaborative partnerships and knowledge sharing. The Knowledge Hub Initiative commenced in 2015 as a pilot with five Knowledge Hubs positioned in industry or research organisation in financial services, energy efficiency, transport logistics, digital creative and medical technology. ARTD developed an evaluation framework for the Imitative and designed the evaluation of the initial implementation period, based on consultations with stakeholders and program logic. The evaluation design used data collected by the Hubs on engagement and their activities to build capacity for knowledge sharing and innovation. 

Evaluation of the stakeholder support for the 2015 smoke-free outdoor dining campaign

Centre for Population Health

Evaluation of the stakeholder support for the 2015 smoke-free outdoor dining campaign

Centre for Population Health

The NSW Ministry of Health conducted a media campaign in 2015 to inform the community and relevant industry stakeholders about the new legislation to ban outdoor smoking in dining areas of restaurants, cafes and hotels. As part of the campaign, the Ministry engaged industry, LHD and non-profit organisations to provide support for the media campaign. This ARTD project evaluated the support provided by these stakeholders specifically the nature, extent and value of that support.

The evaluation included stakeholder on line surveys and interviews. A full report and summary brochure for stakeholders were prepared by ARTD. The findings will be used to improve stakeholder engagement strategies in future NSW Health campaigns.

 

Restart Programme

(Department of Employment, 2015)

Restart Programme

(Department of Employment, 2015)

The Restart Programme provides an incentive in the form of a wage subsidy to employers to hire and retain a full time job seeker aged 50 or older, who has been unemployed and on income support for six months or more. Initially this was provided over 24 months. ARTD was commissioned to design an evaluation strategy to inform decisions about the nature and delivery of Restart to maximise employment outcomes. The strategy being developed commenced with a literature review of key factors and approaches to evaluations of subsides in the international literature. We interviewed key stakeholders and facilitated a programme logic workshop to identify the program theory of change. The framework (in progress) will identify methods appropriate throughout the lifecycle of the program, ranging from laboratory based experiments to monitoring, qualitative data collection and randomised controlled trials.

Mandatory Alcohol Interlock Program

(Transport for NSW, 2015)

Mandatory Alcohol Interlock Program

(Transport for NSW, 2015)

The NSW Government legislated that serious and repeat drink drive offenders be ordered to place an alcohol interlock device on their family or work vehicles. This is a complex multi-component program being delivered in partnership by Transport for NSW, Roads and Maritime Services, police, courts and general practitioners. ARTD designed the evaluation framework to guide future evaluation studies and identify the performance information needed to regularly monitor the effectiveness of delivery, take-up and drop-out rates and the impact of the program on repeat drink driving rates in NSW.

Framework for Sustaining Aboriginal Tenancies

(NSW Department of Family and Community Services, 2014)

Framework for Sustaining Aboriginal Tenancies

(NSW Department of Family and Community Services, 2014)

We used a consultative and iterative process to develop a framework for Sustaining Aboriginal Tenancies across social housing. The project began with a review of published literature, consultations across the state with Aboriginal service providers, and a state-wide forum of service providers from across the social housing sector. We used the information to develop a draft framework, which was then tested with Housing NSW and community housing providers to ensure key issues were not overlooked and difficult issues were confronted head-on. The project delivered an evidence-based framework to guide social housing teams in supporting positive tenancy outcomes for Aboriginal clients. It has been published and printed for distribution by FACS to the social housing sector.

Accreditation program for counsellors

(NSW Office of the Children

Accreditation program for counsellors

(NSW Office of the Children

The Office of Children's Guardian administered an accreditation program for counsellors of people who sexually offend against children. ARTD was engaged to review the program based on submissions from peak bodies, in-depth interviews with key stakeholders and an examination of program documents. The review documented the benefits and risks of the program to the government, professionals and the community, and found that it is still needed, even with changes in registration of professions over the last five years. The review also made recommendations about the program’s retention and ways delivery could be improved.

Treasury Managed Fund Agency Survey

(NSW Self Insurance Corporation, 2015)

Treasury Managed Fund Agency Survey

(NSW Self Insurance Corporation, 2015)

ARTD developed an agency satisfaction framework for determining the level of satisfaction with claims manager services among NSW public sector agencies. A survey was developed based on the framework, which has been implemented yearly to determine whether claims managers are meeting their contractual obligations. ARTD developed an online delivery system for the survey and has implemented it and analysed the survey data each year since 2010. Analysis involves comprehensive statistical analysis, including weighting results to account for agency size.

Murray Darling Basin Environmental Water Knowledge and Research Project

(Murray-Darling Freshwater Research Centre, 2015)

Murray Darling Basin Environmental Water Knowledge and Research Project

(Murray-Darling Freshwater Research Centre, 2015)

The Murray-Darling Basin Environmental Water Knowledge and Research project is a five-year collaborative research project to improve the knowledge base for managing environmental water in the Basin. ARTD designed the evaluation strategy for the project on behalf of the Murray-Darling Freshwater Research Centre, which is coordinating the whole project for the Australian Department of Environment. The project involved two rounds of stakeholder consultation to develop the evaluation strategy, including a program logic and detailed evaluation work plan for the two phases of the project.

Aboriginal consultation model

(NSW Department of Family and Community Services, 2010 and 2012-14)

Aboriginal consultation model

(NSW Department of Family and Community Services, 2010 and 2012-14)

Protecting Aboriginal Children Together (PACT) is a Keep Them Safe initiative, designed to respond to recommendation 8.5 of the Special Commission of Inquiry into Child Protection Services in NSW to ‘consider establishing a ‘Lakidjeka’ type model of consultation to provide an Aboriginal perspective in relation to the best ways of keeping children and young people safe’. In 2010, ARTD worked with Community Services, the Aboriginal Child, Family and Community Care State Secretariat (NSW), and community stakeholders to develop the service model, guidelines for providers, and a methodology for determining service costs. In 2011, two sites were selected and local Aboriginal non-government services were funded to pilot PACT, working alongside Community Services’ caseworkers.

ARTD was subsequently engaged to evaluate the pilot to inform decisions about further rollout across NSW. We delivered the project in partnership with an evaluation steering committee. With the program’s extended establishment period, the focus of the two-stage evaluation was on assessing how implementation worked in practice and gathering early indications about what the model can achieve in two very different sites. We interviewed PACT and Community Services’ staff, families and community representatives to understand how the model was working from their perspectives. We also analysed program monitoring and administrative data to assess implementation and early outcomes, and collected costs and workload data to identify any adjustments needed to the cost modelling developed for the pilot. The evaluation informed ongoing management of the pilot.

IPART stakeholder research

(Independent Pricing and Regulatory Tribunal of NSW, 2013)

IPART stakeholder research

(Independent Pricing and Regulatory Tribunal of NSW, 2013)

The Independent Pricing and Regulatory Tribunal (IPART) provides an integrated system of economic and licence regulation in NSW, and in carrying out its tasks interacts with a range of agencies. ARTD developed an agency interaction quality framework and a survey to measure satisfaction among agencies. The survey has been implemented bi-annually since 2001, and has included an online survey and stakeholder interviews. This research has allowed IPART to be strategic about improving quality, with satisfaction among agencies improving significantly over time in areas targeted by IPART.

National Asbestos Plan options for evaluation

(Asbestos Safety and Eradication Agency, 2015)

National Asbestos Plan options for evaluation

(Asbestos Safety and Eradication Agency, 2015)

The Asbestos Safety and Eradication Agency is responsible for monitoring the progress of a National Strategic Plan for Asbestos Management and Awareness.  Responsibilities for implementing the plan are dispersed across jurisdictions and issue areas. ARTD prepared a discussion paper that outlined options for evaluating the national plan, taking account of constrained resources and existing data collection databases.

Best practice model and implementation guide for diagnosis support

(Ageing, Disability and Home Care, NSW Department of Family and Community Services, 2012-13)

Best practice model and implementation guide for diagnosis support

(Ageing, Disability and Home Care, NSW Department of Family and Community Services, 2012-13)

The time of a child‘s identification as at risk of diagnosis with disability can be an emotional one for families as they struggle to come to terms with what it means for them, identify and access the services they need, and develop natural supports. Ageing, Disability and Home Care’s EarlyStart Diagnosis Support Program was established to meet the needs of families around this time. In 2012, after the program had been operating for three years, ARTD reviewed the program model in light of emerging research evidence, the experience of initial implementation, and the evolving policy context, in particular the transition towards the National Disability Insurance Scheme. The first step was a scan of the available literature on diagnosis support and a report on service models, outlining implications for the program. Next, we consulted with service providers, families and other stakeholders and provided a second report, which identified how existing program delivery compared with the research evidence and how it fitted in the emerging policy context, and made recommendations for refining the program model. In the final stage, we worked with Department staff and service providers to develop a logic and outcomes framework for the program and a Practice and Induction Package to support consistent and effective delivery of the program across NSW, and continuous improvement.

Stakeholder consultation for school counselling services review

(NSW Department of Education and Communities, 2012)

Stakeholder consultation for school counselling services review

(NSW Department of Education and Communities, 2012)

In 2012, the Department undertook a review of counselling services available to children and young people in NSW Government schools. ARTD managed the stakeholder consultation component. We collected feedback from 5,000 stakeholders over three months. Over 600 school counsellors and district guidance officers, nearly 900 principals, and nearly 200 students completed online surveys. We interviewed principals/ deputy principals (n=65), school counsellors (n=34), district guidance officers (n=19), parents (n=23), and representatives of 6 government agencies, and ran focus groups with students from across NSW, regional support staff and peak organisations. We randomly selected school staff for interviews using a stratified sampling strategy to ensure we spoke to staff working in the full range of contexts across NSW. We also reviewed formal submissions from agencies and organisations. Our report provided the Department with an overview of the range and nature of school counselling services, their strengths and weaknesses, and trends in student counselling needs. In 2013, we prepared a monograph to summarise the findings for publication.

National Partnership Agreement on Preventive Health

(Australian National Preventive Health Agency, 2012-14)

National Partnership Agreement on Preventive Health

(Australian National Preventive Health Agency, 2012-14)

The National Partnership Agreement on Preventive Health (NPAPH) was a $973 million COAG funded reform to improve the health of Australians through Commonwealth and State and Territory Governments working in partnership to deliver healthy lifestyle programs, social marketing and enabling infrastructure. In 2012, ARTD, in partnership with the Prevention Research Collaboration, University of Sydney, developed an evaluation framework and strategy for the Agreement through three rounds of consultation with preventive health managers in the Commonwealth and State and Territory health departments, and stakeholders in peak organisations. All parties endorsed the framework and strategy, which provided a basis for the three-year national evaluation.

We were subsequently commissioned to undertake the evaluation, focusing primarily on the systems and processes of the partnership approach, within the context of the overall delivery of the NPAPH funded programs. In 2013, we successfully undertook the formative evaluation, which involved interviews with the Implementation Working Group, Healthies Steering Committee representatives and program managers in states and territories; a national evaluation forum to support evaluation of the NPAPH funded programs in states and territories; and an assessment of the partnership. We combined three methods for the partnership assessment—a partnership survey, cooperation assessment and social network analysis—to consider the partnership at all levels and provide a comprehensive understanding of how partnerships were working. The NPAPH was terminated in the 2014 Budget and the two remaining stages of the evaluation were cancelled.

Camp Quality

(Camp Quality, 2014)

Camp Quality

(Camp Quality, 2014)

In 2012, Camp Quality released its new strategic plan for 2012–2017, which makes a strong commitment to the evaluation of both current and new programs. ARTD was commissioned to develop an overarching Evaluation Strategy to align with the Strategic Plan. Through workshops with staff and two rounds of consultation, ARTD developed an agreed Evaluation Strategy, comprising program logic models, an overarching evaluation framework, evaluation questions, priorities for evaluation, and an overall approach to evaluation activities and data collection to 2017. The Evaluation Strategy will help to focus evaluation efforts where they will be most effective and provide a framework and guide for developing more detailed program evaluation plans and implementing evaluations during the life of the Strategic Plan.

Stronger Futures

(Department of Social Services, 2014)

Stronger Futures

(Department of Social Services, 2014)

Stronger Futures is a ten-year legislated package of funded activity designed to increase the health and wellbeing of Indigenous people in communities across the Northern Territory. ARTD provided advice and facilitated a workshop for Australian Government stakeholders to share data collection and other evaluation activity within their respective portfolios, contributing to the development of a Stronger Futures monitoring and evaluation framework.

Person-Centred Approaches Program

(National Disability Services, 2013)

Person-Centred Approaches Program

(National Disability Services, 2013)

The Person-Centred Approaches Program aims to move all NSW disability services toward a person-centred approach in line with the goals of Stronger Together 2 and the principles for the National Disability Insurance Scheme. ARTD was engaged to develop an evaluation framework for the program. Using a review of documents and a workshop with stakeholders, we developed a program logic, then used this as the basis for monitoring and evaluation for the next three years.

Mentoring programs for school students

(Department of Education and Child Development, 2013)

Mentoring programs for school students

(Department of Education and Child Development, 2013)

South Australia’s Community and Secondary Mentoring programs provide students at risk of disengaging from school with access to a mentor for one hour per week during school time. ARTD was engaged to design an evaluation framework for the programs and then to conduct the process and outcomes evaluation in two stages.

To assess outcomes we used a quasi-experimental design, with pre- and post-testing of resilience, optimism, social relations and love of learning among participating students and a control group of similar students who did not receive mentoring. To understand which students mentoring worked for, why and how, we also used a realist approach—forming a hypothesis from qualitative data from teachers, mentors and students, and then testing it using quantitative data. The final report made recommendations to further develop the programs and target them at the students most likely to benefit.

Settlement Grants program

(Department of Immigration and Border Protection, 2012)

Settlement Grants program

(Department of Immigration and Border Protection, 2012)

The long-established Settlement Grants program funds community organisations to help newly-arrived humanitarian migrants settle in Australia, targeting those with the greatest needs. ARTD was first engaged to work with Departmental stakeholders to develop an evaluation framework for the program. We then undertook a detailed evaluation of the program. The exploratory design was based on a program logic with a realist orientation. We looked at the targeting of the program, the effectiveness of funded projects, and the suitability of the program settings. We drew on multiple sources of evidence: settlement data, program data, interviews with stakeholders, surveys of grant managers and services, and client interviews. We also undertook ten in-depth case studies at sites around Australia. The evaluation led to recommendations to change program settings and improve performance monitoring.