ARTD's evaluation of the Australian Volunteers for International Development (AVID) program has been endorsed by the Independent Evaluation Committee for Australian Aid. The Committee noted the report was thoughtful and based on strong analysis. ARTD was commissioned to evaluate this program, which places volunteers with organisations in 37 countries, to enhance its effectiveness.
ARTD’s evaluation of the Mental Health Consumer Perceptions and Experiences of Services (MH-CoPES) Framework has been published on the NSW Consumer Advisory Group Inc website (NSW CAG). The Framework has four steps—data collection, data analysis, reporting and feedback, action and change—that NSW mental health services are expected to implement in repeated cycles. NSW CAG was funded to develop the Framework to provide a way of ensuring mental health consumer perspectives inform service quality improvement. Consumer involvement in evaluation is a requirement of the National Standards for Mental Health Services but previous research shows a number of barriers to this, including staff attitudes toward consumer participation, lack of clarity about how to engage consumers and support to do so, and fear consumer participation will lead to unrealistic expectations. Consumers can also be reluctant to engage in evaluation processes because of fear of repercussions, concerns about maintaining their privacy and confidentiality, and prior experience of tokenistic consultation.
Our evaluation of the MH-COPES Framework drew on a literature scan, a survey of all services, and consultation with service managers, staff, consumer workers and consumers.Key findings were that following significant investment in the Framework by the NSW Ministry of Health and strong support from senior management, most mental health services in NSW had begun implementing the Framework. However, they had encountered a number of barriers in doing so. After the initial two-year implementation period only a minority of services had completed all four steps. But positive experiences among those that had done so suggest the Framework has the potential to support consumer participation and feed into quality improvement if some adjustments can be made to the Framework, consumer questionnaires, and structural supports.
ARTD is now evaluating the HomeStay Support Initiative on behalf of the Queensland Department of Housing and Public Works. This initiative is an early and post-crisis intervention service that aims to support vulnerable people—particularly singe adults, families and older people—to address issues threatening their ability to maintain a tenancy. It provides a case management approach and links clients to relevant mainstream and specialist information and support services within their local community. The evaluation will explore how the HomeStay service has been established and is delivering support across sixteen sites, with a focus on four sites for case studies. It is funded under the National Partnership Agreement on Homelessness.
ARTD consultant Dr Ioana Ramia presented her research findings about the impact of income and education on subjective wellbeing at the Australian Social Policy Conference 2013. You might assume wellbeing would be associated with the material satisfaction that comes with higher educational achievement, better job outcomes and higher income, but the research says different. Researchers have often found people with no tertiary education are happier or more satisfied with their lives than those with a tertiary education.
Ioana’s research, conducted for her doctorate through the University of NSW, used data from the 2010 Household Income and Labour Dynamics (HILDA) Survey to look at this issue in Australia. She found that the impact income has on subjective wellbeing differs between those with and without a tertiary education, but the results vary with the measure of income used (individual or household).
When asked about wellbeing, those with a tertiary education think more of their homes and their free time, while those without a tertiary education think more about their salary, job prospects and neighbourhood. While they're answering the same question, they're assessing different things. For those without a tertiary education, happiness was correlated to how much money they had, but not exhaustively. After hitting the median income for people without a tertiary education people, more money does not increase happiness.
The NSW Government launched its new Evaluation Toolkit to support NSW government agencies to undertake program evaluation projects on 21 August 2013 at the Why Evaluation Matters: Basing Decisions on Evidence Forum in Sydney. ARTD’s Chris Milne and Sue Leahy worked with Professor Patricia Rogers of RMIT University in developing the toolkit for NSW public sector managers in line with the NSW Government Evaluation Framework. It takes managers who are responsible for a program evaluation through seven steps—from initial conception, through commissioning and managing, to disseminating and supporting use of the findings. The Toolkit will be refined over time based on user feedback.
ARTD is now evaluating the behaviour change component of the NSW Government’s Home Power Saving Program (HPSP). This program, which started in July 2010, was established to help 200,000 low-income households reduce their power usage and save on energy bills by June 2014. It includes three main components provided at no cost to participants: 1) a home power assessment, 2) a tailored action plan that identifies ways the household can save power, and 3) a kit of energy efficient items (including a standby saver power board, light bulbs and other small items and a water-saving showerhead if needed). The $63 million program is managed by the Office of Environment and Heritage and delivered across the state by about 100 contacted energy experts.
Based on recommendations from ARTD’s interim evaluation of the program in 2012, OEH introduced a new version of the program in June 2013 to test behaviour change intervention methods that could support greater outcomes for participants. OEH designed the behaviour change trials with BehaviourWorks at Monash University and the NSW Department of Premier and Cabinet’s Behavioural Insights Team (seconded from the UK Cabinet Office). The trials are using three key triggers identified in the behaviour change literature— commitment, social norms and loss aversion—to try to improve participants’ implementation of tips in their power savings action plans. From June 2013, each new participant is being randomly allocated to one of the three trial groups—which receive different types of follow-up support aligned with one of the behaviour change triggers—or a control group—which receives the program as usual.
ARTD’s evaluation of the trial, being led by Andrew Hawkins and Florent Gomez-Bonnet, is using a randomised control trial (RCT) to test the impact of each type of intervention on behaviour change and energy savings (measured through analysis of external billing data), combined with a mixed-methods design, based on the Theory of Planned Behaviour, to try to understand observed behaviour. Together, these approaches will provide robust evidence about the value of the program and evidence to support refinements to further promote energy efficient behaviour among low-income households.
You can find out more about the work of the UK’s Behavioural Insights Team, or Nudge Unit,here. For more detailed information on the approach, check out, Richard Thaler and Cass Sunstein’s book Nudge or Daniel Kahneman’s Thinking, Fast and Slow.
The NSW Centre for Education Statistics and Evaluation (CESE) has released the findings of ARTD's analysis of the 2011 Cross-sectoral Impact Survey (CSIS). This survey captured responses from 662 of the 936 NSW schools participating in a Smarter School National Partnership (SSNP). In total, 4,376 individuals completed a survey: 393 principals, 1,331 executives and 2,652 teachers. This is the first of a wave of surveys being administered every year up to 2017.
The 2011 CSIS provides a snapshot of the extent of change in key education practices achieved in Smarter Schools National Partnership schools by September 2011. The CSIS documents the extent of reform at the individual staff member, school and system level and the sustainability of the reforms, through each round of surveys. The information being collected is both relative and retrospective—the survey asks respondents to compare education practices in schools and classrooms prior to participating in the SSNP with where they are now. This allows the survey to account for different subjective starting points and to ask about the added value of being involved in an SSNP.
ARTD is currently finalising the report on the findings of 2012 CSIS and preparing to administer the 2013 survey.
Housing NSW has published the report from ARTD’s evaluation of four long-term housing and support projects funded under the NSW Homelessness Action Plan (HAP), which set the direction for state-wide reform of the homelessness service system to achieve better outcomes for people who are homeless or at risk of homelessness. ARTD’s evaluation focused on processes outcomes achieved for the service system through four different project models. The project included a literature review on the supportive housing model, online surveys of project stakeholders, site visits to collect in-depth data, analysis of monitoring and administrative data and a cost-effectiveness analysis. The evaluations are to inform future homelessness planning.
Ageing Disability and Home Care (ADHC) has released the high-level findings from ARTD’s evaluation of four models of early intervention for children with autism aged 0 to 5 years and their families. The four programs—Footprints (Autism Behavioural Intervention NSW), Building Blocks, More Than Words and Autism Pro (Aspect NSW)—were funded throughStronger Together: A new direction for disability services in New South Wales, 2006–2016. All of the programs were designed in line with best practice principles, but they varied in terms of their philosophical orientation, delivery (mode, intensity, duration) and the intervention focus (child or parent).
The evaluation compared the outcomes for children across the four programs, as well as the cost-effectiveness of the programs, drawing on quantitative and qualitative data. The findings showed that children who took part in one or more of the four programs—within the context of receiving other services, supports and therapies through other sources, including the Helping Children with Autism Package—demonstrated improved skills, abilities and behaviours. Their parents’ knowledge, understanding and ability to cope also increased. The evaluation will inform ADHC’s decisions about the future direction of programs to support children with autism and their families.
This work builds on our other work in the autism area, in particular the evaluation of the Helping Children with Autism Package for the Commonwealth Department of Families, Community Services and Indigenous Affairs, and adds to the evidence base about effective supports for children with autism and their families.
ARTD is continuing to champion collaboration in government by sponsoring the NSW Institute of Public Administration Australia’s (IPAA) Collaboration Award again this year. The Collaboration Award recognises individuals who
– have established effective joint working arrangements that bring together colleagues from multiple teams, departments or other organisations
– can clearly identify and communicate the benefits of a collaborative approach
– have delivered improved, coordinated and efficient services.
Our Principal Consultant Sue Leahy presented the award to the 2013 winner, Dion Peita, at a special reception after IPAA’s Special Forum: Reforming to Create Value on Wednesday 8 May. Dion was recognised for his innovative work leading a collaboration between the Australian Museum and the Fairfield Office of Juvenile Justice NSW.
Concerned with the over-representation of Pacific youth in the NSW criminal justice system, in 2008, Dion began exploring the potential for the Australian Museum to help at-risk young people stay out of jail. Juvenile offenders of Pacific background now have the chance to access the Museum’s cultural collections as part of special programs. This has been achieved through collaboration between the Museum and organisations that provide services to at-risk young people. As a result of Dion’s vision, hundreds of young people have gained awareness of their cultural background through handling cultural artefacts and engaging with cultural experts, discussing issues around cultural identity and artistic expression.
Natalie Ironfield recently completed a three-month cadetship with ARTD through theCareerTrackers Indigenous Internship Program. Our consultants took her through the business of evaluation and research from project design to data collection and analysis. Michael Combs, Founder and CEO, thanked ARTD for sponsoring the internship and said, ‘It is vital that we continue to break down barriers and encourage reconciliation in Australia and I believe professional employment is one key element in realising this goal.’ Natalie has now returned to the Australian National University in Canberra, where she is completing her studies in International Relations. We hope to continue our relationship with Natalie in the future and wish her well with her studies this year.
AVID places Australian volunteers with organisations in 37 countries as part of the Australian Government’s foreign aid contribution. Growing volunteer numbers and a recent restructure of the AVID program in AusAID have presented an opportune time for an evaluation, and in 2012 the Office of Development Effectiveness (ODE), AusAID, commissioned ARTD to evaluate AVID to inform future directions. The evaluation, led by Andrew Hawkins, started with a review of international peer-reviewed and grey literature, recently published on the ODEwebsite. Because the scope of the AVID program is so large, ARTD’s work focused on how the program was working in three case study countries: the Solomon Islands, Vietnam and Cambodia. Andrew, with Emily Verstege, Ofir Thaler and ODE representatives, held more than 120 interviews with host organisations, volunteers and AusAID staff during field visits to the case study countries. Other methods included analyses of all articles about AVID published in the Australian media in 2011–12, mapping volunteer assignments in the case study countries against AusAID priorities, analysis of end-of-assignment reports filled out by host organisations and volunteers, and a survey of host organisations. It also drew on a survey of returned volunteers (2006 to 2011) carried out by ORIMA Research. ARTD’s evaluation report will be finalised by June 2013.
The Australian National Preventive Health Agency (ANPHA) has commissioned ARTD Consultants, in partnership with the Prevention Research Collaboration, University of Sydney, to undertake this large two-and–a-half year national evaluation. The National Partnership Agreement on Preventive Health (NPAPH) is an initiative of the Council of Australian Governments (COAG) and the largest ever national investment in preventive health—it aims to achieve better health for all Australians. The National Evaluation of the NPAPH will be conducted in several stages from January 2013 to June 2015. The evaluation will focus on assessing the benefits of the partnership approach for delivering preventive health initiatives across the country and review evidence of the impact of the NPAPH in all jurisdictions.