Ever left a workshop feeling like you didn’t get a chance to contribute and you wouldn’t remember if someone asked you what it was all about the next day? Bored with lectures, post-it notes and role plays?
At ARTD we’re serious about creative facilitation and policy co-design. So we’re using the LEGO Serious Play method, which is based on research that shows tactile learning produces more meaningful understanding, opens up possibilities and enables dialogue.
We put the method to the test with our own staff – asking what would your relationship with your coach look like in your ideal world? After hours of play, laughter and serious discussion, the sceptics were converted.
We found out that staff want a coach who:
We never would have learned all this by white-boarding ideas or asking staff to fill in a survey. Playing really did help to create an honest dialogue and identify new ideas.
Our evaluation report on Weave Youth & Community Services was launched at a community celebration in Waterloo on Tuesday 30 August 2016. The evaluation was part of a larger Stories of Lived Experience project, which also included photographic portraits and a short documentary to capture the faces and stories of Weave’s clients and community.
The evaluation was designed to understand the features of Weave’s support models that make a difference to clients, focusing intergenerational connections and continuity of care, and to use clients’ expert knowledge of their life experiences to contribute to the mental health sector’s development and good practice. The report celebrates the stories of Weave’s clients and community, and makes recommendations that Weave will use to inform their future directions. It also identifies lessons for the wider community sector, especially for services working with Aboriginal people and communities.
The report draws on interviews with over 50 Weave clients, as well as consultations with Weave staff and analysis of service data. To support a culturally sensitive approach that builds capacity in the local community, we worked with Weave and three local Aboriginal organisations—Tribal Warrior Association, South Sydney Aboriginal Corporation Resources Centre, and the Aboriginal Medical Service in Redfern—to deliver the project under ethics approval from the Aboriginal Health & Medical Research Council.
ARTD undertook this project as part of our corporate social responsibility to strengthen the non-government sector. The project was also supported by a grant that Weave received from Inner West Partners in Recovery.
Our 2015 report on measuring energy savings from NSW Office of Environment and Heritage (OEH) 2007-13 energy efficiency programs has been published on their revamped website. The report was part of a four-year evaluation support project, and follows from the 2012 report. The new report syntheses savings from activities undertaken in 2013 and 2014 to address the lack of reliable evidence of energy savings from energy efficiency programs. It highlights the substantial achievement OEH has made in moving from savings based on engineering estimates (deemed savings) to verified energy savings based on ‘before-and-after’ analysis—consistent with international best practice measurement. Methods included billing data analysis and Measurement and Verification (M&V). The measurements demonstrated that the interventions in each of the programs achieved real and substantial energy savings and reductions in electricity bills at sites studied. The capacity to generalise these findings to all sites in each sector varied with the samples of the measurement studies.
Local Government NSW and the NSW Office of Environment and Heritage (OEH) have published the results from ARTD’s survey of councils’ progress and needs in adapting to climate change. The online survey, conducted in June 2015, received 186 responses covering 74 per cent of all NSW councils. Local Government NSW presented the findings at the National Climate Adaptation conference on 7 July 2016. The survey, which followed two surveys conducted in 2010, showed councils’ capacity to assess and plan for climate change impact had increased. In the 2015 survey, over two-thirds of respondents (69%) reported having experienced impacts from climate change in their Local Government Area. Councils welcome any kind of support to prepare and respond to climate change impacts, with increased interest in technical, high-quality information compared to 2010, and growing interest in tools, such as costing of adaptation actions.
Check out Director Andrew Hawkins' latest article ‘Realist evaluation and randomised controlled trials for testing program theory in complex social systems’ in Evaluation.
Realist approaches to scientific evaluation tend to be strong on theory and explanation, but lack adequate tests or means of validating theory. Randomised controlled trials (RCT) are often seen as the gold standard for research, but Andrew argues, while they are useful in some fields of science, they are less useful in complex social systems.
Andrew’s article focuses on the potential for randomisation and experimentation to provide evidence for transfactual (i.e. reusable or portable) context–mechanism–outcome configurations (CMOs) in complex adaptive systems. Realist RCTs are considered but rejected; instead a form of propensity score matching is proposed for testing realist program theory, estimating the effect size of a purported CMO, and generating scientific knowledge for developing more effective interventions into complex social systems.
Andrew Hawkins (ARTD) will present results from the Ticket To Work evaluation at the Disability at Work conference with Michelle Wakefield the Ticket To Work program manager from National Disability Services (NDS). Ticket to Work is a national program that aims to improve the transition to employment of young people with disability. The study aimed to compare the employment and social participation outcomes of young people with intellectual disability who participated in Ticket to Work Program to outcomes that may be expected for young people with intellectual disability. The analysis used data from the Household Income and Labour Dynamics Survey (HILDA) and Survey of Disability Ageing and Carers (SDAC) surveys. While the sample sizes were too small to draw conclusions at this stage, the indicative data is very promising and provides a foundation for a more comprehensive analysis by the end of 2016. The study was part of a broader evaluation that also looked at issues for the strength and integration of the networks that deliver Ticket to Work. The conference is focused on developing employment options for people with disability, and is intended for employment service providers and professionals interested in promoting opportunities for people with disability. It will be held at the National Convention in Canberra on 30–31 May 2016. For more information see the conference program.
In November 2013, ARTD was engaged by NSW Treasury to evaluate Resilient Families, an intensive support service delivered to families in greater Sydney by The Benevolent Society as part of the NSW Government’s Social Benefit Bond scheme.
The purpose of the evaluation is to assess the implementation and outcomes of the service over its first three years of operation from 2013 and assess the appropriateness of the measures place for the purpose of bond payments. Throughout the three-year study the service has moved to a more established stage of implementation, providing a flexible service, responsive to client needs. ARTD’s recommendations have focused on further development of the service to optimise outcomes for families. The Preliminary and Mid-term evaluation reports can now be viewed on the NSW Department of Premier and Cabinet’s website.