Toolkit to support better evaluation in NSW

Aricle Image for Toolkit to support better evaluation in NSW

August 2013

The NSW Government launched its new Evaluation Toolkit to support NSW government agencies to undertake program evaluation projects on 21 August 2013 at the Why Evaluation Matters: Basing Decisions on Evidence Forum in Sydney. ARTD’s Chris Milne and Sue Leahy worked with Professor Patricia Rogers of RMIT University in developing the toolkit for NSW public sector managers in line with the NSW Government Evaluation Framework. It takes managers who are responsible for a program evaluation through seven steps—from initial conception, through commissioning and managing, to disseminating and supporting use of the findings. The Toolkit will be refined over time based on user feedback.

High-profile evaluation of behaviour change trials

August 2013

ARTD is now evaluating the behaviour change component of the NSW Government’s Home Power Saving Program (HPSP). This program, which started in July 2010, was established to help 200,000 low-income households reduce their power usage and save on energy bills by June 2014. It includes three main components provided at no cost to participants: 1) a home power assessment, 2) a tailored action plan that identifies ways the household can save power, and 3) a kit of energy efficient items (including a standby saver power board, light bulbs and other small items and a water-saving showerhead if needed). The $63 million program is managed by the Office of Environment and Heritage and delivered across the state by about 100 contacted energy experts.

Based on recommendations from ARTD’s interim evaluation of the program in 2012, OEH introduced a new version of the program in June 2013 to test behaviour change intervention methods that could support greater outcomes for participants. OEH designed the behaviour change trials with BehaviourWorks at Monash University and the NSW Department of Premier and Cabinet’s  Behavioural Insights Team (seconded from the UK Cabinet Office). The trials are using three key triggers identified in the behaviour change literature— commitment, social norms and loss aversion—to try to improve participants’ implementation of tips in their power savings action plans. From June 2013, each new participant is being randomly allocated to one of the three trial groups—which receive different types of follow-up support aligned with one of the behaviour change triggers—or a control group—which receives the program as usual.

ARTD’s evaluation of the trial, being led by Andrew Hawkins and Florent Gomez-Bonnet, is using a randomised control trial (RCT) to test the impact of each type of intervention on behaviour change and energy savings (measured through analysis of external billing data), combined with a mixed-methods design, based on the Theory of Planned Behaviour, to try to understand observed behaviour. Together, these approaches will provide robust evidence about the value of the program and evidence to support refinements to further promote energy efficient behaviour among low-income households.

You can find out more about the work of the UK’s Behavioural Insights Team, or Nudge Unit,here. For more detailed information on the approach, check out, Richard Thaler and Cass Sunstein’s book Nudge or Daniel Kahneman’s Thinking, Fast and Slow.