Environment, Natural Resources and Waste Management
Litter Prevention Program Evaluation
NSW Environment Protection Authority, Litter Prevention Unit (2021)
THE CHALLENGE
To support enhanced waste and recycling, the NSW Government was delivering Waste Less, Recycle More, a waste and recycling agenda for NSW that aimed to deliver economic, employment and environmental benefits for local communities and transform waste and recycling in NSW. As part of this, Waste Less, Recycle More provided $50 million to support the Litter Prevention Program in NSW, delivered through the Litter Prevention Unit (LPU).
In addition, in 2015 the NSW Government set a Premier’s Priority of reducing the volume of litter in NSW by 40% by 2020.
The Litter Prevention Program under Waste Less Recycle More was delivered over 9 years from 2012-2021, through a wide variety of activities and programs, encompassing: education and awareness campaigns, increasing infrastructure through council and community grants, increased enforcement, rewarding responsible behaviour, and monitoring waste.
THE OBJECTIVE
ARTD worked with the NSW EPA to conduct an evaluation of the Litter Prevention Program over the period 2012–2021, with the goal of providing evidence, insights, and recommendations to inform the design of future litter prevention programs and ensure alignment with the Waste and Sustainable Materials Strategy 2041 and the NSW Plastics Action Plan.
OUR APPROACH
ARTD worked with key NSW EPA stakeholders to develop a rubric approach. This enabled comparison of individual programs as well as collective assessment. Rubrics are used in evaluation studies to consistently apply evaluative criteria, particularly where multiple programs or elements are being evaluated alongside each other.
We applied this approach to look across six areas of program activity over the nine-year timeframe, and gathered evidence from documents, interviews and a partnership survey of stakeholders.
The result is a transparent and easily communicated assessment of program performance, incorporating both qualitative and quantitative data. Visualising the data in a rubric scorecard allows for an easy to understand overview of performance (see below).
IMPACT
Overall, we found that the program was successful in achieving outcomes and in addressing strategic objectives, and that it performed well on other areas with some opportunities for improvement. These were outlined in a set of detailed recommendations endorsed by the NSW EPA in full. Some of these are already being actioned.
The insights will be used to inform the development of future programs and to support the establishment of performance targets for these programs.
You can read more about our methods, rubrics and the results of the evaluation in the published report here.
Evaluation of the Environmental Water Management Program
Department of Planning, Industry and Environment – Environment, Energy and Science (Biodiversity and Conservation Division), 2021
THE CHALLENGE
Over the last 100 years, the river systems in NSW have been fundamentally changed through extensive regulation, such as dams, weirs, locks and channels. These changes have disrupted the natural patterns and volumes of flows in rivers, negatively affecting the environment. In NSW, the Department of Planning, Industry and Environment – Environment, Energy and Science manages water allocated to the environmental through the Environmental Water Management Program (EWMP).
The EWMP is a complex program operating in a contested policy arena that makes interventions into complex ecological systems. The objective is to ensure an allocation of water to support environmental outcomes. These policy, governance and ecological systems are made up of many interdependent elements that interact and change, often in nonlinear ways (including ‘tipping points’ and exponential growth, which is often irreversible). This is not the kind of problem that is solvable simply through research or experimentation—it is dynamic and must be managed adaptively.
Enabling the EWMP to be an innovative and adaptive ‘learning program’ depends on ensuring that there is strategic Monitoring and Evaluation, and coherent research and development that is integrated and applied.
THE OBJECTIVE
ARTD was commissioned to undertake an evaluation to assess whether the EWMP had made a difference to the health of rivers and wetlands in NSW. The focus was on the implementation of the recommendations from the 2006-13 evaluation, using a systems theory approach in order to evaluate the health of the system which supports the underlying environmental system which the program aims to maintain and improve.
OUR APPROACH
This evaluation was designed and implemented using a systems evaluation approach. This process commenced with a workshop with program staff to define the system and its component parts.
Elements important to this approach include:
- defining the EWMP as a system, with several subsystems
- determining how the EWMP’s subsystems are functioning
- evaluating systems that support the management of environmental water in NSW
- identifying opportunities for improving the EWMP.
Data sources included a document review (145 program documents), field visit to the Macquarie River region, a survey of stakeholders (n=125 respondents) and interviews with staff and stakeholders (n=43 interviews).
An iterative and interactive process using a combination of methods and data sources was used to determine and answer the key evaluation questions.
IMPACT
This evaluation provides guidance for the EWMP in conducting future improvements to the program. It may be used as a tool for program staff and stakeholders to disseminate learnings throughout the program and support evidence-based decision making that can adapt to changing environments.
Recommendations focussed around disseminating information and enhancing local community engagement; strengthening program capacity and systematising EWAG processes; and developing and implementing robust systems of gathering evidence.
Final Review of the National Partnership Agreement on Implementing Water Reform in the Murray-Darling Basin
Department of Agriculture, Water and the Environment, 2020
THE CHALLENGE
Managing Australia’s waterways to ensure healthy systems, sustainable agriculture and certainty for affected communities and water users is no mean feat, particularly when those waterways cross State and Territory borders. This is the challenge the National Partnership Agreement on Implementing Water Reform in the Murray-Darling Basin (the NPA) aims to address.
The institutional framework underpinning the NPA is complex due to the joint authority of the Commonwealth and States for water resource management, compounded by the scale and breadth of policy instruments implemented to achieve significant water reforms in the Murray-Darling Basin.
ARTD Consultants was engaged by the Department of Agriculture to assess how effective and efficient the NPA was in contributing to the implementation of Murray-Darling Basin water reforms.
THE OBJECTIVE
ARTD were tasked with conducting the final review of progress made by the Basin States in achieving the agreed outcomes of the NPA 12 months before the end of the NPA, and with making recommendations for improvements for current and future National Partnership Agreements.
OUR APPROACH
ARTD’s review looked at effectiveness of the NPA in contributing to water reforms, operations and efficiency of the NPA implementation arrangements, and possible enhancements.
Due to the complexity of the institutional framework and interplay of policies and agreements at a State and Federal level, we discerned effectiveness and possible enhancements to the NPA by looking at the interactions of parts, their performance together, and enablers and barriers, to provide recommendations for future agreements.
We reviewed 171 documents provided by the Department, predominantly relating to annual assessments.
We undertook stakeholder interviews with representatives from all Basin States, the Department of Prime Minister and Cabinet (PM&C), the Commonwealth Environmental Water Office (CEWO), the Murray-Darling Basin Authority (MDBA), The Treasury and the Department.
And we undertook analysis of wider policy and institutional settings that frame the agreement— including those that relate to water policy in the Murray-Darling Basin (MDB), and those that relate to financial relations in the federation, such as the Council on Federal Financial Relations’ (CFFR) short guide to reviewing National Partnerships.
IMPACT
The report identifies areas of improvement for current and future National Partnership Agreements, including on structure of future agreements, payment structures, partnership guidance and the role and transparency of intergovernmental arrangements.
The report can be read on our Published Work page
Evaluation of NSW EPA’s Bin Trim Program Round 3
NSW Environment Protection Authority, 2018–19
THE CHALLENGE
The amount of waste being sent to landfill sites across NSW (and Australia) is of growing concern. Each year, 6.5 million tonnes of business waste goes to landfill, and small and medium enterprises (SMEs) account for 99.7% of all businesses [10]. While recycling rates are increasing, so too is NSW and Australia’s population, which means that overall waste generation is increasing.
The NSW EPA’s Bin Trim Program aims to reduce the amount of waste sent to landfill. The Bin Trim Grants Program awards funding to organisations (Grantees) that provide waste experts (Assessors) to help individual businesses avoid, reduce, re-use and recycle materials. Assessors are responsible for targeting and recruiting businesses, with 400 or fewer full-time equivalent employees, across the state and providing support at no cost to businesses. The Bin Trim Rebates Program, which complements the Bin Trim Program, enables eligible businesses, small-scale recyclers and facility managers to receive rebates (between $1,000 – $50,000) to fund up to 50 per cent of the cost of small-scale recycling
equipment.
THE OBJECTIVE
ARTD’s evaluation and review of the Bin Trim Program covered both the Bin Trim Grants Program and the Bin Trim Rebates Program. Administered by the EPA, as part of the Waste Less, Recycle More initiative, the Bin Trim Program was established to work directly with businesses across NSW and assist them to identify and implement changes to improve their waste management practices. To date, more than 22,000 businesses have been engaged through three rounds of the Program.
OUR APPROACH
ARTD’s evaluation of Round 3 of the Grants and Rebates Program covered five areas of the program including governance, effectiveness of implementation, outcomes, value for money and impact of program changes.
The evaluation included:
- developing and presenting an updated program logic model including measures of success to determine the effectiveness of the Program updating the EPA Business Recycling Unit’s existing risk framework
- analysing program data provided by the EPA to enhance to provide an overall understanding of the Bin Trim Program
- analysing and evaluating the audit data captured in the EPA’s Bin Trim App and other reporting data
- conducting a desktop review into equivalent programs and schemes around Australia and internationally
- conducting semi-structured interviews with stakeholders including EPA staff, Grantees, Assessors and participating businesses
- conducting a Value for Money analysis following the 4Es definition.
IMPACT
ARTD’s work helped support the design, planning and implementation of Round 4 of the Bin Trim Program. The interim and final evaluation and risk assessment reports were used by the EPA to better understand Program issues and risks as well as key learnings, outcomes and achievements.
Related projects
Monitoring and Evaluation (M&E) Framework for the Community Litter Grant (CLG) program (EPA, 2019): ARTD was commissioned to develop a M&E Framework for Round 5 of the CLG program. Delivered by the EPA, as part of the Waste Less, Recycle More initiative, the program provides funding to community groups who want to do something about litter in their local environments. $50 million has been committed to 2021 to deliver on the Premier’s Priority to reduce little volume in NSW by 2020.
