By Georgia Marett
Economic evaluation is increasingly becoming an integral part of evaluation. Much of our work includes some assessment of cost effectiveness or benefits relative to costs. As a behavioural economist, I have long been interested in economically evaluating programs which are trying to produce complex benefits (rather than just monetary savings). It is important to look at the whole picture of a program so that an economic evaluation can appropriately capture the value of all benefits.
With this in mind, I recently undertook a course run by the University of Melbourne and attended a webinar run by the Dementia Centre for Research Collaboration. These both focused on economic evaluation techniques that are used to evaluate health programs. However, I believe that these techniques can be applied to other sectors.
What types of economic evaluation are out there?
There are a variety of economic evaluation techniques.
In essence, the choice between the above options depends on the available resources (time and budget), data quality (cost and outcomes data) and the desired outcome—do you want to compare two programs, quantify the cost of your problem, or analyse the cost/benefit of one or more program/s?
Cost-benefit analysis is one of the more widely-known forms of economic analysis and is often used as part of an evaluation to translate the benefits of a program into monetary terms.
The costs and benefits of a program are compared to determine a net benefit (or cost) along with a benefit-cost ratio (BCR). The BCR is calculated as the ratio of the sum of a program’s benefits, relative to the cost of the program. The breakeven point for the BCR is 1—where a BCR between 0 and 1 represents a net cost, while a BCR above 1 represents a net benefit.
This kind of analysis is appropriate when all costs and benefits can be translated into monetary units (i.e. costs and benefits are denominated in dollars only, there are no externalities or hidden costs/ benefits). However, this is not always the case. What if the program generates benefits that are hard to monetise, such as increased school engagement or increased confidence when dealing with service systems, for example? In this case, a cost-benefit analysis can be more difficult or even impossible.
A cost-effectiveness analysis sidesteps the issue of monetising benefits by examining two programs (or one program and the business-as-usual condition) on one outcome measure. For example, you could compare two programs that aim to reduce heart disease on the outcome of ‘deaths prevented’. The formula for calculating an incremental cost-effectiveness ratio (ICER) is:
The ICER value can be used as a ‘decision rule’, whereby its value determines whether a program will be funded—if the ICER is too high then the program will be considered too expensive and not funded.
But what happens if your program has more than one aim? What if, for instance, your program could have an effect on physical and mental health? Unfortunately, a cost-effectiveness analysis is a narrow, uni-dimensional measure of success where disparate outcomes cannot be compared and multiple measures that would make up a general understanding of ‘quality of life’ are not able to be included. In addition, this type of analysis is only able to tell you the relative efficiency of the programs being compared, not their absolute efficiency. It will also not tell you if none of the programs are any good or consider the total cost of the programs (it only considers the relative costs of the two programs).
To include a measure of benefits that encompasses a wider range of possible positive outcomes, a cost-utility analysis uses what is known as a ‘preference-adjusted unit of consequence’. This value is derived from examining people’s preferences towards various benefits and non-benefits. It can be used to compare two programs that have vastly different outcomes.
In health economics, the most commonly used unit is the quality-adjusted life year (QALY) which considers both the quality and length of someone’s life when they are living with various conditions. QALYs are very interesting because of both the way they are developed and the way they are used. For more discussion on the advantages and disadvantages of QALYs see this paper.
In cost-utility analyses, QALYs (or some other measure) are used as the outcomes measure when calculating the ICER (as above in the cost-effectiveness analysis). In this case, the ICER is usually expressed as the incremental cost to gain an extra QALY. This gives a clear indication of the relative benefits of one program compared to another. It also allows for the comparison of a broader range of programs than cost-effectiveness analysis because the benefits do not have to be the same across the programs being compared.
Where does economic evaluation fit within a broader evaluation?
Economic analysis can add another dimension to evaluation as long as its limitations (and advantages) are adequately understood. Each program is different and economic analysis must fit in with the other aspects of an evaluation. The results must be treated sensitively and assessed alongside qualitative outcomes to get a full picture of the impact of a program. Acknowledging the strengths of the type of economic analysis being used and compensating for limitations by building-out other parts of the evaluation will ensure programs understand their full impact.
For more on how evaluators can assess value for money, check out Julian King’s Approach to Assessing Value for Money which sets out how economic analysis can be integrated into evaluation.