Ruby Leahy Gatfield and Sue Leahy
The 2019 Their Futures Matter Conference, held in Sydney on 11 February, was an important reminder of the need to work with, not for, communities, when building evidence of outcomes.
Their Futures Matter (TFM), a landmark, whole-of-government reform designed to improve outcomes for vulnerable children and families in NSW, is committed to using evidence and evaluation to inform service design, outcomes measurement and investment. As put by TFM Executive Director, Gary Groves, ‘gone are the days that government funds something that sounds nice, without real rigour around outcomes. It’s nice to have 100 referrals walk in the door, but now I really want to know the outcomes for those referrals.’ The reform promises that evidence, monitoring and evaluation will drive continuous improvement across all areas of the system response and service delivery.
While we’re excited about this, it’s important to remember the many ways that different communities can understand, define and measure outcomes. The way that programs are designed and measured should be decided in close collaboration with community. While standardised measurement tools and RCTs have their place, a program or intervention that works for one cohort may not be appropriate for another.
This point was made best by Conference Chair and international affairs analyst, Stan Grant, who reflected honestly that as a child, he lived with many of the risk factors that predict poor life outcomes – living itinerantly, attending multiple schools, and having an incarcerated parent, not to mention the intergenerational trauma of the stolen generation. Today, a ‘Safety and Risk Assessment’ may have allocated him as a ‘high-risk’ child. Despite this, he stressed that ‘the very worst thing that could have happened’ would have been for the state to remove him from his family. He believed that the genuine love and care of his family outweighed the risk factors. Stan’s case, like many, underscore the need to recognise non-western views on safety (and other outcomes more broadly).
At the conference, ARTD also showcased our recent successful experience of working with, not for, community under TFM’s Aboriginal Evidence Building Partnership pilot. The pilot aim was to build an evidence base of promising programs and services that are improving outcomes for Aboriginal children and families. It did this by linking Aboriginal service providers with evidence building partners to work together to build providers’ data collection and evaluation capabilities.
Key to the success of our two pilot partnerships was our commitment to a partnership approach – establishing mutual trust and agreed ways of working early on, communicating regularly and openly, rescoping workplans to best meet current and future needs, and demonstrating an unwavering commitment to capacity-building and self-determination.
While the pilot required providers to embed standardised, validated tools to measure wellbeing outcomes, we worked closely with them to identify their additional data collection needs. We did this to ensure the data collected reflected what was most important to the service and its community and could be used on the ground to inform and improve program delivery. We are excited about the rollout of the pilot this year, and TFM’s commitment to a partnership-driven and capacity building approach to ensuring the service system meets the needs of Aboriginal children and families.
Conference keynotes also shone light on some of the other encouraging achievements of TFM to date. These included having more than 1,000 families engaged in new family preservation and restoration programs and establishing the first, human services cross-agency, longitudinal (+25 years) data set in NSW, providing large scale, de-identified matched data.
TFM Program Director & Investment Approach Lead, Campbell McArthur, said that beyond having what he deemed ‘the best dataset in Australia’, it’s the insights that the data can bring us that are truly exciting. The commitment to evidence means we are better able to compare and contrast the experiences of different cohorts, to better understand what works, for whom, in what circumstances. It also allows the system to identify population-level trends earlier and take evidence-based responses.
Attended by over 700 government and non-government representatives, we left the conference ready for the work ahead under TFM. Director for Children and Families in Scotland, Michael Chalmers, reminded us that ‘joining up services and creating change takes time and is difficult. But its important to keep your eyes on the prize: improving outcomes for our most vulnerable children and young people’.
By Ruby Leahy Gatfield
Empowerment has emerged as another buzz word. While it is often paid lip service, what does empowerment really mean? And how do we know if we’re achieving it?
Empowerment is a complex concept. It is both a process and an outcome that can be seen at the individual, organisational and structural levels, enabling positive growth and sustainable change. In the context of Aboriginal and Torres Strait Islander communities, who have experienced a history of systematic oppression, empowerment is critical for healing and improving overall wellbeing.
While there are many programs and interventions designed to empower Aboriginal people and communities, there is little quantitative evidence about their impact. Here’s where the GEM – the ‘Growth and Empowerment Measure’ – steps in.
On 1 February, Melissa Haswell from the Queensland University of Technology, facilitated GEM training at the National Centre of Indigenous Excellence. She explained that the GEM is the first validated, quantitative tool designed to measure empowerment in Aboriginal communities.
Based on extensive consultation, the tool measures different dimensions of empowerment as defined by Aboriginal people. It also aims to be a strengths-based and empowering process in and of itself, using scenarios as a way for people to trace their personal journey.
So, what does the GEM mean for evaluators? Firstly, it is a culturally-safe tool, developed by Aboriginal people, for Aboriginal people. When working with Aboriginal people in social research and evaluation, it is crucial that we use culturally-appropriate methods that:
The GEM offers just that. It also comes in the context of the growing development of other validated and culturally-appropriate measurement tools, such as Dr Tracey Westerman’s upcoming tool for measuring the cultural competencies of child protection staff.
More broadly, the GEM also enables us to quantifiably measure more holistic wellbeing outcomes, beyond discrete system indicators, such as increased school attendance or instances of out-of-home care. It recognises empowerment as fundamental to the overall health and wellbeing of individuals and communities, giving programs and services a better understanding of their impact. This is particularly important, given the lack of evidence about ‘what works’ for improving the wellbeing of Aboriginal people, families and communities.
To support Aboriginal organisations to embed the use of the GEM and other subjective measures of wellbeing, we are working with Their Futures Matter to develop tools and resources to support evidence building across the Aboriginal service sector. Stay tuned for more…
 Haswell, M.R., Kavanah, D., Tsey, K., Reilly, L., Cadet-James, Y., Laliberte, A., Wilson, A., & Doran, C. (2010). Psychometric validation of the Growth and Empowerment Measure (GEM) applied with Indigenous Australians. Aust N Z J Psychiatry. 44(9): 791–9. Available at: https://www.ncbi.nlm.nih.gov/pubmed/20815665
 Productivity Commission. (2016). Overcoming Indigenous Disadvantage: Key Indicators 2016. Canberra: Productivity Commission.
 Stewart, M. & Dean, A. (2017). Evaluating the outcomes of programs for Indigenous families and communities. Family Matters. No. 99, pp.56–65.
By Jack Rutherford
When my friends and family ask me what my job is, I say something along the lines of, “I work at a public policy consulting firm. We mostly evaluate government policies and programs, and we gather our data mostly through surveys and interviews.” Knowing that I majored in biology, they tend to follow this by questioning whether I want to work in a field they see as so far removed from biology and science.
While the work I do to at ARTD feels meaningful and fulfilling, I’ve been left wondering… will I be able to access the parts of biology that I love while working in public policy?
When I saw that the 2018 ACSPRI Social Science Methodology Conference was discussing how to integrate social and biological research, I jumped at the opportunity to attend.
I attended the first day of talks at the University of Sydney on 12 December. The conference featured diverse expertise from a range of local and international backgrounds, including Naomi Priest from the Australian National University and Melissa Wake from the Murdoch Children’s Research Institute, Tarani Chandola from the University of Manchester and Michelle Kelly-Irving from the Université Paul Sabatier in France. Their talks highlighted ways biological concepts and methodologies have and can contribute to social research, with a focus on the use of biomarkers in social studies.
To many, biomarkers are a new concept. Put simply, a biomarker is an objective measure of biological processes. For example, increased blood pressure can be used as a biomarker for increased levels of stress.
Chandola explained that the benefits of biomarkers include reducing measurement errors that can arise in surveys and being able to tell more holistic stories than those derived purely from self-reported data. In the example of stress, participants may underreport how stressed they feel or report not feeling stressed at all, while their blood pressure and other biomarkers suggest otherwise.
Because biomarkers can be used to determine the effects of the social environment on humans, they can be used in evaluation to provide enlightening data on complex social issues. For example, Kelly-Irving spoke about how adverse childhood experiences impact the physiology of the adult. According to her research, adverse childhood experiences tend to become more frequent with increasing social disadvantage. These experiences have neurodevelopmental impacts, which can affect health mediators (such as one’s likelihood of smoking, their BMI etc.) and social mediators (such as educational attainment), which all later influence mortality.
Priest also presented findings showing how different types of racial discrimination affect children’s health by increasing their BMI, waist circumference and blood pressure.
If biomarkers can be used as indicators of the effects of the social environment, and a public policy or program aims to affect social change, then biomarkers can be used to measure the effectiveness of said policy/ program. For example, if we were evaluating a program designed to combat adverse childhood experiences, we could compare the biomarkers of those who took part in the program with those who did not.
While it may seem complicated, biomarkers have the potential to enable social researchers and evaluators to draw clearer pathways between cause and effect. This is particularly useful in an increasingly complex social environment.
Despite their benefits, biomarkers don’t come without their own set of challenges. For one, sampling participants’ biology needs a heavy consideration of ethics regarding consent, risk, and data and sample security. Asking participants for samples of biomarkers may result in increased opt-outs and reduced sample sizes, therefore impacting the predictive power of significance testing.
Sampling methods for certain biomarkers may also be time and resource intensive, particularly when training is involved.
It’s also important that this sort of research isn’t used to exacerbate inequalities any further, i.e. that results are not presented in a way that provides certain groups with perceived biological evidence for their prejudices.
Using biomarkers also requires careful consideration of the conceptual framework. Before incorporating biomarkers into a project, evaluators must be certain they are valid indicators of the particular aspect of the social environment. Sometimes one biomarker is not descriptive enough. Indeed, Kelly-Irving found that a model comprised of many biomarkers was better than a model using just one in determining the impacts of adverse childhood effects.
The conference suggested that as our collective understanding of biological concepts, methodologies, and data increases, it will become easier to integrate the biological and the social. Social research and evaluation will benefit from a rich data source, with potential to support our understanding.
Generation Victoria, an ambitious project directed by speaker Melissa Wake, aims to gather biological and social data from all children born in Victoria between 2021 and 2022 with the goal of addressing social epidemics like school failure, obesity and mental health. Far-reaching projects like this become possible by merging the biological and the social.
On a more personal level, I find it exhilarating that I might be able to marry my passion of biology with the work I do at ARTD. Moving forward, I aim to look for and shape opportunities to integrate this thinking into our work, and I encourage you to do the same.
In using our own biology as measures of the effects of the social environment, the future of evaluation is, quite literally, within us all!