Evaluation of Oranga Mahi

Client: Ministry of Social Development

The context

A number of programmes support disabled people and people with health conditions to prepare for, find and stay in work. Such services may be tailored to the individual’s needs and goals. The Oranga Mahi programme was established in 2016 to deliver a set of cross-agency prototypes in partnership with several District Health Boards (DHBs) and Primary Healthcare Organisations (PHOs) throughout New Zealand. The Oranga Mahi programme aims to support people with disabilities and/or health conditions to improve their wellbeing and enter sustainable employment. The purpose of the Oranga Mahi evaluation is to inform strategic decisions about the programme.

The evaluation

The evaluation plans to adopt a quasi-experimental approach, notably propensity score matching using the information in the integrated data infrastructure (IDI).

Formative evaluation of the New Zealand Healthy School Lunches programme

The context

Food insecurity is a real issue for many children in New Zealand. Published results from the New Zealand health survey in 2015/16 (Ministry of Health, 2019) showed that approximately 19% of children in New Zealand lived in households with moderate to severe food insecurity (i.e. they lacked access to sufficient amounts of nutritionally adequate and safe foods). Due to structural and systemic problems such as poverty and inequality, children from Māori and Pacific households are disproportionately affected by food insecurity. Further, with increasing numbers of families/whānau experiencing unemployment and receiving job seeker support since March 2020, it is expected that food insecurity will become even more pronounced in New Zealand.

Kā Ora, Kā Ako | the healthy school lunches programme is an initiative by the New Zealand Government to reduce food insecurity by providing access to free, nutritious lunches for all learners at participating schools and kura daily. The whole-school programme targets schools and kura with the highest concentrations of learners from disadvantaged backgrounds. The programme is intended to ensure not only that nutritious food is available to learners every day, but that the food (quality, quantity and variety) promotes learners’ consumption  by being appealing.  It was expected that this will lead to improved school attendance as well as improved diet and nutrition for learners , and therefore greater food security and wellbeing.

The evaluation

The evaluation was set out in two phases to accommodate different information needs as the delivery model matured over time. During the first phase, the evaluation focused on building an awareness of the process, while answering questions about efficacy and cost-effective of the delivery models while delivery was underway and the design was being refined.  A series of case studies were used, focussing on the contextual elements relevant to programme delivery at each school while allowing comparability across different delivery models.

Evidence was captured and discussed with key stakeholders throughout the early implementation of the programme, using evidence to inform decision making and support the overall success of the programme.

Improving the Health Promoting Schools self-assessment tool

Client: Ministry of Health

Evaluation is about finding out what works well and what doesn’t. An effective evaluation will help you celebrate your success and help you improve your programme. This begins with collecting the right data and providing accurate information – and this means having the right measurement tool.

In collaboration with our partners Cognition Education, Standard of Proof tested whether the Health Promoting Schools (HPS) school self-assessment tool was providing accurate indications of school health and wellbeing so that the school workforce can trust the results and make decisions about the next steps in their delivery approach.

We applied the Rasch measurement model to the evidence collected from the self-assessment tool. This technique provided a robust reflection of the tool, and provided useful modifications to the self-assessment – making sure it provided reliable evidence.

The analysis examined:

  • Does the assessment tool have clear questions and response categories so it can provide reliable measures of all groups?
  • Are the range of questions and response categories both easy enough to accommodate schools with low ability, and hard enough to accommodate schools with high ability?
  • Are the scaled response categories able to provide accurate measures of growth over time?
  • Does the overall assessment tool provides a single measure of school health and wellbeing, which can reliably test any relationship with student achievement?
  • The results were presented to a national hui with the HPS workforce, and the findings are being used by the Working Group to further improve the value of the self-assessment tool for school communities.

Measures for He Poutama Rangatahi initiative

Client: Ministry of Business, Innovation and Employment

Psychometric analysis ensures young people can get the support they need

“Standard of Proof helped a government agency develop a questionnaire which would help organisations understand the strengths, needs and progress relevant to each young people sustaining employment over time. The factors that could help these organisations identify needs and progress towards their long term goal – sustained employment – included aspects of individual’s foundational needs, employability and opportunity.

The intended use of the questionnaire highlights the importance of providing accurate information. If the data misrepresent a young person’s ability to sustain work, inappropriate decisions might be made that could have negative effects on the person. For example, the young person may be placed into work too early, before they are able to sustain a job, which may have a lasting negative impact on their future employment opportunities. Inaccurate information may be worse than having no information at all.

Standard of Proof, with the support of the University of Western Australia, applied the Rasch unidimensional measurement model to determine if the organisations and young people could use and interpret the data as intended, and if government organisations could trust the results. Of even greater importance at this early stage, the technique produced diagnostic information necessary to improve the questionnaire, making it easier for young people to respond to and providing information that everyone can rely on in order to inform the ‘right’ next steps.

Evaluation of the Northland District Health Board rheumatic fever prevention programmes

Client: Northland District Health Board

Evidence identifying the most efficient and effective delivery approach

The Northland District Health Board (DHB) is one of 11 district health boards delivering a locally relevant response to address rheumatic fever rates in the community. The Northland rheumatic fever prevention programme involves several Māori providers delivering targeted throat swabbing services in their communities. The DHB wanted to understand the delivery model that provided the greatest value (best outcomes) for money (lowest cost), and Standard of Proof was contracted to implement the evaluation to inform future funding decisions. 

Standard of Proof pulled together data from various sources, and worked with providers and the DHB to understand how different activities contributed to their achievements as well as the cost and efficiency of activities.

As a result of the evidence and evaluation process, the DHB and providers are now in an informed position to support programme providers achieve success in the most cost effective way.