Impact evaluation of the New Zealand Healthy School Lunch programme

Client: Ministry of Education

The context

The New Zealand healthy school lunches programme is an initiative by the New Zealand Government to reduce food insecurity by providing access to free, nutritious lunches for all learners at participating schools and kura daily.  The whole-school programme targets schools and kura with the highest concentrations of learners from disadvantaged backgrounds. The programme is intended to ensure not only that nutritious food is available to learners every day, but that the food (quality, quantity and variety) promotes learners’ consumption. it was further expected that the provision of sufficient quantitities of nutritious food will lead to a series of expected outcomes, including improved school attendance as well as improved diet and nutrition for learners, and therefore greater food security.

The evaluation

The evaluation required rigorous data, including generalisable results comparing “paired” schools (those with and without the programme) to assess impact of the programme for accountability purposes.

The evaluation adopted a real-world approach, making use of the rolling enrolments to integrate a stepped-wise design and assess the net effects of the programme on learners. A clustered sample design at the class-level was used to track food availability and consumption, satiety and wellbeing among young people aged 5 to 13 –years-old. Further, propensity score matching was used on both student attendance data.

Evaluation of Oranga Mahi

Client: Ministry of Social Development

The context

A number of programmes support disabled people and people with health conditions to prepare for, find and stay in work. Such services may be tailored to the individual’s needs and goals. The Oranga Mahi programme was established in 2016 to deliver a set of cross-agency prototypes in partnership with several District Health Boards (DHBs) and Primary Healthcare Organisations (PHOs) throughout New Zealand. The Oranga Mahi programme aims to support people with disabilities and/or health conditions to improve their wellbeing and enter sustainable employment. The purpose of the Oranga Mahi evaluation is to inform strategic decisions about the programme.

The evaluation

The evaluation plans to adopt a quasi-experimental approach, notably propensity score matching using the information in the integrated data infrastructure (IDI).

Evaluation of the Alert programme pilot

The context

The Alert Program® (ALERT) was developed by occupational therapists, Williams and Shellenberger, in the United States. ALERT promotes a shared language to communicate arousal levels in children, and provides tools and strategies to strengthen their self-regulation skills. The Ministries of Health and Education worked together with two primary schools in the Wellington region to pilot ALERT during Term’s 3 and 4 of the 2019 school year. The pilot adapted ALERT as a school-wide approach and provided school staff with training and support. As staff learn and embed the skills, students are expected to improve their ability to self-regulate and both teachers and students will improve their overall wellbeing.

The evaluation

Standard of Proof evaluated the pilot ALERT programme and assessed its effectiveness in the New Zealand context. We used a mixed method approach, pulling together and then synthesising different types of data from various sources. We also used the pilot as an opportunity to test evaluation measures, data collection approaches and the overall systems required for an eventual impact evaluation while increasing the evidence base over time.

We thoroughly enjoyed working closely with the project working group and the pilot schools, and are proud to have delivered purposeful evidence that is informing decisions on future investments in the programme.

Formative evaluation of the New Zealand Healthy School Lunches programme

The context

Food insecurity is a real issue for many children in New Zealand. Published results from the New Zealand health survey in 2015/16 (Ministry of Health, 2019) showed that approximately 19% of children in New Zealand lived in households with moderate to severe food insecurity (i.e. they lacked access to sufficient amounts of nutritionally adequate and safe foods). Due to structural and systemic problems such as poverty and inequality, children from Māori and Pacific households are disproportionately affected by food insecurity. Further, with increasing numbers of families/whānau experiencing unemployment and receiving job seeker support since March 2020, it is expected that food insecurity will become even more pronounced in New Zealand.

Kā Ora, Kā Ako | the healthy school lunches programme is an initiative by the New Zealand Government to reduce food insecurity by providing access to free, nutritious lunches for all learners at participating schools and kura daily. The whole-school programme targets schools and kura with the highest concentrations of learners from disadvantaged backgrounds. The programme is intended to ensure not only that nutritious food is available to learners every day, but that the food (quality, quantity and variety) promotes learners’ consumption  by being appealing.  It was expected that this will lead to improved school attendance as well as improved diet and nutrition for learners , and therefore greater food security and wellbeing.

The evaluation

The evaluation was set out in two phases to accommodate different information needs as the delivery model matured over time. During the first phase, the evaluation focused on building an awareness of the process, while answering questions about efficacy and cost-effective of the delivery models while delivery was underway and the design was being refined.  A series of case studies were used, focussing on the contextual elements relevant to programme delivery at each school while allowing comparability across different delivery models.

Evidence was captured and discussed with key stakeholders throughout the early implementation of the programme, using evidence to inform decision making and support the overall success of the programme.

Evaluation of the Northland District Health Board rheumatic fever prevention programmes

Client: Northland District Health Board

Evidence identifying the most efficient and effective delivery approach

The Northland District Health Board (DHB) is one of 11 district health boards delivering a locally relevant response to address rheumatic fever rates in the community. The Northland rheumatic fever prevention programme involves several Māori providers delivering targeted throat swabbing services in their communities. The DHB wanted to understand the delivery model that provided the greatest value (best outcomes) for money (lowest cost), and Standard of Proof was contracted to implement the evaluation to inform future funding decisions. 

Standard of Proof pulled together data from various sources, and worked with providers and the DHB to understand how different activities contributed to their achievements as well as the cost and efficiency of activities.

As a result of the evidence and evaluation process, the DHB and providers are now in an informed position to support programme providers achieve success in the most cost effective way.