Monitoring the Incredible Years Autism programme

Client: Ministry of Education

The context

Incredible Years Autism programmes are a recent extension of the basic Incredible Years programmes, with the specific focus on children on the autism spectrum and the caregivers and teachers who support them. IYA is one of several social wellbeing initiatives focused on key adults (i.e. caregivers and teachers/kaiako) in the lives of children aged 2 to 5 years who are either diagnosed with or showing symptoms of autism. Because of these initiatives, it is expected to see children with enhanced social-emotional competence and communication skills, and becoming increasingly engaged. We also expect to see teachers, parents and family/whānau feel more supported and confident, helping improve engagement among their children. IYA programmes are provided through regional NGOs, the Ministry or a combination of both. Each IYAP or IYAT programme is delivered by two specially trained and IY-accredited group leaders.

The monitoring

One of the challenges was in measuring these outcomes and in large enough numbers to detect change while we protect and support those that we seek information from. The agreed approach required a measurement system embedded within delivery whereby providers to collect and share relevant information from (consenting) participants and their children. It was essential that any information shared complies with the Ministry’s information management requirements alongside New Zealand’s Privacy Act regulations. With this in mind, we developed the relevant forms alongside an approach for sharing information between the Ministry, programme providers and participants.

An online data management system was developed to address gaps and improve use of data relevant to programme providers, programme participants and the Ministry of Education. Data is being systematically collected and collated through the web app, resulting in remarkably high levels of consent and completion rates across a longitudinal design, and quality monitoring data being used to support the overall success of the programme.

Impact evaluation of the New Zealand Healthy School Lunch programme

Client: Ministry of Education

The context

The New Zealand healthy school lunches programme is an initiative by the New Zealand Government to reduce food insecurity by providing access to free, nutritious lunches for all learners at participating schools and kura daily.  The whole-school programme targets schools and kura with the highest concentrations of learners from disadvantaged backgrounds. The programme is intended to ensure not only that nutritious food is available to learners every day, but that the food (quality, quantity and variety) promotes learners’ consumption. it was further expected that the provision of sufficient quantitities of nutritious food will lead to a series of expected outcomes, including improved school attendance as well as improved diet and nutrition for learners, and therefore greater food security.

The evaluation

The evaluation required rigorous data, including generalisable results comparing “paired” schools (those with and without the programme) to assess impact of the programme for accountability purposes.

The evaluation adopted a real-world approach, making use of the rolling enrolments to integrate a stepped-wise design and assess the net effects of the programme on learners. A clustered sample design at the class-level was used to track food availability and consumption, satiety and wellbeing among young people aged 5 to 13 –years-old. Further, propensity score matching was used on both student attendance data.

Evaluation of the Alert programme pilot

The context

The Alert Program® (ALERT) was developed by occupational therapists, Williams and Shellenberger, in the United States. ALERT promotes a shared language to communicate arousal levels in children, and provides tools and strategies to strengthen their self-regulation skills. The Ministries of Health and Education worked together with two primary schools in the Wellington region to pilot ALERT during Term’s 3 and 4 of the 2019 school year. The pilot adapted ALERT as a school-wide approach and provided school staff with training and support. As staff learn and embed the skills, students are expected to improve their ability to self-regulate and both teachers and students will improve their overall wellbeing.

The evaluation

Standard of Proof evaluated the pilot ALERT programme and assessed its effectiveness in the New Zealand context. We used a mixed method approach, pulling together and then synthesising different types of data from various sources. We also used the pilot as an opportunity to test evaluation measures, data collection approaches and the overall systems required for an eventual impact evaluation while increasing the evidence base over time.

We thoroughly enjoyed working closely with the project working group and the pilot schools, and are proud to have delivered purposeful evidence that is informing decisions on future investments in the programme.

Formative evaluation of the New Zealand Healthy School Lunches programme

The context

Food insecurity is a real issue for many children in New Zealand. Published results from the New Zealand health survey in 2015/16 (Ministry of Health, 2019) showed that approximately 19% of children in New Zealand lived in households with moderate to severe food insecurity (i.e. they lacked access to sufficient amounts of nutritionally adequate and safe foods). Due to structural and systemic problems such as poverty and inequality, children from Māori and Pacific households are disproportionately affected by food insecurity. Further, with increasing numbers of families/whānau experiencing unemployment and receiving job seeker support since March 2020, it is expected that food insecurity will become even more pronounced in New Zealand.

Kā Ora, Kā Ako | the healthy school lunches programme is an initiative by the New Zealand Government to reduce food insecurity by providing access to free, nutritious lunches for all learners at participating schools and kura daily. The whole-school programme targets schools and kura with the highest concentrations of learners from disadvantaged backgrounds. The programme is intended to ensure not only that nutritious food is available to learners every day, but that the food (quality, quantity and variety) promotes learners’ consumption  by being appealing.  It was expected that this will lead to improved school attendance as well as improved diet and nutrition for learners , and therefore greater food security and wellbeing.

The evaluation

The evaluation was set out in two phases to accommodate different information needs as the delivery model matured over time. During the first phase, the evaluation focused on building an awareness of the process, while answering questions about efficacy and cost-effective of the delivery models while delivery was underway and the design was being refined.  A series of case studies were used, focussing on the contextual elements relevant to programme delivery at each school while allowing comparability across different delivery models.

Evidence was captured and discussed with key stakeholders throughout the early implementation of the programme, using evidence to inform decision making and support the overall success of the programme.

Measures for He Poutama Rangatahi initiative

Client: Ministry of Business, Innovation and Employment

Psychometric analysis ensures young people can get the support they need

“Standard of Proof helped a government agency develop a questionnaire which would help organisations understand the strengths, needs and progress relevant to each young people sustaining employment over time. The factors that could help these organisations identify needs and progress towards their long term goal – sustained employment – included aspects of individual’s foundational needs, employability and opportunity.

The intended use of the questionnaire highlights the importance of providing accurate information. If the data misrepresent a young person’s ability to sustain work, inappropriate decisions might be made that could have negative effects on the person. For example, the young person may be placed into work too early, before they are able to sustain a job, which may have a lasting negative impact on their future employment opportunities. Inaccurate information may be worse than having no information at all.

Standard of Proof, with the support of the University of Western Australia, applied the Rasch unidimensional measurement model to determine if the organisations and young people could use and interpret the data as intended, and if government organisations could trust the results. Of even greater importance at this early stage, the technique produced diagnostic information necessary to improve the questionnaire, making it easier for young people to respond to and providing information that everyone can rely on in order to inform the ‘right’ next steps.