Data quality assessment to support initiatives focussed on those most-at-risk youth

Client: Ministry of Business, Innovation and Employment

The context

High rates of young people not in employment, education or training (NEET) have become a concern internationally. Research suggests disengaged youth are more likely to experience social exclusion, increased psychological distress, and poor quality of life,[1] impacting on individuals’ ability to lead fulfilling lives with purpose, balance and meaning to them. Beyond wellbeing, the costs[2] related to NEETs are significant,[3] for instance, the estimated per capita cost of each NEET rangatahi in New Zealand was $26,847 over the 2012-2015 time period. Investing in NEET rangatahi could benefit employers by enabling them to access prospective employees to match their needs, while enabling young people to find jobs, train for better jobs, improve their incomes, support their whanau and engage and contribute to their communities.

He Poutama Rangatahi/Youth Employment Pathways (HPR) Strategy was launched in January 2018 as an investment in people (not programmes). The Strategy focuses on a sub-group of young people, in particular those 15-to 24-year-olds currently NEET who face the greatest challenges in gaining sustained employment.

The HPR monitoring and evaluation plan focused on addressing the Strategy’s accountability requirements over time. Knowing that achieving and evaluating[5] the goal of sustained employment would take time, the focus was to first monitor quality and progress across the individual initiatives and rangatahi over the initial 36 months. A monitoring approach and reporting system was established to estimate progress systematically across HPR providers according to the expected achievements – providers first target and enrol those most at risk, then engage and support these individuals, and ultimately help them make progress on the pathway towards sustained employment. Specific measures and reporting tools were created to help MBIE and providers delivering the initiatives monitor this progress along a theoretical pathway.

The project

To guide effective decision making, government required data that represents progress accurately, demonstrating useful and up-to-date information about HPR providers and rangatahi. Providers collect and report statistics quarterly to MBIE. Data quality was assessed according to five quality dimensions – completeness, consistency, timeliness, validity, and uniqueness – for each dataset provided to MBIE from the different providers. The strengths and weaknesses of the different data, as well as an overall quality score, were summarised so Ministry officials could understand the quality of data and ultimately how they could use these data. The quality review was also used to help understand if other activities were required to improve data quality and reporting among the providers.

[1] Stain, H., et al. (2019). Study protocol: a randomised controlled trial of a telephone delivered social wellbeing and engaged living (SWEL) psychological intervention for disengaged youth. BMC Psychiatry, 19(1), 136.

[2] The 2016 OECD report (Society at a Glance) defines NEET costs as the gross labour income NEET could command if they were employed, measured as the gross labour cost (including social security contributions).

[3] Estimated loss of income totalled between 0.9% – 1.5% of the OECD GDP. OECD (2016). Society at a Glance 2016: OECD Social Indicators. Paris: OECD Publishing.

[4] McGirr, M. (2019). Not just about NEETs: A rapid review od evidence on what works for youth at risk of limited employment. Ministry of Education.

[5] The impact evaluation was designed to estimate what was achieved for local youth and also whether this achievement would have occurred without HPR support. The difference the activities made would represent the value-add of HPR, and use Statistics New Zealand’s Integrated Data Infrastructure for this purpose.

Image credit: “Work hard… always” by leonfhl is licensed under CC BY-NC-ND 2.0

Evaluation of Oranga Mahi

Client: Ministry of Social Development

The context

A number of programmes support disabled people and people with health conditions to prepare for, find and stay in work. Such services may be tailored to the individual’s needs and goals. The Oranga Mahi programme was established in 2016 to deliver a set of cross-agency prototypes in partnership with several District Health Boards (DHBs) and Primary Healthcare Organisations (PHOs) throughout New Zealand. The Oranga Mahi programme aims to support people with disabilities and/or health conditions to improve their wellbeing and enter sustainable employment. The purpose of the Oranga Mahi evaluation is to inform strategic decisions about the programme.

The evaluation

The evaluation plans to adopt a quasi-experimental approach, notably propensity score matching using the information in the integrated data infrastructure (IDI).

Synthesis of evidence supporting an inclusive development approach

Client: Ministry of Foreign Affairs and Trade

The context

The Second United Nations Decade for the Eradication of Poverty (2008–2017) highlighted the need to address social exclusion and called for more inclusive approaches to overcome poverty in its multiple dimensions. This was reiterated in the Sustainable Development Goals (SDGs) and Agenda 2030’s  commitment to ‘leave no one behind’. Inclusion is now globally recognised as an essential pre-requisite for effective aid. In the Pacific Reset programme, women’s economic empowerment and engagement of youth are the priorities.

Despite the prominence of inclusive development as a concept, ambiguity still exists around the term.Also, evidence of its overall impact on development effectiveness and the more qualitative aspects of wellbeing is sparse.

The review

Standard of Proof conducted a systematic literature review of research studies and evaluations of development interventions with an inclusive approach. We focused on reviewing the evidence on traditionally excluded groups, such as women, people with disabilities, young people, rural communities, that aligned with the Ministry of Foreign Affairs and Trade’s focus areas relevant to development interventions. The report provided the Ministry with a working definition of inclusive development, discussed the different reasons for considering inclusive development as a component of development activities, and collated evidence on the impact and effectiveness of this approach.

We’re pleased to have supported the Ministry of Foreign Affairs and Trade in deepening their understanding of the benefits to an inclusive approach to development by synthesising the evidence behind it.

Measures for He Poutama Rangatahi initiative

Client: Ministry of Business, Innovation and Employment

Psychometric analysis ensures young people can get the support they need

“Standard of Proof helped a government agency develop a questionnaire which would help organisations understand the strengths, needs and progress relevant to each young people sustaining employment over time. The factors that could help these organisations identify needs and progress towards their long term goal – sustained employment – included aspects of individual’s foundational needs, employability and opportunity.

The intended use of the questionnaire highlights the importance of providing accurate information. If the data misrepresent a young person’s ability to sustain work, inappropriate decisions might be made that could have negative effects on the person. For example, the young person may be placed into work too early, before they are able to sustain a job, which may have a lasting negative impact on their future employment opportunities. Inaccurate information may be worse than having no information at all.

Standard of Proof, with the support of the University of Western Australia, applied the Rasch unidimensional measurement model to determine if the organisations and young people could use and interpret the data as intended, and if government organisations could trust the results. Of even greater importance at this early stage, the technique produced diagnostic information necessary to improve the questionnaire, making it easier for young people to respond to and providing information that everyone can rely on in order to inform the ‘right’ next steps.

Evaluation of the Northland District Health Board rheumatic fever prevention programmes

Client: Northland District Health Board

Evidence identifying the most efficient and effective delivery approach

The Northland District Health Board (DHB) is one of 11 district health boards delivering a locally relevant response to address rheumatic fever rates in the community. The Northland rheumatic fever prevention programme involves several Māori providers delivering targeted throat swabbing services in their communities. The DHB wanted to understand the delivery model that provided the greatest value (best outcomes) for money (lowest cost), and Standard of Proof was contracted to implement the evaluation to inform future funding decisions. 

Standard of Proof pulled together data from various sources, and worked with providers and the DHB to understand how different activities contributed to their achievements as well as the cost and efficiency of activities.

As a result of the evidence and evaluation process, the DHB and providers are now in an informed position to support programme providers achieve success in the most cost effective way.