Improving the Health Promoting Schools self-assessment tool

Client: Ministry of Health

Evaluation is about finding out what works well and what doesn’t. An effective evaluation will help you celebrate your success and help you improve your programme. This begins with collecting the right data and providing accurate information – and this means having the right measurement tool.

In collaboration with our partners Cognition Education, Standard of Proof tested whether the Health Promoting Schools (HPS) school self-assessment tool was providing accurate indications of school health and wellbeing so that the school workforce can trust the results and make decisions about the next steps in their delivery approach.

We applied the Rasch measurement model to the evidence collected from the self-assessment tool. This technique provided a robust reflection of the tool, and provided useful modifications to the self-assessment – making sure it provided reliable evidence.

The analysis examined:

  • Does the assessment tool have clear questions and response categories so it can provide reliable measures of all groups?
  • Are the range of questions and response categories both easy enough to accommodate schools with low ability, and hard enough to accommodate schools with high ability?
  • Are the scaled response categories able to provide accurate measures of growth over time?
  • Does the overall assessment tool provides a single measure of school health and wellbeing, which can reliably test any relationship with student achievement?
  • The results were presented to a national hui with the HPS workforce, and the findings are being used by the Working Group to further improve the value of the self-assessment tool for school communities.

Measures for He Poutama Rangatahi initiative

Client: Ministry of Business, Innovation and Employment

Psychometric analysis ensures young people can get the support they need

“Standard of Proof helped a government agency develop a questionnaire which would help organisations understand the strengths, needs and progress relevant to each young people sustaining employment over time. The factors that could help these organisations identify needs and progress towards their long term goal – sustained employment – included aspects of individual’s foundational needs, employability and opportunity.

The intended use of the questionnaire highlights the importance of providing accurate information. If the data misrepresent a young person’s ability to sustain work, inappropriate decisions might be made that could have negative effects on the person. For example, the young person may be placed into work too early, before they are able to sustain a job, which may have a lasting negative impact on their future employment opportunities. Inaccurate information may be worse than having no information at all.

Standard of Proof, with the support of the University of Western Australia, applied the Rasch unidimensional measurement model to determine if the organisations and young people could use and interpret the data as intended, and if government organisations could trust the results. Of even greater importance at this early stage, the technique produced diagnostic information necessary to improve the questionnaire, making it easier for young people to respond to and providing information that everyone can rely on in order to inform the ‘right’ next steps.

Good measurement ensures an efficient approach and accurate data

Client: Ministry of Education

Good measurement ensures an efficient approach and accurate data

Standard of Proof recently provided support for a national randomised trial in New Zealand. Interesting in this case, the evaluation randomly allocated “clusters” of people (rather than individuals) into control and treatment groups. Why is this important? Because your sample size needs to be significantly larger in such evaluation designs.

Let’s take an example. A class includes a group of students (ie, a “cluster”). If your evaluation and activity are being delivered across multiple schools, how can you randomise students easily and efficiently? One approach (more) common in clinical trials is to embed a clustered design element in your evaluation; in this example, we would randomise by class – our “cluster” – rather than by student.  Although this may be a simpler allocation approach, it may not be as efficient for data collection and analysis.

Clustered data have a pre-existing group structure. Using this class example again, we would reasonably assume that children’s achievement scores are in some way influenced by their cluster (i.e., the teacher, their class, the characteristics of having enrolled in this specific school, etc). This “relatedness” in the cluster will have a significant effect on your sample size requirements. If measuring progress at the student level, what may have been a study requiring 200 participants may now require 2000.

Such considerations are useful to know up-front. Given the right balance, you can certainly design an efficient and practical randomised trial.”

Implementation plans support collaboration

Client: Ministry of Education

“Implementation plans support collaboration”

Evaluation frameworks typically set out the focus and scope of the evaluation (the ‘what’), including the approach and methods, indicators and rubrics (the ‘how’), and a high-level timeline for the deliverables (the ‘when’). Frameworks also often suggest ways of working, and one often-heard suggestion is that the evaluation team will or should work “collaboratively” with key stakeholders.  But what does “collaboration” mean in practice, and how will it affect the evaluation?

Experience has shown that individuals interpret collaboration differently. Nevertheless, researchers have identified a number of key factors that influence successful collaboration, such as leadership, communication (Hogue, Parkins, Clark, Bergstrum & Slinski, 1995; Keith, Perkins, Zhou, Clifford, Gilmore & Townsend, 1993), participation and membership (Keith et al,1993, Borden, 1997). Instruments are also available to help guide successful collaboration (Borden & Perkins, 1999), but how such factors are embedded into the evaluation process is not often made explicit.

Evaluation plans sometimes omit details about collaboration, such as defining the persons (the ‘who’) and capabilities (the ‘why’) that will take part in the different tasks (the ‘what’).  However, making these elements of collaboration explicit provides an opportunity to support your evaluation standards.

Standard of Proof recently tested the feasibility of an “impossible” group randomised trial. While testing the overall system for feasibility, our work focussed on developing the structures and processes to ensure the highest standard of evidence to inform decision-making. The approach required collaboration, and as one element of our efforts, we built an adaptive management process around the Joint Committee Standards, ensuring people (the ‘who’) were engaged in relevant activities (the ‘what’) to support the utility, accuracy, feasibility, propriety and accountability of the evaluation (the ‘why’). Pulling together the expertise and the explicit approach to collaboration made the “impossible” randomised trial possible.

Beyond the ‘who’, ‘what’ and ‘why’, it is important to define the time required to realise the collaborative effort (the ‘how much’). When delivering large-scale or challenging evaluations, such “hidden” costs can break an evaluation, often resulting in low participation and engagement, and poor quality data. By making the ‘how much’ elements of collaboration explicit, you will increase your opportunity to successfully undertake a challenging large-scale evaluation with quality evidence.”