PAX Good Behavior Game - A Cluster-randomized Controlled Evaluation and Qualitative Analysis of Effects on Work Environment, Learning, Mental Health and Health Economic Aspects of Implementing a Classroom-based Intervention
PAX Good Behavior Game - A Cluster-randomized Controlled Evaluation and Qualitative Analysis of Effects on Work Environment, Learning, Mental Health and Health Economic Aspects of Implementing a Classroom-based Intervention
Two important challenges that primary school in Sweden is confronted by are increasing levels of mental illness among children and adolescents and increasing rates of stress related symptoms and sick leave associated with mental illness in primary school teachers. Primary school is an important arena to address these issues.
PAX is implemented as a part of the regular classroom activities and gives the teacher tools to promote behaviors that are constructive and learning oriented. The programme has been shown to improve student health and performance both short- and long-term. The teachers also receive tools to lessen stressors in their working environment. A Swedish uncontrolled pilot study showed increased well-being in children and reduced teacher stress as well as improved work related well-being at post-intervention.
The aim of this project is to study the effects of PAX on student well-being and teacher stress. In a cluster-randomized design, 84 teachers from 28 schools (clusters) will be randomized to either implement PAX, or to an active control group that receives the same amount of support but with different content. The PAX group will get a two-day education in PAX at the start of the term, followed by an implementation-term. The measurements include self-assessment, blind observer assessments, objectively measuring sound levels, data on sick leave and qualitative interviews.
Baseline measurements are performed before implementation and primary end point is one year after. Costs and savings related to PAX will be measured and put in relation to the quality-adjusted life years of students and teachers. Long-term effects on student performance, health and participation in society will be evaluated in a follow-up project.
Design
A cluster-randomized controlled trial will be conducted randomizing approximately 28 schools (clusters) with a total of 84 teachers/classrooms into either receiving training and supervision in PAX GBG or receiving equally much training and supervision for implementing one of the control interventions.
Teachers in both arms will be implementing and using the intervention they were randomized to during the first year and then start implementing also the other intervention, while continuing using the first. When the baseline measure occurs at the end of grade 2 and the initial intervention is implemented for pupils during grade 3, the pupils in these classes will only receive one intervention since they move up to middle school where none of the methods are used. The teacher will instead start over with new pupils in grade 1 and use both methods with them. The sub-group of pupils that receives only one intervention will thus preserve the randomized controlled design also for long-term follow-ups.
Procedure
Ideally, half of the classrooms will be enrolled to start the intervention/control intervention during the spring semester and half during the autumn semester. The purpose of this is to control for the occasional effect of the point in time during the school year, and calendar year, when the intervention is introduced.
Credible control interventions requiring the same amount of training and supervision will be used, to control for unspecific effects of implementing a new method. It also makes it possible to promote the study to schools and teachers as a way to implement and compare several useful methods, although for different purposes, which helps to avoid too high expectation or placebo effects for PAX and to minimize possible disappointment and nocebo effects related to having to wait one year before starting with PAX. The school will have the possibility to choose one of the two control interventions - a math application, 'Count On Me!', or Collegial learning community. This serves two purposes; (1) the possibility to choose will further reduce the risk of nocebo or being dissatisfied if not randomized to PAX and (2) it will enable more schools to participate if one of the control interventions would be seen as impractical to implement. Since neither fo the control interventions are deemed to have the same specific effects as PAX, at least not to the same extent, they will both act as controls for unspecific effects.
A total of seven measures will be administered in relation to pupils, teachers and the classroom during the two years, with the baseline (Measurement 0) at the end of the semester before the implementation starts (Semester 0) and the primary endpoint (Measurement 4) one year afterwards. To reduce the burden for teachers and pupils, the full range of outcome measures will be conducted only on two occasions (Measurement 0 and Measurement 4) and otherwise a reduced set of measures will be used for pupils and teachers (For measures on classroom level, the full set of measures will always be applied). The reduced set of self-rating measures for teachers will include work related stress (primary outcome), classroom management, general stress, sleep problems, work ability and observed negative/positive effects of the method being implemented. For pupils, the reduced set will contain questions on well-being (primary measure). Teachers will also measure the well-being for each pupil at each measurement point.
Following time frames will apply to the measurement points:
Measurement 0 (M0) - semester before the first implementation semester, a 4-weeks administration window ending two weeks before the end of the semester.
Measurement 1 (M1) - the first implementation semester, a 6-weeks administration window starting two weeks after the start of the semester (and preferably two weeks after the completion of the intervention training).
Measurement 2 (M2) - the first implementation semester, a 4-weeks administration window ending two weeks before the end of the semester).
Measurement 3 (M3) - the second implementation semester, a 4-weeks administration window starting two weeks after the start of the semester).
Measurement 4 (M4) - primary endpoint, the second implementation semester, a 4-weeks administration window ending two weeks before the end of the semester.
Measurement 5 (M5) - the third implementation semester, a 4-weeks administration window ending two weeks before the end of the semester.
Measurement 6 (M6) - the fourth implementation semester, a 4-weeks administration window ending two weeks before the end of the semester.
Participants and recruitment
Schools, teachers, classrooms and other school staff
The cluster-randomization will be performed on the school-level, and will in turn decide which teachers and pupils that are included. The schools will be recruited via an established network of PAX GBG trainers in Sweden and the research groups other connections with school-related networks and the recruitment will be country wide. The schools will be recruited from at least two different Regions and at least four different municipalities (kommuner) to increase the generalizability of the results. In total, around 28 schools will be included, depending on the number of classrooms participating from each school.
A total of 84 classes, with an expected total of up to 100 teachers (as some classes have more than one teacher) will be included in the project.
Other school staff is recruited only to participate in the qualitative evaluation of expectations and experiences of the interventions. The selection of these will be pragmatic and theoretically lead to reaching a saturation of qualitative data, with a maximum of four extra staff from each school.
Pupils and their participation in interventions and data collection
In total, 2200 pupils will be included, with approximately 26 pupils included from each classroom. All the pupils in a class related to the included teachers/classrooms will participate in the intervention since those are all implemented in the regular operation/teaching at the school and focused on the teachers' way of teaching and managing the class and the classroom and is not possible to exclude specific pupils from. Pupils can however be included in the data collection in three different ways:
Randomization
To ensure that the socioeconomic status (SES) of the schools do not differ too much, each school will be classified as being low, middle, or high SES and then, when there are enough schools in a cohort, randomized consecutively according to one of three corresponding block-randomized lists with block sizes varying randomly between 2, 4 and 6. The randomization will be handled by an independent third party, the Karolinska Trial Alliance, to ensure the researchers are not able to assess the lists or otherwise predict randomization results.
Interventions
The implementation procedure will be similar for each intervention, with smaller adjustments to fit specific needs of each method. Each starts with brief initial written information about the interventions, followed by one to two days of training/education for teachers. This is then followed by regular supervision face-to-face or via video/telephone depending on practical aspects and possible restrictions due to the Covid-19 pandemic. Supervision always includes some form of observation and monitoring of the use of the method by the supervisor, as a source to further inform the supervision. A total of six supervision sessions are planned: 4 sessions during the first implementation semester (held during the 1st to 4rth quarter of the semester respectively, starting with 2 individual sessions, followed by 1 group session and then by 1 individual session) and 2 sessions during the second implementation semester (held during the 1st and 2nd half of the semester respectively, 1 group session followed by 1 individual session). Teachers in all groups will also be able to receive briefer supervision on demand or additional supervision sessions initiated by the supervisor if deemed necessary.
PAX Good Behavior Game
PAX Good Behavior Game (GBG) is based on its precursor programme Good Behavior Game (GBG) that trains teachers to use principles of social learning in order to maximize the childrens' task-oriented and prosocial behaviors and minimize the occurrence of disruptive and off-task behaviors. PAX GBG is packaged into ten specific tools that the teachers are trained to introduce in their classroom management practice. The program introduces verbal and visual cues that facilitate prosocial and attentive behaviors One of the core elements of the program is the PAX-game, meaning that the children - while focusing on ordinary school tasks like math - get to play a game in teams where each team tries to make as few classroom rules infractions as possible during a certain period of time. The teams that succeed in achieving predetermined goals get a reward (often a fun and desirable activity) that serves as a reinforcer of prosocial and task-oriented behaviors.
Supervision is performed individually with each teacher (4 sessions) and in group (2 sessions) and includes that the PAX-trainer observes a lecture given by the teacher before each supervision session.
Count On Me! (Control condition)
Count On Me! is an adaptive application teaching fluency in addition and subtraction facts (i.e., math facts), equality, compose and decompose numbers and a few more basic mathematical skills taught in the first grades of schooling. Count on me! is set in adventure game where the child has an avatar that travels thru different worlds/platforms by using their mathematical skills. The child starts of in the home of the avatar, in a house like environment, where the child discovers a magical map that brings the child into a different world called Numberia. During the adventures in Numberia the child learns that it has certain skills, called the mathemagical powers, that will be used to liberate Numberia from the evil Prince Claw that has captured all the beautiful things (e.g., the birds that used to sing in the forest).
Count on me! was developed within a behavior analytic framework using a nonlinear programming process originally developed by Tiemann and Markle and later slightly revised by other researchers while creating Headsprout Early Reading and Headsprout Reading Comprehension. Instructional design elements from Tiemann and Markle, explicit instruction and the Morningside model of generative instruction were used to build the exercises. The math exercises contain at least one of three recurrent phases used in all training: the modeling, guide, and fluency phase. Modeling of new skills involves animations and hand-drawn examples along with audial instructions and requires frequent untimed student responding. Guided practice of newly modeled skills helps students achieve high levels of accuracy without timing. Finally, fluency building is aimed at developing proficiency with the skill at hand using timing. The guide and fluency phases comprise gamified elements. The application has been scientifically evaluated among Swedish low-performing elementary school pupils having shown significant improvements in math skills among the children having used the application with the largest improvement among the children with lower cognitive skills.
Prior to the implementation start, the teachers' session focus on the functionality and content of the application and the way it corresponds to the math study plan for the class they're teaching. The training session will include the methods to introduce the application in the classroom, move forward and overcome common challenges. The teachers will be encouraged to start using the application in the classroom directly after the training session. The teachers will be recommended to use the application at least 4 times each week, approximately 15 minutes each time.
Supervision will be held individually and in groups and will partly be based on the supervisors monitoring reports on how much and how the application is used.
Collegial learning community (Control condition)
Muijs and colleagues has defined five cyclical steps for professional learning for teachers. The first step is to assess the skill or knowledge gaps for students, using various data sources (for example testing, grades, surveys, student interviews). The second step is to reflect upon what skills the teacher needs to acquire in order to help the students to meet the goals. The third step is to deepen the professional knowledge and skills. The fourth step is engaging the students in the new practice, and in the fifth step to evaluate the impact the new practice has had on student outcomes.
According to Timperley and other researchers, a feasible format for this kind of learning is a Collegial learning community, where teachers take help from their peers in each of these steps in a structured way. For example, way of deepening tacit or procedural knowledge (in step two) is to visit a colleague during class and make observations on their classroom practice.
Timperley points out the importance of having a professional supervisor for maintaining the structure of the learning process. One common pitfall that the supervisor helps the teacher avoid is shifting focus from professional learning and instead ending up in entrenching current teacher practices, instead of learning new skills.
Collegial learning communities can focus on practically any kind of teacher skills. The current intervention provide material to focus on any of the following: 1) formative assessment, 2) how to formulate and follow up on homework in order to engage students 3) How to teach fact knowledge in order to facilitate higher reasoning with students. These themes are chosen to mimimize the overlap with the content of PAX GBG but still be highly relevant for teachers.
The intervention will be based on a web resource, and 6 supervision sessions with assignments in between. Supervision and observation will be combined in that the supervisor initially observes when the teachers perform a group supervision with their colleagues, and then gives immediate feedback to the whole group. This is followed by a further discussion of the method and brief, 10 to 15 minutes long individual sessions with each teacher.
Measures
Background data and the effectiveness and implementation outcomes of the intervention will be assessed with a range of different measures described below. The timing of the measures is described in table 1 above, and in some cases commented more on below.
Extra background measures at baseline (M0) for schools, classrooms and teachers The measures described here are not the only one used as background data, but those specifically measured only at baseline.
The SES for each school district measured in average income for parents and average level of education for parents (based on the data from Statistics Sweden) Number of pupils, teachers and extra resource persons in each class The pupil's need for additional help and extra resources, as well as presence of a neurodevelopmental diagnosis.
Extra COPSOQ-III questionnaire subscales Quantitative Demands, Work Pace, Influence at Work, Predictability, Social Support from Supervisor, Social Support from Colleagues, Sense of Community at Work.
Extra items from TALIS, the questions about employment and about other teachers at school.
Effectiveness measures
The effects of implementing PAX will be assessed on school level, teacher level, classroom level and student level.
Classroom-focused measures
Classroom-observations by a trained observer will be conducted during two or three lessons during the same or conceding days, for a minimum time of 20 minutes for each class. To increase standardization of the measure, only lessons where the focus is on theoretical subjects will be included. The observations will be based on an observational schedule that has already been tested in the PAX pilot trial and will be slightly adapted in a separate project prior to being implemented in the current trial. The schedule will focus on direct observation of the following aspects: 1) pupils being on-task/off-task (primary outcome), 2) number of pupils disruptive behaviors, 3) time for transition between different tasks and 4) number of positive/negative teacher-pupil interactions. The observations will be conducted by trained observers and will never identify single pupils, and hence produce anonymous data. However, the gender of the pupil will be recorded for assessment (1) - (4) to provide opportunity to conduct gender analyses.
After each observed lesson, both observers and teachers will rate the observed lecture using a questionnaire that is being developed together with the observational schedule focusing on overall ratings of aspects like severity of disruptive behaviors in the classroom, level of noise in the classroom, how focused the class was during the lecture etc. Again, this data will be rated at classroom-level, with no possibility to identify individual students.
During one week connected to the fixed measurement occasions, data on noise levels will be collected in each classroom. The noise levels will be assessed by a measuring device that will be placed in the classroom without disturbing classroom environment. The device will measure and store very short snippets of sound with random intervals of on average three seconds, making it impossible to identify source and content of the sound activity (e. g. hear a conversation) but being enough to calculate the decibel level at different ranges of frequency. Thus, this data is also anonymous.
School attendance and the need of additional personnel in the class will be measured at the class level.
The topic of the intervention's effect on the classroom, both positive and negative, will be covered by the qualitative interviews with the teachers and other school staff (described in the section for Implementation measures). Also, four free-text questions to the teacher at M3, M4 and M6 will cover this, prompting them to describe positive and negative effects of the intervention in relation to (a) themselves and (b) to the pupils, since the last measurement point. The pupils will also be asked about their attitude to PAX/control intervention.
Measures of teachers' health and work
A number of self-rated measures will be collected from teachers using a secure platform used for conducting online psychological treatments and requiring two-step authentication. The measures will include self-rated health-related measures, self-rated work-related data and registry-based data on health-care and prescription drugs consumption.
Pupil-focused measures
Self-rated questionnaires be answered by the pupils, with the aid of trained research assistants who will make sure that the students have understood the questions and the response alternatives and minimize the risk of them disturbing or influencing each other. Different ways of administering the questionnaire battery are available (administration via paper-and-pen or tablet in the whole class, in smaller groups, or individually, with the administrator giving instructions, sometimes also using a pre-recorded instruction) to be able to adapt to what would be most feasible for a specific classroom. The pupils will rate their health and wellbeing. The teachers will fill out the dysregulations scale (5 items) of the Strengths and Difficulties Questionnaire. Registry-based data will be used to analyse the pupils' health-care and prescription drugs consumption as well as the parents' need to take stay home from work to take care of a sick child. The children's academic attainment in mathematics and the Swedish language will be measured using standardized tests.
Implementation measures
The pre-conditions for the implementation process will be assessed using both quantitative and qualitative data collection. Semi-structured qualitative interviews will be conducted with randomly selected 25-50% of all the participating teachers. The goal of the interviews is to shed light on aspects that will not be captured by the standardized instruments, both in terms of the intervention's effectiveness and quality of implementation. The interview guide is being created as a part of a smaller pilot project. The interviews will be conducted by phone or video and will take approximately 20 minutes on average. The interviews will be conducted three times with each selected teacher: immediately after the PAX-training, at M2 and M4 (see the specification of the measurement points above). The interviews will be recorded, transcribed and analyzed using qualitative content analysis.
Expectations and experience of the implemented intervention
In order to study expectations, experiences and outcomes of the implemented intervention, the teachers and other school staff on several occasions will fill out the Normalisation Measure Development Questionnaire (NoMAD). The same dimensions will also be studied by by conducting interviews and collecting objective data, such as those on long-term use of the intervention according to the classroom observations, the proportion of the eligible teachers using the intervention in each school etc.
Intervention Fidelity
The teachers' fidelity to the method will be measured by direct observation conducted by the supervisor and by using a self-rated fidelity instrument for teachers to fill out.
Adverse events
Information on adverse events will be collected starting at M3 and onwards as open-ended questions that the teachers fill out in the platform. Questions on negative effects will also be included in the qualitative interviews.
Statistical Analyses
Statistical power of 80% for the pupils' psychological symptoms (the between-group effect size estimated from the pilot study equalling d=0.6) can be reached with a cluster-randomized design including 28 schools, 3 classes per school and 25 pupils in each class (a total of 2100 pupils), the expected data loss of 20% and the clustering effects known from similar studier in school environments of rho2=.1 (school level) and rho3=.25 (teacher level). Since the number of pupils can exceed 25, we expect the total number of included pupils to be 2200.
Primary endpoint for outcomes will be at the one-year follow up (M4), before the teachers starts to use also the method they were not randomised to initially. The measures focusing on the teachers' and students' health, well-being and work/study environment, as well as classroom-based measures that are collected repeatedly will be analyzed with a hierarchal general linear model with a maximum of three nested levels (school, classroom/teacher, and pupil). The difference in the slopes of change over time between the PAX group and the control interventions will be tested for each outcome variable. A set of mixed effect models will be performed to detect factors (background factors and implementation-related factors) that influence the outcomes of the implementation (fidelity, acceptability, appropriateness etc.). Mixed effect models will also be used to detect whether implementation-related variables (e g readiness for change, fidelity, acceptability etc) affect the main outcomes of the trial (e g teachers' and students' health, well-being and work/study environment).
The anonymous data from pupils will be used in two ways: (1) for missing data analyses by comparing the levels and changes in the anonymous data to the identifiable data and (2) to perform sensitivity analyses where all data are analysed on time and intervention group level only, i.e. ignoring information on school, classroom and pupil, even when those exists. The statistical methods will thus have less power, but on the other hand data from more pupils can be utilized and since the amount of missing data will be lower than in the main analyses, the risk for biases due to non-random attrition will be lower. For the anonymous data, if data from fewer than 10 pupils at one time-point and intervention are collected, the data will be discarded and deleted completely, in order to keep a safe margin from the possibility of in some way indirectly connect the data to a pupil.
Cost effectiveness
The teachers' mental health measured with GHQ-12 will be recalculated to quality adjusted life-years (QALY:s). The students health-related quality of life, measured with CHU9D, will be used to calculate QALY:s. The costs for the implementation (e g time for training and supervision), will be calculated together with the possible savings related to reduced sick leave absence for the teachers, as well as the need for additional personnel in the schools and reduced need of student health resources. The total QALY:s during the follow-up period will be studied using area under the curve method. The difference health outcome between the intervention group and the control condition will be analyzed using regression models. The costs and savings will be compared to the gains expressed in QALY:s, both for the teachers and the students. The results will be presented as a cost-effectiveness quota and illustrated as a cost-effectiveness plan, that will show the uncertainty in the results when it comes to the costs and health-effects for the intervention and the control group.
ON SCHOOL LEVEL:
Inclusion Criteria:
Exclusion Criteria:
ON TEACHER LEVEL:
Inclusion criteria:
Exclusion criteria:
viktor.kaldo@lnu.se0046707373859
ekaterina.ivanova@ki.se0046765807416