Assessment of the usability/ acceptability of the Little Journey app with children aged between 3-12 years old with a diagnosis of autism spectrum disorder (ASD) and experience of attending hospital for a procedure, and their parent (or carer), using a test scenario and semi-structured interview.

Introduction

500,000 children in the UK undergo outpatient surgery annually, with 1 in 10 children having a neurodiversity diagnosis. Children often find visits to hospital scary, however neurodiverse chil-dren are twice as likely to experience anxiety and poorer outcomes. Patient care does not cur-rently focus on preparing children for their hospital attendance, instead focussing on distraction, restraints, and sedatives during their hospital visits.

Little Journey's app intends to psychologically prepare children for their hospital visit, and sub-sequently aims to reduce premedication rates, on-the-day cancellations and pre-operative anxi-ety. If effective, this will bring cost-benefits for the NHS. The app provides virtual tours of the rooms, including surgical theatres, ward areas and playrooms, and age-tailored animations to describe the hospital-specific clinical pathway and procedures. The app also provides relaxation activities and games for children and information articles, fasting guidelines and checklists for their caregivers.

Little Journey are developing an additional module named 'A Little About Me'. This module has several quizzes for children to complete, including questions about how their communication preferences, what's important to them, what they like and how they feel about being in hospi-tal. This is designed to support quicker relationship building between clinical staff and patients and enable support to be provided for neurodiverse children and their parents (or carers). It's important to have real end users provide feedback during the development of this module to support its optimisation for children with ASD.

This study aims to assess the usability of the Little Journey app with 7 children and their parents (or carers) aged between 3-12 years with a diagnosis of ASD and experience of attending hospital for a procedure. Children with an ASD diagnosis will be recruited from Birmingham Children's Hospital's and their responding to recruitment flyers circulated via UHB and BCH staff channels and placed within the vicinity of Birmingham Children's Hospital. Interested and eligible indi-viduals will be invited to participate in a 60-minute session. During the session participants will use the Little Journey app, this will be recorded to film the child and parent's (or carer's) inter-action with the app. The participants will then be asked to provide verbal feedback via a semi-structured interview.

Study Rationale and Design Justification

Formative human factors/ usability testing is completed during the development of technologies to assess user interactions, identify user errors and highlight areas of improvement before wide-spread use and adoption. This study will be conducted in line with the international standard IEC/TR 62366-2:2016 (guidance on the application of usability engineering to medical devices); although the app is not a medical device the principles of usability testing are transferable to this study.

This study researching the Little Journey app is being completed to support the software's con-tinued co-development and improvement, with particular focus on children with ASD and their parents (or carers). The study will provide opportunity for children with ASD and their parents (or carers), within a safe and controlled environment, to interact with the user interface of the Little Journey app and provide qualitative feedback collected via a semi-structured interview. This will be completed in a single visit and will take approximately 1 hour.

Management of the usability/ acceptability study

The usability study conduct, data management, data analysis and study reporting will be under-taken by the NIHR HRC-DDR Team, part of University Hospitals Birmingham NHS Foundation Trust. A pseudo-anonymised report will be provided to Little Journey Ltd, the study sponsor and developer of the Little Journey app.

Participant Identification and Recruitment

The study participants will represent the intended users of the app. 7 children with a diagnosis of ASD aged between 3-12- years old with previous experience of attending hospital for a procedure will be recruited. Their parent (or carer) will be present throughout the study involvement.

There will be two methods of recruiting individuals. Firstly, staff from BCH will screen potentially eligible individuals during their pre-admission checks (children aged between 3 and 12 years old with ASD). Potentially eligible individuals will be sent both the parent PIS and child PIS by a BCH staff member which will provide an overview of the study background, their involvement, and the benefits and risks of taking part. If interest-ed in participating, individuals will be asked to email or call the NIHR HRC-DDR Team who will then answer any questions about participating, confirm eligibility and then schedule a study visit. These contact details are included in the parent PIS.

Secondly, REC-approved recruitment flyers will be placed around the Birmingham Children's Hospital site, surrounding areas, and circulated via staff channels at BCH and UHB. Advertisements will signpost individuals to emailing or calling the NIHR HRC-DDR Team. When a potential participant/ family member/ carer contacts the NIHR HRC-DDR team an electronic parent PIS and child PIS will be provided. A member of the NIHR HRC-DDR team will answer any questions about participating, confirm eligibility and then schedule a study visit.

A limited database of name, date they contacted the NIHR HRC-DDR, means of contact which the potential participant provided, and date the PIS was provided will be kept at NIHR HRC-DDR only for the purposes of contacting those individuals to provide further information. If participants respond that they would not like to participate in the study, their name and contact de-tails will be deleted.

Once an interested participant's parent has confirmed that they have read the PIS and are eligible to participate, a study visit will be scheduled to take place within the Clinical Research Facility at Birmingham Children's Hospital. Participants will have at least 48 hours to consider the information in the child and parent/ carer PIS before being contacted by a member of the re-search team to see if they remain interested in taking part in the study or participating in the study.

Participant Attendance

Individuals who are interested in participating in the study will contact the NIHR HRC-DDR. They may be contacted by telephone or email thereafter to confirm their interest in participa-tion. Prior to scheduling a study visit, they will be asked to confirm that their child is eligible to participate and that they have read the participant information sheet. Once confirmed, a day will be scheduled to undertake the study.

Participant Eligibility Confirmation

On attendance, participant's parents (or carers) will verbally confirm that the child being consented has an ASD diagnosis and experience of attending hospital for a procedure. Individual's medical records will not be looked at by a member of the research team to confirm this. Confirmation of eligibility will be recorded on the Eligibility Form. All eligibility will need to be fulfilled to be consented into the study.

When a subject fails to fulfil any element of the inclusion criteria, the participant will be thanked for their time but not advanced into the study. The partially completed eligibility form will be destroyed.

The inclusion/ eligibility criteria are shown below:

• A diagnosis of autism spectrum disorder (ASD)
• Aged between 3-12 years old
• Experience of attending hospital for a procedure
• Willing and able to participate in study, including using an app and providing verbal feedback

Participant Informed Consent Process

At least 48 hours (adequate time) will be given for consideration by the participant before taking part. The CI, or an individual delegated by the CI, must record when the Participant Information Sheet (PIS) has been given to the potential participant. This provision will typically be done electronically with a hard copy at participant discretion. Participants will have at least 48 hours to consider the information before being contacted by a member of the research team to see if they remain interested in taking part in the study or before participating in the study. They will be asked if they meet the inclusion criteria and then either signposted to attend or thanked for their time and consideration. If they are not eligible then all data, including name, contact details, and date the PIS was sent, will be deleted.

Where a participant meets all elements of the inclusion criteria, they will be invited to consent.

It is the responsibility of the investigator, or a suitable person delegated by the investigator, to obtain informed consent prior to participation in the study. This will follow an adequate explanation of the aims, methods, anticipated benefits, and risk of the study to the child and the par-ent (or carer). The person taking consent will be GCP trained, suitably qualified and experienced, and have delegated authority from the CI on the delegation log. The investigator or de-signee will explain that the participants are under no obligation to enter the study and that they can withdraw at any time during the study without having to give a reason.

Parents (or carers) will confirm consent by completing the Informed Consent Form on behalf of themselves and their child. The child will not be able to participate in the study without their parent consenting to both the participation of the child and themselves. All children will be asked to complete an Assent Form with the assistance of a suitably trained member of staff following an adequate and age-appropriate explanation of the study. Where the child is willing to participate but unable to complete the assent form due to age or ability, and the parent (or carer) has completed the consent form on the child's behalf, the child can still be enrolled in the study. Their willingness to participate will be assessed verbally, i.e. answering 'Yes; when asked if they would like to take part, pointing to the 'Yes' columns on the Assent Form. This will be recorded on the Assent Form. If the child states that they would not like to take part we would not continue with their participation in the study.

A copy of the signed informed consent form and assent form will be given to the participant on request. The original signed forms will be retained by the NIHR HRC-DDR team at UHB.

When a participant consents, they will be allocated the next available study number (subject ID number). For enrolled participants, this number will consist of the prefix "LJ" followed immediately by a sequential number (001 for the first subject, 002 for the second subject and so on). This number will be the unique identifier of the child and noted on the consent form and all other documentation relating to that individual.

Timing of Recruitment

Participant recruitment will only commence when the study has been initiated by the Sponsor and Birmingham Children's Hospital has issued the 'open to recruitment' letter.

Once the participant has confirmed eligibility and been consented to participate in the study, the participant will be considered enrolled in the study.

Duration of Study

Each participant's study session is expected to take approximately 1 hour. The study is anticipated to take 4-5 separate days, and the entire duration of the study is anticipated to be 1 month.

Methods

This study is a single-arm qualitative study assessing the usability of the Little Journey app in 7 children with an ASD diagnosis aged between 3-12 years. The schedule is indicated in Table 1.

Interested participants will be provided with Participant Information Sheets (PIS) for children and parents/carers which covers the essential information in the protocol, the risks and benefits associated with taking part. Participants will have at least 48 hours to consider the PIS before participating in the study. The aforementioned consent process will be undertaken before participant enrolment and progression into the study.

Training/ Explanation of the Little Journey App

A member of the research team will demonstrate how to use the app to the child and their parent (or carer), ensuring this is tailored to the age and ability of the child. Prior to starting the task section of the session, the child and their parent (or carer) will be asked if they are confident to complete the task by themselves. If they do not feel confident, further training/ demonstration will be provided by a member of the research team and there will be opportunity for the participant to walk through the tasks together, speaking out loud the actions involved. If the participant does not feel confident after second standard training cycle, a member of the re-search team will assist throughout the task completion section of the usability session. If the child and their parent (or carer) is confident, the usability task session begins and support will be provided where required.

Task Completion

A set of tasks (Task List) will be provided to the child and their parent (or carer) for completion. The session will be moderated by a member of the research team who will guide the attendees through the tasks, providing assistance where required. This will allow the usability of the app, as shown in Figure 1, to be assessed.

The tasks are detailed in the Task List. The tasks have been designed to utilise a range of the app's functionality, with particular focus on the aspects which will be utilised by children with ASD. This includes the below:

1. Setting up the app.
2. Taking a virtual tour of a hospital and accessing the age-tailored animations and information of a non-invasive hospital procedure.
3. Completion of child quizzes related to their values, likes and communication style.
4. Completion of parent quizzes related to the needs and preferences of themselves and their child.
5. Accessing the 'My Buddy' character.

The app will detail a fictional simulated non-invasive hospital procedure, the description and detail of which has been tailored for children and uses animated characters. The procedure will not be applicable to the participants current care to avoid the provision of conflicting information.

The quizzes form part of the new 'A Little About Me' (ALAM) module. The child will be able to complete the ALAM questions using real or simulated answers, dependent on their preferences. It will be emphasised that this is a study to test how easy the app is to use, and not to collect their data as this will have no implication on their future medical care. These questions will include how the child likes to be communicated with, what their child likes to do in their free time, and how they feel about being in hospital. The parent-facing quizzes focus on gaining additional information intended to enable better support provision from a hospital, including how they feel about the child being in hospital and additional information about how the child feels about being in hospital and additional needs the child has.

The task list will be used as a basis of the session and participants will be given an opportunity to use the Little Journey app as independently and naturally as possible, without interference or influence from the research team. However, the session will be completed pragmatically, de-pendent on the age and ability of the child. This will be guided by the participant and their par-ent, as well as paediatric trained clinical staff at Birmingham Children's Hospital. As a result, not all elements of all tasks may be completed during each session.

Observation

The sessions will be recorded using a video camera. This footage will focus on the participants interaction with the app. The screen of the device will be captured which will allow use of the app to be assessed. This video footage will be edited following the sessions by the NIHR HRC-DDR Team and provided to Little Journey for the uses outlined in the Informed Consent Form.

Semi-structured Interview

Following completion of the task section of the session, the child and their parent (or carer) will be asked the questions outlined in the Semi-Structured Interview Questions document. These questions focus on the acceptability and usability of the Little Journey app. These questions will form the basis of the interview, however further questions may be asked based on the use of the app in the session or feedback provided by the child and their parent (or carer). The questions may be tailored to suit the age and ability of the child, however all questions will remain focused on the usability and acceptability of the Little Journey app.

The interview will be audio recorded to allow the interviews to be captured and later transcribed ahead of qualitative analysis. These recordings will only be available to members of the NIHR HRC-DDR Team, as required for transcription and analysis.

Protecting human subjects and participants' rights

The topmost requirement of the formative human factor test is to ensure that none of the participants suffer physically or emotionally. Every participant is free to join and also to leave the test at any time. For assurance that every participant is aware of their rights, every participant will be provided with a participant information sheet at least 48 hours prior to their participation. Additionally, the participant information sheets will be discussed with the child and their parent (or carer) before they are consented into the study.

Participants will be asked to remove any identifiable items to maintain their privacy. When requested the participant's faces may be censored in the video recordings using face tracking pixilation software. Participants not wishing video recordings to be made or released outside of the study team should indicate this before the study or via a request made within 48 hours of participation. If requested, the video footage will be permanently deleted.

Video footage collected from the study will be edited and sent to Little Journey via a protected Vimeo link for download.

Risks

Emotional distress. The risk of any emotional distress has been reduced as far as possible. The study will not be specific to the child's medical care. The app will detail a fictional simulated non-invasive hospital procedure, the description and detail of which has been tailored for chil-dren and uses animated characters. Paediatric participants will be accompanied by a parent (or carer) for the duration of their participation in the study in addition to trained paediatric healthcare professionals. The child and parent (or carer) will be able to take a break, or stop taking part, at any point of the study. We therefore believe that the risk of emotional distress or upset is low.

Benefits

The study will not change how any children in the study will be looked after in hospital and therefore have no direct benefits to participants. However, the feedback collected as part of the study will be used by Little Journey to further co-develop and improve the app, which may improve the ease of use and accessibility of the Little Journey app. This may lead to reduced pre-hospital anxiety in parents and children and improved hospital experiences in the future.

Participants will be provided with a £50 Love2Shop voucher to reimburse them for their time, and any travel expenses will also be reimbursed.

Data Analysis and Management

All study documentation will be kept by the NIHR HRC-DDR team and stored at UHB until archived in line with UHB's archiving policy. If participants withdraw from the study all documentation, except the participant's consent forms, will be destroyed.

The semi-structured interviews will be transcribed by the NIHR HRC-DDR Team. Following this, the team will complete thematic analysis to identify key themes, feedback, and recommendations. Thematic analysis will matrix the data so feedback is linked to particular elements and features of the Little Journey app. The themes selected from the interviews and video footage will be agreed by at least 3 members of the NIHR HRC-DDR Team, all with previous experience of conducting qualitative thematic analysis for human factors/ usability studies. A report summa-rising the information collected by participants will be written into a report and provided to the Little Journey team, with the aim of supporting continued co-development and improvement of the Little Journey app, with particular consideration to the accessibility of the App for children with diagnosed Autism.

Recorded video footage will be reviewed and edited by the NIHR HRC-DDR Team. The edited video footage will be provided to Little Journey via a protected Vimeo link; this can be down-loaded by Little Journey and used for the purposes outlined in the A/V consent form. Partici-pants will be asked to remove any identifiable insignia before participating in the study and the footage will be edited so participants in the video are not linked to their study number.

Confidentiality

All data will be handled in accordance with the UK Data Protection Act 2018 and GDPR. The semi-structured interview forms and eligibility forms will not bear the participant's names or other identifiable information. The participant's study identification number will be used for identification and this will be clearly explained in the Participant Information Sheet.

Record Keeping and Archiving

At the end of the study, any personal identifiable data will be kept securely at UHB for 3 months. All essential documentation will be archived for 5 years from the declaration of end of study in line with UHB's archiving policy. UHB will inform the Little Journey (Sponsor) if the lo-cation of the study document archive is changed.

Essential documents are those which enable both the conduct of the study and the quality of the data produced to be evaluated and show whether the site complied with all applicable regulatory requirements. All archived documents must continue to be available for inspection by appropriate authorities upon request.

Statistical Considerations

Sample Size International standard IEC/TR 62366-2:2016, guidance on the application of usability engineer-ing to medical devices, states that many usability defects can be found with same sizes between five to eight participants. It also notes that small sample sizes can be used to satisfactorily identify usability defects, with six participants having a usability problem cumulative probability of 0.82. Additionally, there will be no comparison with other technologies which would necessitate a larger sample size. As a result, 7 participants will be recruited to the study. To note, the Little Journey app is not a medical device, however the key principles of usability testing and co-development outlined in this standard are transferable to this research.

Insurance

The Sponsor holds insurance against claims from participants for injury caused by their participation in the study. Participants may be able to claim compensation if they can prove negligence. University Hospitals Birmingham/ Birmingham Children's Hospital does not accept liability for any breach in the hospital's duty of care, or any negligence on the part of hospital employees.

Inclusion Criteria:

Child's diagnosis of autism spectrum disorder (ASD)

Child aged between 3-12 years old

Child is a registered patient at Birmingham Children's Hospital attending hospital as part of their care

Child and parent/guardian/legal representative willing and able to participate in study, including child's cognitive ability to use the app

Child and parent/guardian/legal representative's ability to understand written English and provide verbal feedback in English

Exclusion Criteria:

-

Children aged between 3-12 years old who have a diagnosis of autism spectrum disorder (ASD) and are registered as a patient at BCH.