The purpose of this research is to examine the clinical effectiveness of an investigational AI-powered smartphone app that describes the physical characteristics of an otoscopy image. It is not intended to make any diagnosis. Only healthcare professionals will make the diagnosis. The investigators will compare the diagnosis made by physicians or physician extenders (healthcare professionals) against the description provided by the support AI app tool to determine the clinical relevance of the system and examine its use within a clinical setting.

Timely access to primary care physicians for diagnosing and treating otitis media (infection or inflammation of the middle ear) in pediatric patients is often limited in rural, low-resource, and skin-of-color communities, particularly in remote or at-home settings.

This lack of access increases the risk of undiagnosed and untreated pediatric otitis media, potentially leading to severe and permanent complications, including hearing loss, deafness, tinnitus, impaired language development, cognitive deficits, and delayed educational development. A growing body of literature supports the accuracy (as measured by sensitivity and specificity of the model) of automated image analysis of middle ear conditions with machine learning-based algorithms. This proposed study will evaluate the effectiveness of a smartphone-based software that describes the physical nature of otoscopy images, powered by artificial intelligence (AI) and computer vision, in support of healthcare providers diagnosing acute otitis media and otitis media with effusion.

This pilot study will prospectively examine the value of Belle Otoscopy AI in describing otoscopy images in pediatric patients in low resourced communities who present with ear discomfort. It will also examine the utility of Belle.ai's companion software for healthcare providers caring for patients using commercially low-cost otoscope for at home image capture.

This project is funded in whole or in part with Federal funds from the Advanced Research Projects Agency for Health (ARPA-H), Department of Health and Human Services, under Contract No. 75N91023C00045.

A significant volume of research has been published over the past several years about diagnosing ear conditions through the AI methodology of deep learning on otoscopic images and supporting the accuracy of this analysis. Since 2020, studies have primarily trained convolutional neural networks to detect the tympanic membrane, with a strong focus on otitis media.

At-home ear infection detection can reduce healthcare costs by minimizing unnecessary visits to physicians and emergency rooms, lowering antibiotic prescriptions, and enabling more timely treatment for pediatric patients living in areas or communities where accessing healthcare is difficult.

A 2021 study published in JAMA Otolaryngology-Head & Neck Surgery evaluated the feasibility of patient use of low-cost digital videoscopes and smartphones for examining the ear and oropharynx. Participants were instructed to capture images and videos of their ear canals and oropharynx using digital videoscopes and their smartphones under real-time guidance over a telehealth platform. Among the images obtained using the digital otoscope ear examination, 95% were considered acceptable by the healthcare clinicians and 91% were considered acceptable by a blinded reviewer. The mean time required to acquire images for both ears was 114 seconds (95% CI, 84-145 seconds). Twenty-one participants (91%) were willing to pay for a digital otoscope for telehealth use.(https://pubmed.ncbi.nlm.nih.gov/33475683/).

With this study the investigators plan to build on this framework by evaluating the ability of Belle Otoscopy AI's to describe ear physical conditions to providers and evaluate its performance in real-world telemedicine settings. This pilot will provide an opportunity to further explore the use of advanced tools, such as Belle Otoscopy AI and its potential benefit in an under-resourced telemedicine setting. This study would represent one of the first investigations into the prospective, real-time use of an AI tool within such a setting.

Belle Otoscopy AI's convolutional neural network image referencing technology utilizes deep learning algorithms to analyze an uploaded clinical image and then compare its geometric pattern characteristics to Belle's database of images to provide description of the image's physical characteristics and differential references. For example, the software identifies ear features such as purulence (pus), redness and swelling.

The images and analysis from the Belle Otoscopy AI app can be remotely viewed by healthcare professionals via the bellePro app or the Belle.ai web platform. After the AI powered image comparison is complete, the bellePro (not Belle Otoscopy AI) displays references of similar looking images. The AI also provides a description of the visual elements observed (e.g., redness, swelling, presence of pus) that prompted the AI model to predict an ear anomaly, adding an explainable AI component that aids the healthcare professional in understanding the rationale behind the AI's description.

Data collected from this study will be used for a report and research paper publication, including the possibility of developing additional studies. Also, data from this study will be preserved for any future possible submission to regulatory bodies for review.

The study will address three primary research questions:

1. Does Belle Otoscopy AI accurately describe ear physical features? 1a. How accurate is Belle Otoscopy AI on photos taken during the same visit by clinicians and parents of pediatric patients?
2. Are parents able to capture photos using an off-the-shelf otoscope that meet the AI's requirements, and can they successfully process the images through our product to obtain a result?
3. Is the proposed clinical workflow -- including use by parents at home in coordination with a remote healthcare provider -- effective? 3a. After successful photo capture and analysis by Belle Otoscopy AI, is there a smooth transition to consultation with a physician, should an ear infection be identified? 3b. For the intended use where the product is provided to parents on the day the child is sick, without requiring an in- clinic visit, will the proposed mechanism for getting the hardware and software into the hands of the patient work?

The following describes the workflow:

The parent brings the sick child into clinic with suspected ear infection. The parent asked for consent to participate in a clinical trial. Those who consent continue to the next step. Belle Otoscopy AI is downloaded onto the parent's phone, and they are given a low-cost, off-the-shelf (OTS) otoscope. The parent is trained to use the software and the hardware. The parent takes one photo of the middle ear using the OTS otoscope and processes it through Belle Otoscopy AI. The result is recorded. Prior to using Belle Otoscopy AI, the site clinician uses standard of care method to diagnose the presence of an ear infection. The clinician's preliminary diagnosis is recorded. The site clinician takes photo of the middle ear using the OTS otoscope and processes it through Belle Otoscopy AI. The clinician makes a final diagnosis and the result is recorded. The patient's treatment plan is determined based on the standard of care method, and the patient is treated according to the plan (e.g., prescribed antibiotics). Each patient will receive a subsequent appointment to follow up in person with the provider. At the halfway point between the initial and subsequent appointment, the parent will take a new photo of the middle ear using the at-home OTS otoscope and will submit the image through Belle app. The provider will review the image, and if the condition is determined to have resolved, the upcoming appointment will be canceled. For the purposes of this study, a panel of board-certified otolaryngologists will review all medically relevant information for each study participant to establish the gold standard. The investigators then compare the gold standard to determine: a.) did Belle Ear AI increase the accuracy of the clinician's final diagnosis; b.) did Belle Ear AI facilitate the cancellation of unneeded follow-up appointments; and c.) did Belle Ear AI as used by clinicians detect Acute Otitis Media and Otitis Media with Effusion with sufficient specificity and sensitivity in-clinic to be a diagnostic tool in-clinic.

Participants in this study will download Belle Otoscopy AI onto their personal smartphones and receive a low-cost otoscope to bring home. They will be encouraged to capture images of their middle ear condition at designated time points. These images will be analyzed for condition and severity and provided to the treating healthcare providers. As is standard at the clinic, all patients within this study will be scheduled for a two-week follow-up clinical visit, but some of these scheduled visits may be cancelled at the discretion of the healthcare provider's after viewing the Belle.ai images and analysis.

In total, one image of the presenting condition will be captured by the parent of the patient using the Belle Otoscopy AI tool. All images will be analyzed and compared to the image reference database, with the images along with analytic results will be provided to the treating healthcare provider. Participating healthcare providers will use this information to make a clinical determination if an in-person follow-up visit is required to provide ongoing care. Finally, all images, analytic results, and differential diagnoses will be reviewed by the medical review committee to assess the accuracy of the Belle Otoscopy AI tool in predicting the likelihood of middle ear infection.

When the parent of a pediatric patient initiates the clinic on-site clinical flow, they register the child and then are triaged by front-line clinical staff. At this stage the patient's primary complaint is recorded and entered into the Electronic Medical Record (EMR) system. Parents of patients identified as potentially presenting with a qualifying middle ear infection condition will be approached for recruitment into the trial. A dedicated study coordinator within the clinic setting will be flagged whenever a qualifying patient is identified within EMR system.

The study coordinator will meet with the parent to briefly explain that the patient may have an opportunity to participate in a clinical trial and will ask if the patient and their parent are interested in knowing more. If they express interest, the coordinator will provide further details about the trial.

The study coordinator will then explain the study's focus, design, and what is required for the patient to participate. Participation is entirely voluntary and parents of the minor patients will be asked to sign an IRB-approved consent form before any study procedures begin. The consent form briefly explains the Belle.ai AI technology, and how it may impact the care they will receive. The consent document describes the purpose of the study, details the potential risks and benefits, provides information on data security and privacy, and describes the compensation being provided for participation. In addition, the study coordinator will describe the actions the patient is asked to undertake, which include requests for them to provide two images of their middle ear condition, one approximately 7 days after treatment at the clinic and one on or around day 14 after enrollment. If the parent is unable or unwilling to capture these images they will not be allowed to enroll into the study and will be returned to the waiting area.

If the parent agrees to the at-home image capture at the time of the initial in-clinic visit setting and provides their written consent, their child will be officially enrolled in the study. The study coordinator will then have the parent download Belle Otoscopy AI onto their smartphone. A scannable QR-code will be provided that will automatically download, install, and verify that the application was correctly set up. The study coordinator will enter a pre-assigned Study ID code number into the app which will serve as the patient's study ID outside of the clinic setting. In addition, the coordinator will collect the parent's phone number, which will be stored along with their ID number in the clinic's database. These code numbers will be linked to the patient's EMR record; however, no identifying information will be stored outside of the clinical setting. The parent will also be provided with an low-cost, off-the-shelf (OTS) otoscope.

The study coordinator will maintain a written key in a study notebook that connects the Study ID code number and patient's phone number to the EMR record number. This study notebook will be kept in a locked drawer within the research office until data collection has ended and the analysis phase of the study begins.

The study coordinator will then demonstrate the use of Belle Otoscopy AI and the otoscope and answer any questions the parent may have about its use. Included in this demonstration will be an example image capture, in which the study coordinator will show the patient step-by-step how to properly operate the tool (this test image will not be included in the EMR clinical record).

The parent will use an OTS otoscope to capture an image of the patient's middle ear and upload it to Belle Otoscopy AI. The AI will analyze the image, generate a result (likely infected or not infected), and update the system of record.

The HCP will conduct a standard evaluation, including history-taking, examination, and diagnosis. Encounter notes will be documented in EMR record.

The HCP will then use the OTS otoscope to capture and upload an image to Belle Otoscopy AI for analysis. The AI will process the image, generate a result, and update the system of record.

The provider will proceed with appropriate treatment, such as prescribing medication. A 14-day follow-up appointment will be scheduled at the primary clinic.

Beginning on Day 7 after study enrollment, patients will receive push notifications prompting them to capture an image using the previously downloaded Belle Otoscopy AI tool. Notifications will continue through Day 10, or until a usable image is received. When study images are received, the study coordinator will receive a notification. If no image is received by Day 9, the coordinator will call the patient to remind them to send the image (see reminder phone script provided in the attachment below) and will continue calling through Day 10 (up to 4 calls per participant). The goal is to receive a clear image of the presenting condition approximately one week after initial treatment. Images received within a 7-10-day window after enrollment will be considered on protocol; those received afterward will be included in the intent-to-treat analyses.

Upon receiving the Belle.ai image, the study coordinator will notify the original provider, who will be prompted to review the case via a companion provider app called bellePro on their work smartphone. Upon review, a refreshed set of [differential diagnoses] or [binary predictions] will be provided within the bellePro app based on the image analysis, at which time the HCP will have the option to update their diagnoses if desired within the encounter notes in EMR record.

After reviewing the AI-analyzed image, the HCP will determine whether to proceed with the Day 14 follow-up appointment or cancel it, replacing it with a final image request. The study coordinator will execute the decision, either canceling the appointment and providing patient instructions or issuing reminder calls.

The subset of patients who did not have their appointments cancelled after review of their Day 7 progress will be expected to attend their scheduled visits. Per protocol, the study coordinator will initiate reminder calls starting on Day 12 and continuing until the morning of the appointment (up to three reminder calls are planned).

Follow-up patients are instructed to come to the clinic on or around Day 14 approximately 30 minutes prior to their scheduled appointment time. If unable to attend on Day 14, efforts will be made to reschedule as close as possible, with all visits occurring by Day 18 to remain within the study window.

The study coordinator will be informed in advance of all follow-up appointments and will meet the study patient in the appropriate waiting area. Due to scheduling constraints inherent in the workflow of the clinic, all clinical trial follow-up appointments will be seen by either the original provider or a provider other than the one who treated the patient on intake. The study coordinator will accompany the patient to their scheduled appointment and inform the follow-up provider that the patient is part of the clinical trial. The coordinator will then wait outside the exam room to allow the clinical visit to proceed as normal. After routine clinical activities, the study coordinator will re-enter to capture a Belle Otoscopy AI image under HCP supervision. All encounter notes will be recorded in EMR record per standard clinic workflow.

If during the Day 7 image review the original treating HCP determines a follow-up appointment is not required, then the study patient will complete the following procedures instead of an in-person visit.

On Day 14 after study enrollment, push notifications will be sent to participating patients' phones requesting a study image to be captured using the Belle Otoscopy AI image capture App. Notifications will continue through Day 18, or until a usable image is received. When study images are received the study coordinator will receive a notification. If no notification is received for a patient case by Day 15, the study coordinator will call the patient to remind them to send the image and will continue calling through Day 18 (a total of 4 calls per participant will be scheduled). Images received within a 14-18-day window after enrollment will be considered on study visit window, those received afterward will be included in the intent-to-treat analyses.

Please note that if the patient's condition significantly worsens after a canceled follow-up, a new clinical encounter will be scheduled to ensure appropriate care. These cases will be analyzed per intent-to-treat.

The study participation window extends from the day the patient enters the clinic (Day 1) through to Day 14-18, when either a final Belle Otoscopy AI image is acquired at the completion or the in-person follow-up appointment. Encounter notes in EMR record will be updated throughout the study, and a final determination will be made as to the disposition of the patient at study completion noting if the case is considered resolved, or if they have an ongoing medical condition. Note that the majority of middle ear cases seen at the clinic are acute, and resolved within the 18-day timeline, although some patients present with chronic and/or episodic conditions or with a more severe condition which may require ongoing care.

HCP's who work in the clinic will be approached to participate in the study. Study procedures will be described, and willing HCP's will be onboarded. The HCP's will initially complete an electronic intake survey prepared within the Qualtrics software system. This intake survey will collect data on provider demographics, training, years of practice, perceived comfort diagnosing ear infections, and comfort using supportive AI technology in the clinic.

Post completion of the intake survey, participating HCP's will proceed to the training phase. This will take place virtually via a group video chat, wherein HCP's will be shown through the bellePro physician app and instructed as to its use in the study. The procedures will be demonstrated as described in this protocol, and any questions from the HCP's will be addressed. The HCP's will then be instructed to download bellePro to their work smartphones, which will be electronically confirmed by Belle. Finally, the activities of the medical review committee will be described (see below), including the workflow associated with reference diagnosis determination.

Once all study procedures are completed, participating HCP's will be asked to complete an electronic exit survey, also prepared within the Qualtrics software system. This exit survey will collect data on HCP experience while using the Belle.ai AI clinical support system, including their willingness to use the system in the future, if they would recommend the App to other clinicians, and any suggestions they may have for improvements to the system.

Inclusion Criteria:

• Patient must be aged 1 to 17 years old.
• Patient must present with a primary middle ear complaint.
• Parent of the pediatric patient must have the ability and willingness to provide informed consent and comply with study procedures and visits.
• Participants must have access to the required technology (e.g., smartphone with internet access) and be capable of using the provided off-the-shelf otoscope for the required image capture.

Exclusion Criteria:

• Patients who are unable to comply with study procedures due to physical or mental health limitations (as assessed by study coordinator).