For several decades, hospital emergency departments have been experiencing congestion, sometimes reaching saturation point, where they are no longer able to fulfil their primary mission: to prioritise patients requiring immediate care due to a clinical situation that could be life-threatening or functionally debilitating. The main reason for this situation is a structural mismatch between medical needs, which have increased due to population ageing, and outpatient care supply, which has remained relatively stable in order to contain healthcare expenditure. As a result, a large proportion of people visiting hospital emergency departments are individuals who have been unable to find a solution to their medical needs in the community and have turned to the hospital as a last resort. These are patients seeking urgent, unscheduled care who have been unable to obtain an appointment with their general practitioner or another primary care professional.

In times of extreme pressure, as sometimes happens in France during the summer, access to hospital emergency departments is limited to patients who have received prior authorisation to attend. Similarly, new ways of managing these requests for urgent or unscheduled care are being sought in the field of medical regulation.

Triage of patients by telephone appears to be an essential step in medical regulation prior to access to hospital emergency departments. Indeed, if solutions are available in the city for patients who do not need to go to the emergency department, this triage will optimise the resources of the healthcare system.

However, quickly assessing patients without visual contact (who may be in a state of emotional distress or face a language barrier) is a particularly delicate task. Several triage algorithms are available to assist telephone operators. However, these require structured clinical information that is not easily and quickly accessible during calls.

For several years now, artificial intelligence (AI) has emerged as a promising alternative for assisting operators, as it enables the management of large amounts of unstructured data, particularly audio exchanges. AI-based classification models using audio data have shown that they could be useful in medical regulation, particularly in cases of cardiac arrest, stroke or myocardial infarction. However, to our knowledge, previous studies have focused on specific disorders, and their models are not capable of handling the vast range of cases inherent in the classification of general front-line emergency calls.

In this context, researchers have developed an AI-based model to identify patients requiring referral to hospital emergency departments among outpatients seeking emergency or unscheduled care through medical call centres. To do so they used telephone calls and medical records from SOS Médecins Grand Paris, a group of approximately 150 general practitioners and emergency doctors who mainly offer same-day home visits in Paris and its neighbouring departments (more than 6.5 million inhabitants).

The objective of this study is to evaluate the model's ability to identify patients requiring hospitalization based on (1) new data from SOS Médecins Grand Paris, but also (2) data from Corsica, (3) to compare the model's predictions with those of a physician, and (4) to determine the general conditions for using the predictions in current practice.

1. Introduction - scientific justification 1.1. Context As the population ages and the prevalence of chronic diseases increases, the number of general practitioners is declining. This growing imbalance between medical needs and supply is leading to difficulties in accessing primary care, particularly urgent and unscheduled care.

   To address this issue, France has set up a healthcare access service (SAS). In the event of urgent or unscheduled care needs, the SAS allows individuals whose primary care physician is unavailable to access a healthcare professional. Depending on the situation, the service can provide medical advice, offer a teleconsultation, refer the person to a local doctor or emergency department, or even call out a mobile emergency and resuscitation unit (SMUR) or ambulance. The SAS functions as a triage system, providing a response tailored to each individual's needs.

   The aim of this project is to study certain aspects of the validity of an artificial intelligence (AI) triage model for patients seeking urgent or unscheduled care, enabling the identification of patients who will require hospital treatment.

   This project follows on from a previous study that developed a model using AI to triage patients using SOS Médecins Grand Paris. As in the SAS, people who call SOS Médecins present their problem to a medical dispatch assistant (MDA), who in most cases arranges a home visit.

   The SOS Médecins databases contain audio recordings of care requests and the medical decisions made by doctors after their home visits, including whether or not the patient was referred to hospital (the data on patient outcomes is exhaustive). A predictive model of patients' use of hospital services following a home visit by an SOS doctor was therefore developed using AI. The model "listens" to the exchanges between the patient and the MDA and predicts the use of hospital services following the doctor's home visit.

   The model and its development are the subject of an article currently submitted for publication in an international journal. During the test phase, it achieved an area under the ROC (receiver operating characteristic) curve of 0.80. Ultimately, the AI-based model will need to operate in real time and provide information to MDAs to assist them in their work. For the time being, the aim of this project is to complete the validation of the system. It will then need to be implemented and tested in real-world conditions in order to determine the benefits of patient triage by MDAs using the model's predictions.

   It should be noted that while SOS Médecins is not the same as SAS, it is similar. First, the two structures operate in a similar way: an MDA contacted by telephone attempts to respond to a request for urgent or unscheduled care. The range of possible responses is the same for both structures (SOS does not only offer home visits, it also conducts face-to-face or video consultations and transfers certain calls to emergency medical service), as is the use of a doctor's expertise if necessary (medical regulation). Secondly, SOS Médecins is one of the downstream effectors of the SAS. Older and better known to the general public and healthcare professionals than the SAS (which currently covers only 60% of the national territory), SOS could have a recruitment process very similar to that of this new service. To our knowledge, there are not yet any statistics on the SAS that would allow a comparison of the activities of the two structures. In this context, validation within the SAS of the model developed with SOS data will therefore be necessary. If, in the future, data show that the two structures are similar, the SAS could benefit from this project developed at SOS. SOS data have a significant advantage over SAS data: they contain information on patient outcomes, including whether or not they went to the hospital emergency department. This labeling of calls enabled supervised learning of an AI model.

   This project, which to our knowledge is unique, paves the way for triage assistance that can be used in the SAS (and perhaps to some extent in emergency departments).

   1.2. Benefits for patients and the healthcare system Early identification of patients at high risk of hospitalization, some of whom have conditions requiring emergency care in a specialized setting, should facilitate clinical decision-making and ultimately improve patient health. This primarily concerns patients with conditions whose prognosis is directly linked to the speed of care delivery, such as severe ischemic, respiratory, traumatic, or infectious conditions, who will benefit from being hospitalized without waiting for a doctor to visit them at home. The benefit for patients will therefore be a reduction in lost opportunities due to triage errors.

   In addition, triage upstream of emergency departments (as proposed by the previous Minister of Health and implemented in some understaffed emergency departments and is routine practice in Denmark) will improve the functioning of hospital emergency departments, where excess activity has an impact on the mortality of elderly patients receiving care. In terms of public health, the introduction of an effective triage system could therefore improve access to urgent and unscheduled care.

2. Research objectives and evaluation criteria The AI-based triage model used to identify patients who need to be referred to hospital has already been developed and tested using calls to SOS Médecins Grand Paris. An article is currently being published. Therefore, this project does not focus on the development or internal validation of the triage model, but on other aspects related to its validation: prospective and external validation; comparison of its predictions with those of a medical expert; investigation of social discrimination that could be associated with its use; and the determination of the actual conditions of its use in healthcare settings.

   2.1. Objective and primary endpoint The prospective evaluation of the performance of predictive models on real-world operational data is a crucial step towards integrating these models into the clinical context and assessing their impact on patient care.

   - Primary objective (PO): Prospective validation of the triage model used to identify patients who will be referred to hospital following a home visit by a doctor.

   The area under the ROC curve (AROC) provides an overall measure of the performance of a classification system.

   It should be noted that all the data used in this study are retrospective. The model was designed using data from 2023 and earlier (up to and including 2017), and the "prospective" data are those subsequent to its design and will be the data for 2024.

   - Primary endpoint (PE): Comparison of AROCs obtained from representative samples of retrospective and prospective data from SOS Médecins Paris.

   2.2. Secondary objectives and endpoints External validation on patient data from separate geographic sites is necessary to understand how models developed at one site can be safely and effectively implemented at other sites.

   - Secondary Objective #1 (SO1): External validation of the triage model

   • Secondary Endpoint #1 (SEP1): Comparison of AROCs obtained from representative samples of retrospective data from SOS Médecins Grand Paris and Ajaccio.
   • SEP2: Comparison of the triage model's predictions with those of an expert physician.
   • Secondary Endpoint #2 (SEP2): Comparison of correct prediction rates between the triage model and an expert physician on a balanced non-representative sample (i.e., comprising 50% of patients referred to the hospital following a home visit by the physician, rather than 3% as is the case in reality) of data from SOS Médecins Grand Paris.

   Several AI-based triage systems have been found to be discriminatory. Two well-known examples illustrate this point:

   • the selection of computer engineers applying for jobs based on their resumes: the system had been trained using existing engineers, who were predominantly male, and it disadvantaged women;
   • The detection of melanomas from dermatoscope images: the system had been trained using light skin tones and was not as effective on dark skin tones.
   • OS3: Study of potential discrimination phenomena associated with the model, in relation to the social characteristics of patients.
   • Secondary endpoint #3 (CJS3): Comparison of AROCs obtained from representative samples of SOS Médecins Grand Paris data for both sexes.

   The transition from a validated predictive model to an efficient device in real-world conditions is a well-known challenge in AI.

   - SO4: Determination of the real-world conditions of use of the model by MDAs.

   - Secondary endpoint #4 (SEP4): Drafting of specifications for the implementation of the triage system by MDAs

3. Selection and exclusion of individuals from the study 3.1. Inclusion criteria Individuals of any age who received a home visit from an SOS doctor following a call in 2024 or earlier.

   3.2. Exclusion criteria Individuals who do not speak French. Individuals without health insurance.

4. Description of the study 4.1. Type of study This is essentially a study to validate an AI-based patient triage model. It will be conducted using observational data.

   4.2. Research process 4.2.1. Location of the research The study will be conducted at the SOS Médecins premises. Medical experts from SOS Médecins Grand Paris will be recruited to carry out the SO2.

   4.2.2. Recruitment methods Data collected by SOS Médecins in the course of its activities will be used.

   4.2.3. Patient care procedures The project focuses on the need to use hospitals to meet the demand for urgent or unscheduled care from patients in the community. However, it will not interfere with patient care.

   4.2.4. Study exit procedures Patients who request to be excluded will be excluded.

   4.2.5. Data collection

   • PO (prospective validation), SO1 (external validation) and SO3 (discrimination):

   The data used will be that already collected in the course of SOS Médecins' activities. Both sites (Paris and Ajaccio, Corsica) use the same software to manage telephone calls from the moment they are received and recorded until the end of the SOS care process (i.e., the end of the home visit in our case), including the entry of administrative and medical data (SOS Médecins file). Here is the list of data that will be used and how it will be processed for analysis:

   - Transcripts of audio recordings of conversations between patients and MDAs. AI predictions will be made based on these transcripts (and only these transcripts).

   • Data from the SOS Médecins file:
   • day of the week, month, year, and time of the phone call;
   • the census area number of the place of visit will be obtained from the address provided by the patient. Census area are the smallest spatial units for which statistical data is available while preserving the anonymity of its inhabitants. Several socio-economic variables from the INSEE (National Institute of Statistics and Economic Studies) census are available at this level, such as unemployment rates, high school graduates, workers, median income, the social disadvantage index (also known as FDep for French DEPrivation index) and EDI (for Ecological Deprivation Index). These different variables provide ecological (as opposed to individual) measures of the social position of the individuals residing there;
   • reasons for the call, defined by the MDA, based on a predefined list but not corresponding to an international classification. Several reasons may be reported for the same call;
   • level of urgency (4 levels), defined by the MDA;
   • date and time of the end of the home visit, defined by the doctor at the end of the visit
   • Consultation diagnoses, defined by the doctor at the end of the visit, based on a predefined list but not corresponding to an international classification. Several reasons may be reported for the same consultation.
   • Patient's status at the end of the visit (closed list: left on site, transfer to hospital emergency department, call of emergency medical services).
   • method of transfer to the hospital emergency department (closed list: by own means, ambulance, fire department, SMUR).
   • SO2 (medical expert):

   Some individual data will be collected from the medical experts participating in the achievement of this objective: age, gender, length of practice overall and within SOS Médecins, specialty.

   In addition, after reading the transcripts of telephone exchanges between patients and ARMs, these physicians will be asked to estimate the probability of patients being transferred to hospital.

   They will be asked two questions:

   • What is the probability that this patient will be hospitalized after the SOS doctor's visit? Answer by placing a cross on the scale from 0 (no hospitalization) to 100 (hospitalization).
   • How certain are you about this probability? Answer by placing a cross on the scale from 0 (no certainty) to 100 (total certainty)

5. Description of the study interventions There are no interventions in this study, which is strictly observational.

   Nevertheless, we describe here the organization of the work that will be carried out to complete SO2 and SO4.

   • SO2 (medical expert):

   In order to compare the triage model's predictions of hospital use with those of a medical expert, doctors from SOS Médecins Grand Paris with more than five years' experience in the organization will be recruited on a voluntary basis. Each of these doctors will be asked to estimate the probability of hospital use for several patients/calls (see §4.2.5. Data collection).

   - SO4 (real conditions): A working group consisting of statisticians, computer engineers, MDAs, SOS doctors, and ergonomics researchers will be formed to consider the best way to provide MDAs with the probability of hospital use. Several issues will need to be addressed: how the probability is displayed on the screen, the probability threshold, whether the call should be transferred to the dispatch physician (and if so, at what threshold), the influence of the probability on the urgency level of the call as defined by the MDA, and the possibility of giving priority to certain urgent requests. After an initial working meeting, a version will be implemented with volunteer MDAs. After initial use in real conditions, feedback will be gathered and a new experiment conducted. Finally, specifications for implementation in real conditions will be drawn up (with reasons given for the choices made) so that the triage model can be implemented on different IT platforms.

6. Statistical analyses 6.1. Calculation of the number of subjects - PO (prospective validation): In order to demonstrate that the AROC on prospective data is different from the ROC on retrospective data, if AROCs of 0.75 and 0.80 are observed respectively, we will need 13,560 patients/calls per sample (i.e., 407 patients who will be referred to hospital and 13,153 patients who will not, in line with the 3% of patients referred to hospital observed at SOS Médecins Grand Paris), assuming an alpha risk of 5%, a power of 90%, and a two-tailed test. This result was obtained using the power tworoc function of the Stata statistical software.

   - SO1 (external validation): According to the same calculation as for the OP, a sample of 13,560 patients will need to be mobilized in Ajaccio, Corsica.

   - SO2 (expert physicians): While all other analyses will be based on representative samples, this analysis will be based on a balanced sample (i.e., composed of 50% of patients who will be referred to the hospital and 50% who will not) for two reasons. With a low rate of hospital referrals (3%) and a low number of calls to be assessed for each of the expert physicians, a possible strategy for a physician wanting to ensure good performance would be to never refer patients to the hospital. Furthermore, a representative sample would not allow for the study of false negatives (patients with a low estimated probability of being referred to hospital when in fact they will be) as there would be too few of them.

   In order to highlight a difference in the correct prediction rate between the triage model and an expert physician, 365 calls from patients who use the hospital (and the same number of patients who do not use it) would be required, assuming a correct prediction rate of 58% (see appendices §12.2 for the determination of this rate) and 68% (a priori choice of 10 additional percentage points) for the triage model and the expert physician respectively, with an alpha risk of 5%, a power of 80%, and a two-tailed test (Stat's power twoproportions function).

   Based on an assessment of one call every 5 minutes, a total of 61 hours of medical expert time will be required. Based on 4 hours of work per physician, this corresponds to 16 medical experts to be recruited.

   - SO3 (discrimination): According to the same calculation as for the PO, a sample of 13,560 patients of each gender will need to be mobilized if we want to show a difference in AROC of the triage model between men and women. The sex ratio of people calling SOS Médecins Grand Paris is close to 1. Both retrospective and prospective samples from SOS Médecins Grand Paris from the PO will be used.

   6.2. Description of the statistical analyses used The study will be conducted in accordance with the TRIPOD-AI recommendations, which are the guidelines for reporting prognostic and diagnostic studies (specifically for AI)

   6.2.1. Selection of individuals to be included in the analyses For all objectives, the selection of individuals to be included will be made by random drawing based on a round number of years. This will cover all hours of the day, days of the week, and months of the year. For SO3, the non-representative sample will contain 50% of patients referred to the hospital.

   6.2.2. Descriptive statistics Descriptive analyses will be performed for all variables collected. Quantitative variables will be described using their mean, median, standard deviation, 95% confidence interval, interquartile range, minimum, maximum, number of missing values, and number of unusable values. Quantitative variables will be described by their percentages and 95% confidence interval. The time elapsed between the call and the end of the visit will be presented. These analyses will be performed on all samples and stratified according to hospital use.

   6.2.3. Analysis of the primary endpoint We begin by presenting the generic analysis strategy that will be used for various purposes.

   To evaluate the performance of the different triage systems (AI and medical expert) on different samples (prospective or retrospective, in Paris or Ajaccio, Corsica) of patients, ROC and precision-recall (PR) curves will be plotted and the areas under these curves (AROC and APR) estimated. The AROC allows us to measure the overall performance of a classification system. APR will also be used because it is particularly informative when evaluating binary classifiers on unbalanced data sets, in which the number of negative results far outweighs the number of positive results (which corresponds to our situation). To compare two triage systems (AI and medical expert, for example) or the same triage system (AI, for example) on two different populations (Paris and Ajaccio, Corsica, for example), their AROC will be compared.

   In addition, to evaluate performance in a meaningful way from an operational perspective, sensitivity (Sen), specificity (Spe), positive predictive values (PPV), and negative predictive values (NPV) will be calculated for different thresholds.

   - PO (prospective validity): The generic analysis strategy for the AI triage model will be conducted on two representative samples, one retrospective (2023 and previous years) and one prospective (2024 years) from SOS Médecin Grand Paris.

   6.2.4. Analysis of secondary endpoints

   - SO1 (external validity): The generic analysis strategy relating to the AI triage model will be conducted on two representative retrospective samples from SOS Médecin Grand Paris and Ajaccio, Corsica.

   - SO2 (expert physicians): The correct prediction rates of the two triage systems (AI and expert physicians) will be compared using a statistical test.

   On a more exploratory basis:

   - The AROCs of the two triage systems will be calculated (generic analysis strategy).

   - The prediction differences between the two triage systems will be modeled based on actual hospital use and patient characteristics.

   - Weighted kappa coefficients will be calculated between the two predictions.

   - sensitivity analyses will be performed based on the level of certainty reported by medical experts (particularly high and low levels of certainty);

   • stratified analyses of actual hospital use will be performed (study of true positives and true negatives).

   Where possible, the analyses will take into account the fact that the same medical expert may have examined several calls/patients.

   • SO3 (discrimination):

   Discrimination will be investigated according to the following social characteristics: age, gender, and social position (estimated based on the socioeconomic characteristics of the census area of the place of visit). . Age and ecological variables of social position will be used in the form of tertiles and quintiles. The generic analysis strategy relating to the AI triage model on representative samples from SOS Médecin Grand Paris stratified according to the above social characteristics will be carried out. The comparison of AROCs will determine whether triage is equally effective according to the above social characteristics.

   In addition, a comparison of successes (true positives and true negatives jointly and separately) according to the above characteristics will also be carried out using univariate and then multivariate logistic models. These analyses will be carried out for different risk thresholds.

   - SO4 (real conditions) Following discussions among colleagues and full-scale testing, specifications for implementation in real conditions will be drawn up so that the triage model can be implemented on different IT platforms. The rationale behind the choices made will be specified in these specifications.

Inclusion Criteria:

• Individuals of any age who received a home visit from an SOS doctor following a call in 2024 or earlier.

Exclusion Criteria:

• Individuals who do not speak French.
• Individuals without health insurance.

Person requiring urgent or unscheduled care

The geolocation of individuals could be identifying.