Can a Tailored Quality Assurance Program Help General Practitioners Maintain POCUS Scanning Competence? A Cohort Study From Denmark
Can a Tailored Quality Assurance Program Help General Practitioners Maintain POCUS Scanning Competence? A Cohort Study From Denmark
Testing of a quality assurance program for general practitioners with an online multiple-choice quiz including still-pictures and video sequences of 10 application specific POCUS examinations. Through 15-20 questions, the quiz tests participants (1) knowledge about indications for performing POCUS examinations, (2) applied knowledge of ultrasound equipment, (3) ability to optimize images, (4) ability to recognize and present structures, (5) ability to interpret images, (6) ability to describe and document findings and (7) medical decision-making. Following the quiz, the participants are offered guidance to improve their performance.
Introduction Point-of-care ultrasound (POCUS) examinations are increasingly used in general practice . General practitioners (GPs) can learn to perform ultrasound examinations by attending ultrasound courses, but after course attendance they are left to maintain and develop scanning competence by themselves . Maintaining scanning competence in general practice is challenged by (1) a low frequency of performed examinations as not all patients seen in general practice present conditions suited for POCUS , (2) a low pre-test probability of disease in general practice which challenges the ability to recognise different pathologies, (3) the lack of supervision as GPs often work alone in small offices , and (4) the short consultations leaving little room for practice.
An ultrasound course tailored for Danish office-based GPs has been developed offering continuous practice under supervision during the three months that the GPs are enrolled in the course. Previous research have shown that this ultrasound course leads to scanning competence within the ten scanning modalities that are included in the course curriculum. The course is designed to make the participants move towards self-directed learning , but we do not know if scanning competence obtained on the course can be maintained over time.
Within some medical specialities, the number of specific POCUS examination performed have been used as an indicator for scanning competence in re-certification . However, little is known about the number of performed POCUS examinations needed to secure skills. In general practice, continuous medical education (CME) programs have been found to increase GPs knowledge and skills in other areas , but as POCUS is just a smaller part of the clinical work in general practice, CME programs for GPs cannot be too time-consuming.
Specific objectives The overarching aim of this study is to investigate whether a quality assurance program, tailored for GPs who use POCUS as part of their patient examination in general practice, can lead to maintained scanning competence 12 months after baseline.
Trial design This is a cohort study designed as a follow-up for a hybrid effectiveness-implementation study (ClinicalTrials.gov Identifier: NCT05274581). In the previous study, 18 GPs attended an ultrasound course targeted to office-based GPs. The ultrasound course consisted of three teaching seminars over three months (Marts - June 2022), a curriculum of 10 POCUS applications, an online learning platform providing educational support before, during and after the teaching sessions (see Figure 1). At the end of the ultrasound course (June 2022) and again 3 months after the ultrasound course (September 2022), all participants had their scanning competences within the 10 POCUS applications assessed by external experts. These experts were blinded to the participants previous experience and learning process. The assessment of scanning competence was done using the OSAUS assessment tool . Following this study participants have implemented POCUS in their clinical practice, where they use POCUS as part of the patient examination.
Study settings This study will be conducted in office-based general practice in Denmark. GPs in Denmark are self-employed and work in office-based general practice clinics. Denmark has a public health care system where almost all patients are listed with a GP for primary health care. Consultations and treatments are free-of-charge for patients. GPs act as gatekeepers for other primary care healthcare providers and secondary care specialists. GPs are paid through a combination of remuneration and fee-for-service financed through taxes8. There is no fee for performing POCUS in primary care and GPs must cover expenses for the ultrasound device and their ultrasound education themselves. However, during this study participating GPs will have a fee for performing POCUS examinations. POCUS scanning-fees will be registered in the medical record system in the practice.
The study will be coordinated from CAM AAU and data will be collected by the research team and by the participating general practitioners in their clinics. All study data will be stored at a secure server at Aalborg University.
The quality assurance program will be developed and delivered by research team, who are experienced GPs, POCUS users and/or teachers. The competence assessment (OSAUS score) will be delivered by a team of POCUS experts (SKA, OG, MBJ, CS and LP), who have extensive POCUS experience.
Outcome measures Background characteristics of participating GPs At baseline the following participant characteristics will be collected: Age (years), gender (M, F, other), previous use of ultrasound (number of months with regular use), previous ultrasound courses of minimum 1 day duration (yes/no), scanner type (low range, mid range, high end), type of practice (collaboration, partnership, solo), location of practice (urban, rural, mixed), number of patients assigned to the practice, and number of GPs working in the practice.
In addition, the results from the OSAUS assessment of scanning competences conducted on all participants in September or October 2022 will be included.
The OSAUS assessment tool has been developed and validated as a generic tool for assessing scanning competence14. The OSAUS scale consists of seven items: 'indication for the examination', 'applied knowledge of ultrasound equipment', 'image optimization', 'systematic examination', 'interpretation of images', 'documentation of the examination' and 'medical decision-making' and each item is rated using a five-point Likert-scale with descriptions of performance ranging from very poor (score = 1) to excellent (score = 5). All items are weighted equally, as high inter-item correlation have been found previously . Hence, for each scanning modality a total score from 7 to 35 points may be achieved.
The OSAUS ultrasound competence assessment tool have been developed as a generic tool to assess ultrasound competence across medical specialities. The assessment tool was developed through a Delphi study14 and it has been used to assess clinicians' ability to transfer learning from an ultrasonography course into diagnostic performance on patients. Furthermore, the tool has been found valid and reliable to distinguish between novice POCUS users and experts .
For the primary outcome(P1) and secondary outcomes (S1-S2), the Objective Structured Assessment of Ultrasound Skills (OSAUS) score after 12 months is used. For the primary outcome (P2), the OSAUS score at baseline and at 12 months is used.
Primary outcomes:
(P1): The summarized OSAUS score after 12 months for all ten modalities will be calculated as percentage of maximum score. To summarize normal and non-normal distributed data similarly mean(SD) as well as median(IQR) will be presented together with minimum and maximum scores.
Primary outcome 2 (P2):
The summarized OSAUS score at baseline and at 12 months for each participating GP will be compared and the proportion of GPs, with an OSAUS score at 12 months larger than or equal to the ultrasound competence score at baseline, will be reported.
Secondary outcomes (S1): For GPs, participating in a POCUS quality assurance program, the OSAUS scores and the item scores after 12 months, for each of the ten scanning modalities included in the curriculum will be calculated and presented as median, IQR, minimum and maximum score.
(S2): The proportion of GPs, who achieve a minimum OSAUS score of three for all seven domains for each of the ten scanning modalities after 12 months will be calculated. As variation in scores can occur between the four expert assessors. The scores will be normalized by multiplying a given accessors score with (0.2 x sum of scores of all five accessors/the given accessors mean score) prior to calculating the proportion of successful GPs.
For the secondary outcome S3, the GPs assessment of their own scanning competence will be used. The GPs will declare to which degree they have POCUS scanning competence within a specific scanning modality to perform the scan un-supervised in general practice (To a very high degree, to a high degree, to some degree, to a lesser degree, not at all, unsure).
(S3): The proportion of GPs, who rate themselves as competent to perform POCUS un-supervised in general practice, after the educational period (three months after baseline), will be calculated for each of the ten scanning modalities.
For the secondary outcome S4, the fee-specific codes registered by the GP during consultation in general practice will be used. Following the use of POCUS in the general practice consultations the GPs will follow normal procedures for registration of activities in general practice using remuneration codes and fee-specific codes for using POCUS in the medical record system. Prior to this study, 10 fee-specific codes for the 10 different POCUS examinations in the curriculum will be installed in the medical record system. The Primary Sector Data Provider Platform (PLSP) will develop an algorithm that allows for the following data extraction on each participating GP in a given time frame: (1) number of POCUS examinations performed and (2) number of consultations. PLSP will deliver monthly aggregated data for each participating GP from baseline to 12 months after baseline.
(S4): The number of performed POCUS examinations by each GP each month during the study (months 1-12 after baseline) will be summarized to demonstrate variation over time.
For the secondary outcome S5 and S6, the results of distributed online quizzes to the participants will be used. During the study, four online quizzes will be send to participants by email (Months 3, 6, 9, and 12). A reminder will follow after two weeks to participants who fail to respond the quiz. The results of these quizzes will be summarized as the total number of completed quizzes for each participant (S5) and the test score for each participant at months 3, 6, 9, and 12 (S6).
For the secondary outcomes S7, S8 and S9, a linear regression model to test associations between OSAUS score after 12 months and the total number of performed POCUS examination for each scanning modality (S7), the number of completed quizzes in the quality assurance program (S8), and the tests scores in the quality assurance program (S9) will be used. Changes in OSAUS scores between baseline and 12 months (S10) are calculated using linear regression. Scores at baseline and after 12 months will also be compared by a paired t-test.
Sample size The potential study population is the 18 GPs who participated in the ultrasound course organized by PLO-e. A participation rate of 80% corresponding to 15 GP is expected.
With 15 participants the total number of possible completed 7-item OSAUS questionnaires will be 15 for each modality and 150 for the combined estimate after 12 months. This is considered sufficient to estimate mean estimates and provide confidence intervals with acceptable spread.
Recruitment All participants, who attended the PLO-e ultrasound course were invited to participate in the study. The ultrasound course was listed in PLO-e course catalogue for 2022 and as such offered to all GP working in primary care in Denmark.
Blinding Expert assessors performing the competence assessment (primary outcomes) have a medical background and are considered experts in the field. They have not been teaching participants in the training program. They will be blinded to the participants prior experience with POCUS, the number of performed POCUS examinations, and any other elements in the participants' learning process.
The principal investigator (SKA) and MBJ acts as expert assessors in this study and as such, they will be blinded to the background information on participants, the data collection and the analysis of data. CAA will be in charge of all participant correspondence, AR will be cleaning the data set and analyzing the data.
Data collection, management, and analysis
Data collection
Multiple Choice Questionnaire (MCQ) MCQ's will be distributed by email to participating GPs at 3, 6, 9, and 12 months. The email includes a link to an online questionnaire prepared and collected using Forms (Microsoft Office, Redmond, USA). A reminder will follow after two weeks to participants failing to complete the MCQ after the initial invitation. Data will be transferred and saved on a secure server at Aalborg University.
Assessment of scanning competence The assessment of participants scanning competence using the OSAUS score will be performed in the same manner using the same expert assessors at baseline and after 12 months.
During the assessment, only the participant and the assessor are present. Each assessment will follow the same structure, where experts will assess the participants by asking the following questions:
- In which clinical scenarios would you perform this POCUS examination (Item 1 in the OSAUS)
The experts will ask participants to demonstrate the POCUS examination (for maximum five minutes) to assess the following:
The experts will present the participants with a picture of common pathology and ask the participants the following questions:
Following the assessment, the participant will receive feedback on the level of performance and educational advice. The total duration of the assessment is expected to last 90 minutes for each participant. The assessor completes an OSAUS score for each of the ten examinations performed by the participant.
Education of POCUS expert assessors The POCUS expert assessors all have prior experience using the OSAUS score. Still, prior to the assessment of the participants' performance in this study, the POCUS experts participate in a 90-minutes online training session with CAA who has prior experience with the education of a team of assessors. During this training session, the POCUS experts are presented to the OSAUS score and the procedure for assessing the participants in this study. The POCUS experts are instructed to rate according to the standard expected from a GP, who is capable of performing POCUS unsupervised in general practice.
Self-rated scanning competence In relation to the expert assessment of scanning competence, participants were asked to fill out a questionnaire where they declare whether or not they have POCUS scanning competence within this scanning modality to perform the scan un-supervised in general practice. The expert assessors were blinded to this declaration.
Evaluation questionnaire After 12 months following the final MCQ-quiz and the final evaluation of scanning competence, we will circulate an evaluation questionnaire for participants. This questionnaire will be developed based on collected experiences during the study.
Data management The OSAUS assessment will be filled out by the expert assessors using paper versions of the registration sheet. The registration sheets and questionnaires will be collected by CAA and checked for completeness. CAA will bring the registration sheets to CAM AAU, where they will be safely stored in a locked cupboard. Two research assistants will independently impute the results in a Microsoft Excel fil along with the results from the MCQ-quiz. The two datasets will then be compared for inconsistencies and these will be resolved by involving a third party.
Hereafter, all digital study related data will be pseudo-anonymized and stored on secure server on Aalborg University and handled according to the General Data Protection Regulation.
Statistical methods for analysing primary and secondary outcomes. All statistical analyses will be performed using STATA version 17 (StataCorp, Texas, USA) and analyzed according to a predefined statistical analysis plan, which will be uploaded prior to the data collection for the primary outcome.
Sub-analyses will be performed to explore the importance of different educational elements in the intervention. These sub-analyses will be defined in the statistical analysis plan.
A predictive mean matching (pmm) imputation method will be used to fill in missing values by the command "mi impute pmm" in Stata , for all OSAUS scores after 3 and 6 months. Predictive mean matching (PMM) is a partially parametric method that matches the missing value to the observed value with the closest predicted mean .
Data monitoring
Harms
The participating GPs will be obliged to report any Suspected Unexpected Serious Adverse Reactions (SUSARs), Serious Adverse Events (SAEs) and Adverse Events (AEs) related or possibly related to the use of POCUS to the adverse event committee.
Ethics and dissemination The study will be performed in accordance with the Declaration of Helsinki. The project was notified to The North Denmark Region Committee on Health Research Ethics (Den Videnskabsetiske Komité for Region Nordjylland, reference number 2022-000764) who responded that according to Danish Law (komitélovens § 14, stk. 2), no ethical approval is needed for this project.
Prior to the collection of register data, we will seek permission from the Danish Regions The project has been registered and conducted according to the regulations of the Danish Data Protection Agency (registration number ID-242-2).
Access to data During the study, only CAA and a research assistant will have access to the data. After the pseudo-anonymization, the following authors will have access to collected data stored at the secure server at Aalborg University (SKA, MBJ and CAA).
Dissemination policy Positive, negative or inconclusive results of the study will be published in a peer-reviewed journal. The project group will also present results at conferences.
Inclusion criteria:
Exclusion criteria: