Development of a New Canadian Endourology Group Stent Symptom Score

Canadian Endourology Group Stent Symptom Score: Development and Validation of A New Questionnaire

Placement of a ureteral stent is a common urological intervention. For decades there was no valid measures available to assess quality of life issues in patients with ureteral stents, which has hampered the understanding of such symptoms and their true impact. In order to improve the outcomes associated with the placement of a stent, a validated tool is needed to measure its impact and the amount of undesirable effects it produces on patients requiring the placement of a stent. In 2003, the team of the Bristol Urological Institute developed a validated questionnaire called the: URETERAL STENT SYMPTOM QUESTIONNAIRE (USSQ). The questionnaire contains 38 items included in 6 sections. Despite the obvious need of a validated questionnaire, the latter remains unused by the community of peer urologists. Many of urologists consider it too long to be used in clinical practice and even for research purposes. This issue motivated the Canadian Endourology Group (CEG) to work collaboratively on the development of the CANADIAN ENDOUROLOGY GROUP STENT SYMPTOM SCORE (CEGSSS) in order to provide clinicians with a more useful and validated tool. To fulfill this objective, the CEG proceeds in three phases. Phase 1. A systematic, deliberative, and participatory approach mostly through face to face meetings, including patients, clinicians, and researchers in the field of Endourology to identify a minimum needs-based set of domains and items that are, clinically relevant to be included in the CEGSSS in order to ensure optimal uptake in the clinical setting. Phase 2. A pilot study to assess feasibility/acceptability and further refine the proposed set of items selected in phase 1 of the study. Phase 3. A multicentric prospective study to evaluate the validity, reliability and sensitivity to change of the CEGSSS.

This research project is conducted by the Canadian Endourology Group (CEG), a panel of experts in the field of endourology in Canada. The CEG is a national member-based organization dedicated to enabling the profession to provide the highest possible standards of endourological care and to advance the science of endourology by collaboratively:

1. Fostering excellence in endourological practice through advocacy, education, research and practice support tools,
2. Leading evidence-based clinical practice through the development of practice standards and guidelines in endourology,
3. Providing continuous professional development for Canadian endourologists along the career-path continuum,
4. Providing leadership in public education for endourological conditions.

This research project is conducted in 3 phases, phase 1 and 2 have been completed and phase 3 is under preparation and will be submitted to the Research Ethics Board soon.

Phase 1: Prioritizing Domains/Items to be included in a minimal needs data set

Objectives:

1. Identify the Domains/Items to be included in the minimal needs-based data set for assessing symptoms related to stent insertion among those used in the USSS.

2. Identify other items to include in the CEGSSS.

3. Reach a final consensus on the Domains/Items to be included in the minimal data set that will constitute the new CEGSSS.

   Methodology A systematic deliberative and participatory approach is used to meet the above objectives.

   Phase 1 is composed of two steps

   Step I: Procedures and data analysis Provided that the USSS questionnaire is a validated questionnaire developed to specifically address the outcomes related to the stent placement, that questionnaire was used as a starting point to identify the Domains/Items to be included in the minimal needs-based data set for assessing symptoms related to stent insertion through those used in the USSQ. A committee constituted of 21 patients and 9 CEG experts worked to address the items of each section of the USSQ in order to assess the importance of each item in determining the impact of having a stent The consultations were done through face to face meetings. Each item (question) was rated using a scale from 1 to 5. Patients and clinicians took into consideration if the item was absolutely essential and clinically useful. The question was accepted if mean patient rating was ≥4.0 with Standard Deviation (SD) ≤0.75 OR if mean expert rating was ≥4.0 with SD ≤0.75. Questions with mean patients or mean experts rating of ≥4.0 with SD ≥ 0.75 were flagged for discussion with experts during the second phase of consultations. If the means and SD didn't respect the criteria, the question was rejected.

   the following summarizes the results of phase I consultation by domain:

   • Urinary symptoms domain (11 questions):
   • 9 questions accepted
   • Pain domain (9 questions):
   • 8 questions accepted
   • 1 question to be discussed
   • General Health domain (6 questions):
   • All questions were rejected. They all had means < 4.00 for both experts and patients.
   • Work performance domain (7 questions):
   • All questions were rejected. They all had means < 4.00 for both experts and patients.
   • Sexual matters domains (4 questions):
   • All questions were rejected. They all had means < 4.00 for both experts and patients.
   • Additional problems domain (6 questions):
   • 4 questions accepted

   Step II: Procedures and data analysis The accepted and flagged questions go through another evaluation using expert's consensus meetings involving 9 experts. All the experts involved were members of the CEG. Five rounds of consultations were necessary to achieve the final CEGSSS. Most of the questions were labeled differently, some of the questions initially rejected in phase I were reintroduced, and other questions were abandoned The score benefited from the adding of a domain on quality of life including two questions. The final CEGSSS contains 15 questions divided in 3 domains: Urination, Pain and quality of life. Each item included in the questionnaire was agreed by all experts, which mean 100% in terms of % of agreement.

   Phase 2: Feasibility and acceptability of implementing the CEGSSS

   Based on the results of the phase 1, an initial draft of the CEGSSS was developed. This version including 15 questions divided in 3 domains has been tested in a prospective pilot study to evaluate the feasibility and acceptability of the CEGSSS among patients before starting the multicentric validation study of the score. The pilot study recruited patients from 4 centers; the Centre hospitalier de l'Université de Montréal (CHUM), the University Health Network- Toronto General Hospital (UHN-TGH), the Mc Gill University Healthcare Center (MUHC) and the Stone Center, Vancouver General Hospital (VGH)

   Methodology Based on the experience the investigators have with this kind of studies, anticipated need of 5 rounds of 5 patients to achieve the final version of the CEGSSS, therefore a sample size of 25 patients was planned. Participants for whom a ureteral stent was inserted during their intervention at the urology unit of the participating centers were approached.

   Procedures Patients for whom a ureter stent was inserted during their intervention at the urology unit were offered to participate to the study. Patients who agree to participate had to complete the CEGSSS and a short survey, that asks about their opinion about problems or ambiguity related to the wording and clarity of the questions as well as if there are important missing information or other issues to be addressed in the questionnaire. The study documents were identified using a Patient ID. Patients completed the documents at some point between the stent placement appointment and the stent removal appointment If they forgot to do so, they had the chance to complete them at the urology unit the day of their appointment for stent removal A prospective design involving iterative cycles have been used. At each cycle, patient's responses have been reviewed and the CEGSSS revised according to the data and feedback collected in the Patient Survey. Each cycle involved 5 patients at a time and this procedure have be repeated until minimal or no negative feedback is obtained and a final version of the CEGSSS is reached. This study needed 3 iterative cycles of testing to achieve data saturation.

   Phase 3: ***PHASE 3 OF THE PRESENT RESEARCH PROJECT IS UNDER DEVELOPMENT, THE FOLLOWING IS A SUMMARY OF IT.

   Objectives:

   • A multicentric prospective study to evaluate the validity, reliability and sensitivity to change of the CEGSSS.
   • A multicentric prospective study to evaluate the validity and reliability of the French version of the CEGSSS.

   Methodology:

   Multicentric study : 6 centers in Canada and 2 centers in the US Sample size: The final sample size will be driven by Factor Analysis which requires the largest sample size as compared to the other psychometric assessments needed to validate the questionaire. N=180 patients

   Questionnaire administration :

   • CEGSSS (n=180)
   • At baseline: 7 days after stent insertion
   • At follow-up: 14 days after stent insertion and before removal
   • USSQ (n=50)
   • At follow-up: 14 days after stent insertion and before removal

   Psychometric assessments :

   • Internal consistency
   • Test-Retest Reliability
   • Exploratory Factor Analysis (EFA)
   • Validity

1. Internal consistency assessment:

   Verify the homogeneity of the questions globally and within each domain

   • Objective: Assess if all the questions in each domain are moving in the same direction.

   This will allow us to correct questions that are inversely correlated with other questions. - Data: internal consistency will be examined on both baseline and follow-up data (n=180*2)

   • Analysis:
   • Cronbach's α calculation (varies on a scale of -1 to 1)
   • For dichotomous items (yes/no) a Kuder-Richardson-20 coefficient (r) will be calculated
   • A Cronbach's α /Kuder-Richardson coefficient ≥ 0.70 will be considered acceptable (good internal consistency)

2. Test-Retest Reliability The questionnaire will be administered to the same patient at different time points with the ureteral stent in place (5-7 days and 10-14 after stent insertion)

   - Objective: Comparison of results of the same individual when it is submitted at different times to under identical experimental conditions

   - Data: Analysis will be done by comparing baseline to follow-up data (n=180 *2)

   - Analysis:

   - Instrument's fidelity will be estimated from the Intra-class Correlation Coefficient (ICC)

   - An ICC > 0,8 would be considered acceptable (good test retest reliability)

3. Exploratory Factor Analysis (EFA) Allows for the extraction of confirmatory factors from the domains and can be used to narrow down the questionnaire

   - Objective: Ensure that the items contribute to the expected dimensions Make a global score or sub-score per domain

   - Sample size calculation : 10:1 (i.e. 10 participants for each item of the CEGSSS) Number of items including subitems= 18 n=180 (10*18)

   - Data: internal consistency will be assessed on both baseline and follow-up data (n=180*2)

   - Analysis: the following measures will be calculated as part of EFA:

   -Factor loadings : correlation between the item and the dimension defined (no item should to highly correlate with more than one factor)

   - Factor loadings > 0.3 will be considered acceptable

   - Explained variance of the measured construct: 50% of explained variance will be considered acceptable

   - Eigenvalues : each dimension should have an eigenvalue > 1 per factor

4. Validity test:

   Compare the results of the CEGSSS questionnaire to the results of a questionnaire that measures the same concept (the USSQ) - Objective: Ensure each domain measures the intended concept for which it was developed. - Data: Analysis will be done on a subset of follow-up data (n=50) - Analysis: - The correlation between the CEGSSS and the USSQ will be measured using: - Pearson correlation if data are normally distributed - Spearman rank correlation if data are not normally distributed

6) French validation of the CEGSSS: The CEGSSS will undergo Forward and backward translation first and then validation in the setting of a multicentric study

A) Forward translation:

The initial questionnaire will be translated into French by two native French speakers:

1 from the institution who knows the intended use of the CEGSSS well and

1 professional translator

Both translations will be compared and discussed between:

the two translators and a 3rd person who is not involved in the translation process The version resulting from this review process will undergo the "Backward translation" process

B) Backward translation:

The translation resulting from the Forward translation process will be translated back into English by two native English speakers:

1 from the institution who knows the intended use of the instrument but not the CEGSSS itself and

1 professional translator Both translations will be compared and discussed between the two translators and a 3rd person who is not involved in the translation process The version resulting from this review process will be compared to the original English version

C) Comparison and agreement process:

The English version resulting from the Forward-Backward translation process will be compared to the original English version

Differences will be discussed by a committee of experts composed of :

All CEG experts involved in the development of the questionnaire A methodologist The translators involved in the forward-backward translation process The differences between the two translations will be discussed and modifications to the French version and/or to the English version will be made accordingly.

A consensus of all committee members must be reached in order to validate the French version

D) Test-Retest Reliability:

The French version will undergo the Test- Retest reliability analysis in the same way as the English version Multicentric study (2 centers ) Inclusion exclusion criteria will be the same except for the language

The French CEGSSS will be administered to 180 patients at two different times:

• 7 days after stent insertion
• 14 days after stent insertion and before stent removal Instrument's fidelity will be estimated from the Intra-class Correlation Coefficient (ICC) An ICC > 0,8 would be considered acceptable (good test-retest reliability)

Ethical considerations Phase1: Participating patients were completely free to take part or not in Phase I of the present project. Patients were approached and invited to participate when they came for their appointments. Relevant information about the objectives and procedures of the consultation was provided during their appointment, they were also informed of the intention which was to create a new questionnaire. Then, the potential participants were asked if they accept or not to be part of the patient's staff. If they answered yes, this constituted their consent and they did not have to sign a formal informed consent form.

Participating experts were all members of the CEG. The members of the CEG are involved in all the projects initiated by CEG, therefore, they consented to participate in the consensus process when they agreed to join the group.

Phase 2: There is no physical or psychological harm associated with this study. Less than 20 minutes are required to complete the CEGSSS and Patient Survey. It is possible that some patients may feel slightly uncomfortable answering certain questions, but had the option to refuse to answer. If need be, they can contact the Study (SC) coordinator for support and discussion. Furthermore, the participants have been given the opportunity to report in the Patient Survey which question(s) was (were) upsetting/inappropriate to them. In line with the hospital policy, the Principal Investigators (PI) report to their local Research Ethics Board any adverse events occur. Participants have been informed that refusal to participate in the study or withdrawing from it will not affect in any way the quality of care they would receive, nor will it jeopardize their relationship with their health care professionals.

All patients' nominal data, contact information, medical record number, and Patient ID are stored on the secured files of each institution and each center's nominal data will be accessible only to their research team. Research data will be collected using de-identified forms. The CHUM's PI and SC will both be able to access all research data. At the end of the study, data will be send to the CHUM and stored on a secure file protected by a password on the CHUM network drive of Dr. Naeem Bhojani. Data will be kept for a period of 10 years.

Funding: This project is financed by the funds of Dr. Naeem Bhojani.

Conflict of interest: None

Impact: Developement of a shorter validated ureteral stent symptom questionnaire for clinical as well as research objectives

Inclusion Criteria:

• Age ≥ 18 years
• Patient with ureteral stent
• Ability to communicate in English

Exclusion Criteria:

• Inability to provide an informed consent due to physical or mental inability
• Active malignancy
• Obstruction
• Long-term stent (>2 weeks)