Scientific Validity Assessment and Optimization of AI-Generated A3/A4 Type Questions for the Chinese Medical Licensing Examination

Scientific Validity Assessment and Optimization of AI-Generated A3/A4 Type Questions for the Chinese Medical Licensing Examination: An Empirical Analysis Based on the "Answering-Generating" Closed Loop and Transformation of Competency Assessment

This is a cross-sectional study that primarily employs quantitative analysis, supplemented by qualitative assessment. The research is conducted in two stages: Phase I consists of a model performance comparison experiment, and Phase II involves an item quality evaluation experiment. The entire study adheres to the principles of single-blinding, randomization, and standardization to ensure scientific rigor and reproducibility.

The single-blind design is implemented during the "standardized testing" phase, where the system intersperses AI-generated items with those authored by human experts. Participants remain blinded to the source of each item (AI-generated vs. human-authored) throughout the testing and scoring processes, thereby ensuring the objectivity of the evaluation results.

Inclusion Criteria:

• 1.Professional Status: Medical students currently enrolled in a Standardized Residency Training (SRT) program.

  2.Educational Background: Holders of a Bachelor of Medicine degree or higher, with foundational clinical knowledge.

  3.Informed Consent: Voluntarily participate in the study and provide written informed consent.

  4.Technical Competency: Proficient in using digital platforms to complete assessments and scoring.

Exclusion Criteria:

• 1.Conflict of Interest: Individuals involved in the AI model training, prompt engineering, or the creation of the human-authored question bank for this study.

  2.Inability to Complete: Presence of visual/auditory impairments or severe illness that precludes completion of the assessment within the specified time.

  3.Investigator's Discretion: Any other condition that, in the opinion of the investigator, renders the participant unsuitable for the study.