Testing an AI Tool to Help Primary Care Clinicians With Specialty Consultation Questions

Physician Usability and Output Quality Evaluation of a Retrieval-Augmented Language Model System for Specialty Medical Consultation: A Human-Computer Interaction Study

The goal of this study is to test an artificial intelligence (AI) tool called SAGE. SAGE helps primary care doctors with questions that often need a specialist. Primary care doctors are the doctors people usually see first. SAGE reviews a case and suggests what a specialist might advise.

The main questions it aims to answer are:

• Do doctors make sound, timely care decisions when they use SAGE?
• Do those decisions match what a specialist would advise?

Researchers will compare the decisions doctors make with and without SAGE.

Doctors in the study will:

• Review made-up patient cases (these are not real patients)
• Make a decision for each case, their usual way and with SAGE

Inclusion Criteria:

• Physician practicing in primary care internal medicine and/or family medicine
• Attending physician, or resident physician at postgraduate year 2 (PGY-2) or higher
• Stanford-affiliated (faculty, staff, or trainee)
• Able to complete a single, approximately 60-minute remote (Zoom) session conducted in English

Exclusion Criteria:

• Not currently practicing in primary care internal medicine or family medicine
• Member of the study team or otherwise involved in the development of SAGE
• Unable to complete the remote study session