Neoadjuvant chemotherapy is an important part of the systematic treatment of breast cancer, and it is of great clinical significance to predict the efficacy of neoadjuvant chemotherapy in early stage. The emergence of multi-modal artificial intelligence model has brought new ideas for it. However, the limited ability of artificial intelligence to integrate multi-modal data, the lack of multi-modal models, and the insufficient level of evidence in clinical promotion of artificial intelligence are all scientific problems that need to be solved. In the early stage of the study, a variety of artificial intelligence accurate prediction and auxiliary diagnosis and treatment models for breast cancer were constructed based on magnetic resonance imaging and pathomics, etc., and the effectiveness of the models in predicting the curative effect of neoadjuvant chemotherapy for breast cancer was explored. In order to further improve the predictive efficiency of the model and fill the gap in the systematic study of multi-modal data fusion model, this clinical study intends to combine pathological images, magnetic resonance imaging, diagnostic report text and clinical variables to establish an artificial intelligence large language model based on multi-task and multi-modal data fusion to accurately predict the efficacy of neoadjuvant chemotherapy for breast cancer. A multicenter, bidirectional cohort study was conducted to explore the predictive effectiveness of the model.

This is a multicenter, bidirectional cohort study. Retrospective training cohort, retrospective validation cohort and prospective test cohort were designed.

Data of patients treated in the North Ward of Sun Yat-sen Memorial Hospital of Sun Yat-sen University from January 1, 2002 to August 31, 2023 were retrospectively collected for training cohort, and data of patients treated in the South ward of Sun Yat-sen Memorial Hospital of Sun Yat-sen University for internal validation cohort; Data on patients treated at external centers between January 1, 2002 and August 31, 2023 were retrospectively collected for external validation cohort. Data on patients admitted to Sun Yat-sen Memorial Hospital at Sun Yat-sen University after January 1, 2024 were prospectively collected for the test cohort. Patient data collected included: pathological images and report texts of breast puncture specimens before neoadjuvant chemotherapy, breast magnetic resonance images and report texts before neoadjuvant chemotherapy, postoperative pathological reports and clinical information, etc.. An artificial intelligence large language model based on multi-task and multi-modal data integration was established to accurately predict the efficacy of neoadjuvant chemotherapy for breast cancer, and its predictive efficacy was tested by retrospective validation cohort and prospective double-blind test cohort. The retrospective cohort of this study was followed up to collect clinical data, magnetic resonance imaging and reports, and surgical pathology reports of patients, etc.. When patients had disease recurrence, the DFS time of patients was recorded, and when patients did not have disease recurrence, the last follow-up time was recorded. Baseline data survey was completed during hospitalization of prospective cohort patients. Pathological reports of breast tumors surgically removed after neoadjuvant chemotherapy were obtained during follow-up, as well as the time of disease recurrence and the time of death of patients. Clinical information such as magnetic resonance imaging and reports were collected during follow-up. Follow-up until the end of the 2-year study.

Inclusion Criteria:

• Women
• Pathological diagnosis of non-metastatic invasive breast cancer (stage II-III)
• At least 4 cycles of neoadjuvant chemotherapy
• Radical surgery was performed after neoadjuvant chemotherapy
• There are pathological images and reports of breast puncture specimens before neoadjuvant chemotherapy
• 'There are MRI images and reports of breast MRI within 2 weeks before neoadjuvant chemotherapy
• There are standard clinical records

Exclusion Criteria:

• Inflammatory breast cancer
• Bilateral breast cancer
• Newly diagnosed stage IV breast cancer
• Other tumors have not been completely removed or less than 3 years after surgery
• Treatment other than neoadjuvant therapy had been performed before surgery

A total of 840 people will be enrolled in this study, including 300 patients treated in the North Ward of Sun Yat-sen Memorial Hospital of Sun Yat-sen University, 200 patients treated in the South Ward of Sun Yat-sen Memorial Hospital of Sun Yat-sen University, 170 patients treated in external centers. Prospective collection of data from 170 patients treated at Sun Yat-sen Memorial Hospital.