Eligible patients with estrogen receptor positive breast cancer will undergo a biopsy and be randomized to receive endocrine therapy (ET) versus endocrine therapy with palbociclib (PET) in a 1:2 ratio. After 1 cycle (28 days) another biopsy will be obtained, and both arms will receive avelumab (A) for 3 additional cycles. Patients will then undergo breast surgery.

Patients will be treated with endocrine therapy +/- palbociclib (125mg, taken orally, daily for 21 days, 7 days off ) for 1 cycle (1 cycle =28 days) and undergo a repeat biopsy, MRI, and blood draw at Cycle 2, Day 1. Afterwards avelumab will be added to both arms. Avelumab will be administered 10mg/kg IV every 14 days (1 cycle = 2 doses = 28 days). Patients will be treated for 3 cycles of avelumab with endocrine therapy +/- palbociclib (thus 4 cycles total, including run-in without avelumab).

The kind of endocrine therapy given will depend on menopausal status and will be a standard treatment. For premenopausal women, the endocrine therapy will be daily tamoxifen (20mg taken orally daily for 28 days), plus either Leuprolide (3.75mg taken intramuscularly) or Goserelin (3.6mg taken subcutaneously) on Day 1 of each cycle. For postmenopausal women, it will be daily letrozole (2.5mg taken orally).

Patients will be treated so long as there is no clinical evidence of progression and therapy is tolerated. Patients who experience progressive disease (25% increase) of their tumor will cease study treatment and undergo end-of-study assessment (including repeat MRI) and surgery. Otherwise, patients completing all 4 cycles of therapy will undergo MRI and surgery.

Inclusion Criteria:

• Stage II-III ER-positive breast cancer
• Tumor evaluable either by ultrasound or by touch.
• Age ≥ 18 years.
• Eastern Cooperative Oncology Group performance status of 1 or less.
• Adequate organ and bone marrow function within 28 days prior to registration.
• Females of child-bearing potential and males must agree to use adequate contraception prior to study entry, for the duration of study participation, and for 30 days following completion of therapy.
• Females of child-bearing potential must have a negative pregnancy test within 7 days prior to registration on study.
• Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study

Exclusion Criteria:

• Evidence of metastatic disease or inflammatory breast cancer.
• Patients not felt to be sensitive to endocrine therapy, such that a neoadjuvant endocrine-based approach would not be appropriate (i.e. PR-negative, high grade/Ki67, high gene expression profile, clinically aggressive presentation)
• Previous treatment with endocrine therapy within the last 10 years (i.e. tamoxifen, aromatase inhibitors).
• Previous treatment with CDK4/6 inhibitors, or immune checkpoint inhibitors.
• Use of SSRIs and/or any concomitant use of medications outlined in section 6.5 within 4 days prior to enrollment. If patients are on stable doses and there are no good alternatives, the treating physician may discuss with Study Chair.
• May not be receiving any other investigational agents.
• May not be receiving immunosuppressive therapy within 2 weeks of study entry.
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to corresponding endocrine therapy (tamoxifen, aromatase inhibitors, GnRH agonists), palbociclib, and avelumab are not eligible.
• May not have had a prior diagnosis of cancer if it has been < 3 years since their last treatment (with the exception of squamous cell carcinoma or basal cell carcinoma of the skin or cervical intraepithelial neoplasia). NOTE: Patients with a history of breast cancer or breast cancer treatment within the last 10 years are also excluded. Any previous radiation to affected breast is excluded.
• Autoimmune disease within the last 3 years with the exception of: Vitiligo or alopecia; Hypothyroidism on stable doses of thyroid medication; and Psoriasis not requiring systemic therapy
• Uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible: Ongoing or active infection requiring systemic treatment (including HIV, TB, hepatitis viruses), symptomatic congestive heart failure, cerebral vascular accident/stroke (< 6 months prior to enrollment), myocardial infarction (< 6 months prior to enrollment), unstable angina pectoris, cardiac arrhythmia, psychiatric illness/social situations that would limit compliance with study requirements, any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints
• Other severe acute or chronic medical conditions including immune colitis, inflammatory bowel disease, immune pneumonitis, pulmonary fibrosis or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior; or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study
• Vaccination within 4 weeks of the first dose of avelumab and while on trials is prohibited except for administration of inactivated vaccines
• Prior organ transplantation including allogenic stem-cell transplantation
• Female patients who are pregnant or nursing are not eligible.