This is a single-institution, prospective phase II trial with an initial safety run-in to evaluate the efficacy and safety of neoadjuvant pembrolizumab combined with chemoradiotherapy and adjuvant pembrolizumab in patients with locally advanced esophageal and gastric cancers (EGC). Chemoradiation therapy (45Gy in 25 fractions with concurrent, weekly carboplatin [AUC 2] and paclitaxel [50mg/m2 of BSA]) with three cycles of pembrolizumab will be administered as neoadjuvant therapy. These patients will also receive three cycles of adjuvant pembrolizumab after surgical resection

Enrolled patients will receive three doses of neoadjuvant pembrolizumab (200 mg administered as an intravenous infusion over 30 minutes every 3 weeks). The first dose of pembrolizumab will be administered approximately 14 days prior to initiating radiotherapy. The second dose will be administered three weeks later (week 1 of chemoradiation). The third dose will be administered 3 weeks later (week 4 of chemoradiotherapy). Pembrolizumab will be given every 3 weeks and may be given at the same time as systemic therapy. All patients will receive radiation treatment (45Gy in 25 fractions at 1.8 Gy/fraction) using image-guided radiation therapy with concurrent, weekly carboplatin (AUC 2) and paclitaxel (50mg/m2 of BSA). Restaging will be performed per standard of care approximately 4-8 weeks after completing chemoradiotherapy. Resection will be performed approximately 6-12 weeks after completing chemoradiotherapy per standard of care. Postoperatively, three additional cycles of pembrolizumab (200 mg every 3 weeks) will be administered as adjuvant therapy.

Inclusion Criteria:

1. Be willing and able to provide written informed consent for the trial.
2. Be 18 years of age or older on day of signing informed consent.
3. Has a pathologic diagnosis of invasive esophageal, gastroesophageal or gastric adenocarcinoma.
4. Staging CT CAP or PET/CT shows no evidence of metastatic disease.
5. Have a performance status of 0-2 on the ECOG Performance Scale.
6. Plan for neoadjuvant chemoradiation.
7. Demonstrate adequate organ function as defined in the study protocol, all screening labs should be performed within 14 days of treatment initiation.
8. Female subject of childbearing potential should have a negative serum pregnancy within 48 hours prior to receiving the first dose of study medication.
9. Female and male subjects of childbearing potential must be willing to use an adequate method of contraception as outlined in the Duke Contraception Policy.

Exclusion Criteria:

1. Is currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigational device within 4 weeks of the first dose of treatment.
2. Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment.
3. Has a known history of active TB (Bacillus Tuberculosis)
4. Hypersensitivity to pembrolizumab or any of its excipients.
5. Has had a prior anti-cancer monoclonal antibody (mAb) within 4 weeks prior to study Day 1 or who has not recovered (i.e., ≤ Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier.
6. Has had prior chemotherapy, targeted small molecule therapy, or radiation therapy for the current diagnosis of EGC.
7. Has a known additional malignancy that is progressing or requires active treatment. Exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer. (all patients with prior radiotherapy must be reviewed by the PI to determine if patient is eligible).
8. Has known metastatic disease. Staging CT C/A/P or PET/CT will be mandatory no more than 45 days prior to enrollment to evaluate for the presence of metastatic disease.
9. Has unresectable disease or is medically inoperable.
10. Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with chronic use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment.
11. Has known history of, or any evidence of active, non-infectious pneumonitis.
12. Has an active infection requiring systemic therapy.
13. Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the subject's participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator.
14. Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial.
15. Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment.
16. Has received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent.
17. Has a known history of Human Immunodeficiency Virus (HIV) (HIV 1/2 antibodies).
18. Has known, active Hepatitis B (e.g., HBsAg reactive) or Hepatitis C (e.g., HCV RNA [qualitative] is detected).
19. Has received a live vaccine within 30 days of planned start of study therapy. Note: Seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist®) are live attenuated vaccines, and are not allowed.
20. Has a diagnosis of scleroderma.
21. Has a known history of allogenic stem cell transplant
22. Has received a solid organ transplant

Neoadjuvant: from the start of neoadjuvant therapy until surgery, approximately 20 weeks Postoperative: within 30 days after surgery Adjuvant: from the start of adjuvant therapy until 90 days after the last dose of adjuvant therapy

Serious adverse events are defined as Grade 3+ adverse events leading to hospitalization