The purpose of this study is to evaluate the safety and clinical activity of cemiplimab and the combination of cemiplimab/fianlimab in microsatellite unstable localized or locally advanced colorectal cancer diagnosed in patients age 70 or greater or in patients age 18 or greater considered poor candidates for surgery or unwilling to undergo surgery.

Inclusion Criteria for Cohort A and B:

• Age ≥70 years.
• Eastern Cooperative Oncology Group (ECOG) performance status 0 -2
• Have histologically proven localized or locally advanced mismatch repair deficient (dMMR) or microsatellite unstable (MSI-H) colorectal cancer.
• Must not have received any prior systemic treatment or radiation.
• Must be agreeable to endoscopic, and CT surveillance for a total of 24 months.
• Patient's acceptance to have a tumor biopsy.
• Patients must have adequate organ and marrow function defined by study-specified laboratory tests and procedures.
• LVEF assessment with documented LVEF ≥ 45% by either TTE or MUGA (TTE preferred) within 6 months from first study drug administration.
• For both Women and Men, must use acceptable form of birth control while on study.
• Ability to understand and willingness to sign a written informed consent document.

Inclusion Criteria for Cohort C and D:

• Age ≥18 years.
• Eastern Cooperative Oncology Group (ECOG) performance status 0 -3.
• Have histologically proven localized or locally advanced mismatch repair deficient (dMMR) or microsatellite unstable (MSI-H) colorectal cancer.
• Patient deemed a poor surgical candidate after evaluation by a surgeon or unwilling to undergo surgery.
• Must not have received any prior systemic treatment or radiation.
• Must be agreeable to endoscopic, and CT surveillance for a total of 24 months.
• Patient's acceptance to have a tumor biopsy.
• Patients must have adequate organ and marrow function defined by study-specified laboratory tests and procedures.
• For both Women and Men, must use acceptable form of birth control while on study.
• Ability to understand and willingness to sign a written informed consent document.

Exclusion Criteria for Cohorts A and B:

• Have received an investigational agent or used an investigational device within 28 days of the first dose of study drug.
• Have expected to require any other form of systemic or localized antineoplastic therapy while on study.
• Have had surgery within 28 days of dosing of investigational agent, excluding minor procedures (dental work, skin biopsy, etc.).
• History of prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CTLA4, or anti-Lag-3 antibodies for any reason in the 5 years proceeding their colorectal cancer diagnosis.
• Currently using any chronic systemic steroids.
• Patient has received a live vaccine within 30 days of the first dose of study drug.
• History of severe hypersensitivity reaction to any monoclonal antibody.
• Uncontrolled intercurrent illness including, but not limited to, uncontrolled infection, symptomatic congestive heart failure, unstable angina, cardiac arrhythmia, metastatic cancer, or psychiatric illness/social situations that would limit compliance with study requirements.
• Active autoimmune disease.
• Any tissue or organ allograft, regardless of need for immunosuppression, including corneal allograft.
• Patient has a pulse oximetry of <92% on room air.
• Patient is on supplemental home oxygen.
• Has clinically significant heart disease.
• Cohort B Only: Troponin T (TnT) or troponin I (TnI) > 2x institutional ULN at baseline.
• Conditions, including alcohol or drug dependence, intercurrent illness, or lack of sufficient peripheral venous access, that would affect the patient's ability to comply with study visits and procedures.
• Unwilling or unable to follow the study schedule for any reason

Exclusion for Cohort C and D:

• Have received an investigational agent or used an investigational device within 28 days of the first dose of study drug.
• Have expected to require any other form of systemic or localized antineoplastic therapy while on study.
• Currently using any chronic systemic steroids.
• Patient has received a live vaccine within 30 days of the first dose of study drug.
• History of severe hypersensitivity reaction to any monoclonal antibody.
• Any tissue or organ allograft, regardless of need for immunosuppression, including corneal allograft.
• Patient is pregnant or breastfeeding.
• Cohort D Only: Troponin T (TnT) or troponin I (TnI) > 2x institutional ULN at baseline.
• Conditions, including alcohol or drug dependence, intercurrent illness, or lack of sufficient peripheral venous access, that would affect the patient's ability to comply with study visits and procedures.
• Participation deemed not in the best interest of the patient.
• Unwilling or unable to follow the study schedule for any reason.