This randomized phase II trial studies how well veliparib or pembrolizumab work with combination chemotherapy and radiation therapy in treating patients with rectal cancer that has spread from where it started to nearby tissue or lymph nodes (locally advanced). Veliparib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Drugs used in chemotherapy, such as modified (m)FOLFOX6 regimen, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Radiation therapy uses high-energy x-rays to kill tumor cells and shrink tumors. Giving veliparib or pembrolizumab with combination chemotherapy and radiation therapy may kill more tumor cells, make the tumor smaller, and reduce the amount of normal tissue that needs to be removed.

PRIMARY OBJECTIVE:

I. To demonstrate an absolute improvement in neoadjuvant rectal cancer (NAR) score for the experimental regimen as compared to concurrently randomized control patients.

SECONDARY OBJECTIVES:

I. To compare overall survival (OS). II. To compare disease-free survival (DFS). III. To compare the rate of pathologic complete response (pCR) (nodes and tumor).

IV. To compare the rate of sphincter preservation.

TERTIARY OBJECTIVES:

I. To compare the proportion of patients who have a tumor resection overall, conditional on beginning induction chemotherapy, and conditional on beginning chemoradiotherapy.

II. To compare time from initiation of chemoradiotherapy to surgery in the subset of patients with tumor resection.

EXPLORATORY OBJECTIVES:

I. To estimate the rate of disease progression during chemotherapy (prior to chemoradiation).

II. To compare the rate of clinical complete response rate (cCR). III. To compare the rate of negative circumferential margin. IV. To compare the rate of completion of all cycles of neoadjuvant chemotherapy.

V. To compare the rate of completion of full course of chemoradiation. VI. To compare the toxicity and safety between interventions. VII. To explore the correlative molecular predictors of response and distant failure.

VIII. To explore the relationship between radiographic findings and pathologic outcomes.

OUTLINE: Patients are sequentially randomized to treatment arms, initially ARM I or ARM II and then later either ARM I or ARM III.

ARM I: Patients receive mFOLFOX6 regimen consisting of oxaliplatin intravenously (IV) over 2 hours, leucovorin calcium IV over 2 hours on day 1, and fluorouracil IV continuously over 46-48 hours on days 1-2. Treatment repeats every 2 weeks for 8 cycles in the absence of disease progression or unacceptable toxicity. 3-4 weeks after last does of mFOLFOX6 patient undergo radiation therapy (RT) and receive capecitabine orally (PO) twice daily (BID) Monday-Friday for 5 weeks in the absence of disease progression or unacceptable toxicity.

ARM II: Patients receive mFOLFOX6 regimen consisting of oxaliplatin IV over 2 hours, leucovorin calcium IV over 2 hours on day 1, and fluorouracil IV continuously over 46-48 hours on days 1-2. Treatment repeats every 2 weeks for 8 cycles in the absence of disease progression or unacceptable toxicity. 3-4 weeks after last does of mFOLFOX6 patient undergo RT and receive capecitabine PO BID and veliparib PO BID Monday-Friday for 5 weeks in the absence of disease progression or unacceptable toxicity.

ARM III: Patients receive mFOLFOX6 regimen consisting of oxaliplatin IV over 2 hours, leucovorin calcium IV over 2 hours on day 1, and fluorouracil IV continuously over 46-48 hours on days 1-2. Treatment repeats every 2 weeks for 8 cycles in the absence of disease progression or unacceptable toxicity. 3-4 weeks after last does of mFOLFOX6 patient undergo RT and receive capecitabine PO BID Monday-Friday for 5 weeks. They also receive pembrolizumab IV over 30 minutes every 3 weeks beginning on day 1 of RT for up to 6 cycles in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up at 30 days and every 6 months for up to 3 years.

Inclusion Criteria:

• The patient must have signed and dated an Institutional Review Board (IRB)-approved consent form that conforms to federal and institutional guidelines

• Eastern Cooperative Oncology Group (ECOG) performance status of 0, 1, or 2

• Diagnosis of adenocarcinoma of the rectum with the major portion of the tumor intact; Note: prior to randomization, the investigator must specify and document each of the following:

  • Distance of the lowest tumor margin from the anal verge; and
  • Intent for sphincter sparing or non-sphincter sparing surgical resection according to the primary surgeon; and
  • The majority of the untreated tumor must be < 12 cm from the anal verge or below the peritoneal reflection as determined by the treating surgeon

• The tumor must be clinically determined to be locally advanced stage II or stage III rectal cancer, and must also meet any ONE of the following criteria:

  • Distal location (as defined by measurement on magnetic resonance imaging [MRI], transrectal ultrasound [ERUS]/pelvic computed tomography [CT] [with IV contrast] scan or palpable on digital rectal examination [DRE]): cT3-4 =< 5 cm from the anal verge, any N

  • Bulky: any cT4 or evidence that the tumor is adjacent to (defined as within 3 mm of) the mesorectal fascia on MRI or ERUS/pelvic CT (with IV contrast) scan

  • High risk for metastatic disease with 4 or more regional lymph nodes (cN2); clinical nodal or "cN" status for eligibility includes the total number of nodes (N2 = 4 or more) in the mesorectal and superior rectal stations measuring >= 1.0 cm in any axis on cross sectional or endoscopic imaging; Note: nodes must measure 1.0 cm or greater to be considered positive for this eligibility requirement

  • Not a candidate for sphincter-sparing surgical resection prior to neoadjuvant therapy (as planned by the primary surgeon)

    • Note: clinical stage of the primary tumor and nodes may be determined locally by rectal endoscopic ultrasound or pelvic MRI (pelvic MRI is strongly preferred); CT scan with IV contrast is acceptable provided there is evidence of T4 and/or N2 disease

• Patients must have the ability to swallow and retain oral medication

• Absolute neutrophil count (ANC) must be >= 1200/mm^3 (within 28 days before randomization)

• Platelet count must be >= 100,000/mm^3 (within 28 days before randomization)

• Hemoglobin must be >= 10 g/dL (within 28 days before randomization)

• Total bilirubin must be =< ULN (upper limit of normal) for the lab unless the patient has a bilirubin elevation > ULN to 1.5 x ULN due to Gilbert's disease or similar syndrome involving slow conjugation of bilirubin (within 28 days before randomization)

• Alkaline phosphatase must be =< 3 x ULN for the lab (within 28 days before randomization)

• Aspartate aminotransferase (AST) must be =< 3 x ULN for the lab (within 28 days before randomization);

  • Note: if alanine aminotransferase (ALT) is performed instead of AST (per institution's standard practice), the ALT value must be =< 3 x ULN; if both were performed, the AST must also be =< 3 x ULN; if AST and/or ALT is >= ULN but =< 3 x ULN, serologic testing for hepatitis B and C must be performed and results for viral infection must be negative

• Serum creatinine =< ULN for the lab and measured or calculated creatinine clearance > 60 mL/min (within 28 days before randomization)

• Serum potassium, magnesium, and calcium levels within 28 days before randomization must be within normal limits (WNL) for the lab

• International normalized ratio of prothrombin time (INR) within 28 days before randomization must be =< ULN for the lab; patients who are therapeutically treated with an agent such as warfarin may participate if they are on a stable dose and no underlying abnormality in coagulation parameters exists per medical history

• Patients with acquired immunodeficiency syndrome (AIDS-related illnesses) or known human immunodeficiency virus (HIV) disease must:

  • Have a cluster of differentiation (CD)4 count >= 200 cells/uL within 30 days before beginning study therapy
  • Be off all antiretroviral therapy (prophylaxis/treatment) more than 60 days before beginning study therapy, and
  • Have no evidence of opportunistic infections

• Pregnancy test (urine or serum beta-human chorionic gonadotropin [HCG]) done within 72 hours before randomization must be negative (for women of childbearing potential only); if urine pregnancy results are positive or cannot be confirmed as negative, a serum pregnancy test will be required

Exclusion Criteria:

• Rectal cancer histology other than adenocarcinoma (i.e., sarcoma, lymphoma, squamous cell carcinoma, mucosal melanoma, etc.)

• Definitive clinical or radiologic evidence of metastatic disease; required imaging studies must have been performed within 28 days prior to randomization; Note: Distant clinical staging to exclude patients with overt metastatic disease is determined by:

  • Chest: CT scan (preferred); chest x-ray posterioranterior (PA) and lateral (acceptable); or positron emission tomography (PET) scan (acceptable)

  • Abdomen: CT scan with IV contrast (preferred); or MRI (acceptable)

  • Pelvis: MRI (preferred) or CT scan with IV contrast (acceptable)

    • (It is recommended that the same imaging tests that are performed before randomization be used at follow-up time points; Note: CT scans of the abdomen and pelvis must be performed with IV contrast)

• History of prior invasive rectal malignancy, regardless of disease-free interval

• Cardiac disease that would preclude the use of any of the drugs included in the GI002 treatment regimen; this includes but is not limited to:

  • Clinically unstable cardiac disease, including unstable atrial fibrillation, symptomatic bradycardia, unstable congestive heart failure, active myocardial ischemia, or indwelling temporary pacemaker
  • Ventricular tachycardia or supraventricular tachycardia that requires treatment with class Ia antiarrhythmic drugs (e.g., quinidine, procainamide, disopyramide) or class III antiarrhythmic drug (e.g., sotalol, amiodarone, dofetilide); use of other antiarrhythmic drugs is permitted
  • Second- or third-degree atrioventricular (AV) block unless treated with a permanent pacemaker
  • Complete left bundle branch block (LBBB)
  • History of long QT syndrome
  • Corrected QT (QTc) >= 450 ms

• Sensory or motor neuropathy >= grade 2

• History of, or any evidence of active, non-infectious pneumonitis

• Active inflammatory bowel disease (i.e., patients requiring current medical interventions or who are symptomatic) or have a history of abdominal surgery or other medical condition that may, in the opinion of the treating physician, interfere with gastrointestinal motility or absorption

• Active autoimmune disease that has required systemic treatment within the past 2 years (i.e., with use of modifying agents, corticosteroids, or immunosuppressive drugs); replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment

• History of active TB (Bacillus tuberculosis)

• Active or chronic infection requiring systemic therapy

• Diagnosis of immunodeficiency or receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of study therapy; the use of physiologic doses of corticosteroids may be approved after consultation with the study principal investigator (PI)

• Active seizure disorder uncontrolled by medication

• Any antineoplastic therapy for this cancer before randomization

• Synchronous colon cancer

• Other invasive cancer within 5 years before randomization; exceptions are colonic polyps, non-melanoma skin cancer or carcinoma-in-situ of the cervix

• Antineoplastic therapy (e.g. chemotherapy or targeted therapy) for other invasive cancer within 5 years before randomization; (for the purposes of this study, hormonal therapy is not considered chemotherapy)

• Prior treatment with an investigational compound being tested in this study (e.g., poly ADP ribose polymerase [PARP] inhibitor, anti-PD-1, anti-PD-L1, or anti-PD-L2)

• Receipt of live vaccination within 28 days before randomization; seasonal flu vaccines that do not contain live virus are permitted

• Major surgery within 4 weeks before randomization

• Any therapeutic pelvic radiation

• Known homozygous DPD (dihydro pyrimidine dehydrogenase) deficiency

• Any of the following because this study involves agents that have known or potential genotoxic or mutagenic, and teratogenic effects:

  • Pregnant women
  • Nursing women who are unwilling to discontinue nursing
  • Men or women of childbearing potential who are unwilling to employ adequate contraception (e.g., hormonal or barrier method of birth control; abstinence) for the duration of study treatment and for 4 months after the last dose of study therapy

• Co-morbid illnesses or other concurrent disease that, in the judgement of the clinician obtaining informed consent, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens or prevent required follow-up