RATIONALE: Drugs used in chemotherapy, such as capecitabine, fluorouracil, and oxaliplatin work in different ways to stop tumor cells from dividing so they stop growing or die. Radiation therapy uses high-energy x-rays to damage tumor cells.

PURPOSE: This randomized phase III trial is studying radiation therapy and either capecitabine or fluorouracil with or without oxaliplatin and comparing them to see how well they work when given before surgery in treating patients with resectable rectal cancer. It is not yet known whether radiation therapy and either capecitabine or fluorouracil is more effective with or without oxaliplatin in treating rectal cancer.

OBJECTIVES:

Primary

• Compare the rate of local-regional relapse in patients with resectable rectal cancer treated with chemoradiotherapy comprising radiation therapy and either capecitabine or fluorouracil with or without oxaliplatin.

Secondary

• Compare the rate of clinical complete response in patients treated with these regimens.
• Compare the rate of pathologic complete response in patients treated with these regimens.
• Determine the increase in the number of patients who are able to undergo sphincter-saving surgery after treatment with these regimens.
• Correlate genetic patterns and the presence or absence of specific tissue biomarkers with response and prognosis in patients treated with these regimens.
• Compare preoperative quality of life (QOL) of patients treated with oral capecitabine versus continuous infusion with fluorouracil.
• Determine the impact of oxaliplatin on neurotoxicity in patients treated with these regimens.
• Compare the toxic effects of these regimens in these patients.
• Compare the convenience of care in patients treated with these regimens.
• Determine the impact of the type of surgical management on QOL at 1 and 5 years postoperatively in these patients.
• Describe the long-term impact of cancer treatment on symptoms (e.g., vitality and neurotoxicity) and QOL at 5 years after randomization (5-year follow-up visit).

OUTLINE: This is a randomized, multicenter study. Patients are stratified according to participating center, gender, clinical tumor stage (stage II vs stage III), and surgical intent (sphincter saving vs non-sphincter saving). Patients are randomized to 1 of 4 treatment arms.

• Arm 1: Patients receive fluorouracil IV continuously and undergo radiotherapy once daily 5 days a week for 5-6 weeks.
• Arm 2: Patients receive fluorouracil and undergo radiotherapy as in arm 1. Patients also receive oxaliplatin IV over 1 hour once weekly for 5 weeks.
• Arm 3: Patients receive oral capecitabine twice daily and undergo radiotherapy once daily 5 days a week for 5-6 weeks.
• Arm 4: Patients receive capecitabine and undergo radiotherapy as in arm 3. Patients also receive oxaliplatin as in arm 2.

Within 6-8 weeks after the completion of chemoradiotherapy, patients with responding or stable disease undergo surgery. Patients with progressive disease are treated at the discretion of the investigator and continue to be followed.

Quality of life is assessed at baseline, at completion of chemoradiotherapy, and at 1 and 5 years after surgery.

After completion of study treatment, patients are followed every 6 months for 5 years.

PROJECTED ACCRUAL: A total of 1,606 patients will be accrued for this study within 4 years.

Inclusion Criteria:

• Patients must consent to participate in the study and must have signed and dated an IRB-approved consent form conforming to federal and institutional guidelines.
• Patients must be > 18 years of age.
• Patients must have a life expectancy of 5 years, excluding their diagnosis of cancer (as determined by the investigator), and must have an Eastern Cooperative Oncology Group (ECOG) (Zubrod) performance status of 0 or 1.
• Patients must have a diagnosis of adenocarcinoma of the rectum obtained by a biopsy technique which leaves the major portion of the tumor intact.
• The interval between the initial diagnosis of rectal adenocarcinoma and randomization must be no more than 42 days.
• Prior to randomization, the investigator must specify the intent for sphincter saving or non-sphincter saving surgery.
• The tumor must be either palpable by digital rectal exam or be accessible via a proctoscope or sigmoidoscope.
• Distal border of the tumor must be located < 12 cm from the anal verge.
• The tumor must be considered by the surgeon to be amenable to curative resection. (Note that curative resection can include pelvic exenteration.)
• The tumor must be clinically Stage II (T3-4 N0 with N0 being defined as all imaged lymph nodes are < 1.0 cm) or Stage III (T1-4 N1-2 with the definition of a clinically positive node being any node > 1.0 cm). Stage of the primary tumor may be determined by ultrasound or Magnetic Resonance Imaging (MRI). Computed Tomography (CT) scan is acceptable provided there is evidence of T4 and/or N1-2 disease.
• At the time of randomization, all patients must have had the following within the previous 42 days: history and physical examination; if technically feasible, a complete colonoscopic examination; if not feasible, a proctoscopic or sigmoidoscopic exam; clinical staging of the tumor; CT or MRI of the abdomen and pelvis (combined PET/CT may be substituted), and a chest x-ray (PA and lateral) or CT scan of the chest to exclude patients with metastatic disease.
• At the time of randomization: Absolute neutrophil count (ANC) must be > 1,200/mm3; Platelet count must be > 100,000/mm3; There must be evidence of adequate hepatic function as follows: total bilirubin must be < 1.5 x the upper limit of normal (ULN) for the lab; and alkaline phosphatase must be < 2.5 x Upper Limit of Normal (ULN) for the lab; and the Aspartate Amino Transferase (AST) must be < 2.5 x ULN for the lab; and If AST is > ULN, serologic testing for Hepatitis B and C must be performed and results must be negative; Calculated creatinine clearance must be > 50 mL/min.
• Patients with prior malignancies, including invasive colon cancer, are eligible if they have been disease-free for > 5 years and are deemed by their physician to be at low risk for recurrence. Patients with squamous or basal cell carcinoma of the skin, melanoma in situ, carcinoma in situ of the cervix, or carcinoma in situ of the colon or rectum that have been effectively treated are eligible, even if these conditions were diagnosed within 5 years prior to randomization.

Exclusion Criteria:

• Findings of metastatic disease.
• On imaging, clear indication of involvement of the pelvic side wall(s).
• Rectal cancers other than adenocarcinoma, i.e., sarcoma, lymphoma, carcinoid, squamous cell carcinoma, cloacogenic carcinoma, etc.
• History of invasive rectal malignancy, regardless of disease-free interval.
• Pregnancy or lactation at the time of proposed randomization. Eligible patients of reproductive potential (both sexes) must agree to use adequate contraceptive methods.
• Any therapy for this cancer prior to randomization.
• Synchronous colon cancer.
• History of viral hepatitis or other chronic liver disease.
• Nonmalignant systemic disease (cardiovascular, renal, hepatic, etc.) that would preclude the patient from receiving any chemotherapy treatment option or would prevent required follow-up. Specifically excluded are patients with active ischemic heart disease (class III* or class IV** myocardial disease as described by the New York Heart Association), a recent history (within 6 months) of myocardial infarction, or symptomatic arrhythmia at the time of randomization. *Class III: Patients with cardiac disease resulting in marked limitation of physical activity. Such patients are comfortable at rest. Less than ordinary physical activity that causes fatigue, palpitation, dyspnea, or anginal pain. **Class IV: Patients with cardiac disease resulting in inability to perform any physical activity without discomfort. Symptoms of cardiac insufficiency or anginal syndrome may be present even at rest.
• Patients who, in the opinion of the investigator, have uncontrolled hypertension.
• Active inflammatory bowel disease (i.e., patients requiring current medical interventions or who are symptomatic).
• Prior pelvic radiation therapy for any reason.
• Known hypersensitivity to 5-fluorouracil, capecitabine, or oxaliplatin.
• Clinically significant peripheral neuropathy at the time of randomization (defined in the NCI Common Terminology Criteria for Adverse Events Version 3.0 [CTCAE v3.0] as grade 2 or greater neurosensory or neuromotor toxicity).
• Existing uncontrolled coagulopathy.
• Inability to take oral medications.
• Participation in any investigational drug study within 4 weeks prior to randomization.
• Psychiatric or addictive disorders or other conditions that, in the opinion of the investigator, would preclude the patient from meeting the study requirements.