This phase I trial studies the side effects and best dose of alisertib when given together with combination chemotherapy in treating patients with gastrointestinal tumors. Alisertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as fluorouracil, leucovorin calcium, oxaliplatin, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving alisertib with more than one drug (combination chemotherapy) may be a better treatment for gastrointestinal tumors.

PRIMARY OBJECTIVES:

I. To determine the maximum tolerated doses and the recommended phase II doses of modified fluorouracil, leucovorin calcium, oxaliplatin (mFOLFOX) given in combination with alisertib (MLN8237) in patients with gastrointestinal cancers.

SECONDARY OBJECTIVES:

I. To describe any anti-tumor activity associated with this treatment. II. To assess baseline tumor expression by immunohistochemistry of aurora kinase A (AURKA), v-akt murine thymoma viral oncogene homolog 1 (AKT), phosphorylated (phospho)-AKT (serine [Ser]473), tumor protein p53 (p53), tumor protein p73 (p73), beta (b)-catenin, v-myc myelocytomatosis viral oncogene homolog (avian) (c-MYC), and cleaved caspase 3.

III. To assess baseline messenger ribonucleic acid (mRNA) expression of AURKA, vascular endothelial growth factor (VEGF), c-MYC, human double minute 2 (HDM2), and cyclin D1 (CCND1), and correlate with response to therapy.

IV. To obtain post-treatment biopsy specimens and confirm target inhibition by assaying for AURKA, phosphorylated (p)AURKA T288, p53, p73, MYC, HDM2, and cleaved caspase 3.

V. To perform quantitative real time-polymerase chain reaction (qRT-PCR), and immunohistochemistry (IHC) on post-treatment specimens for AURKA oncogenic targets: p53-regulated apoptosis inducing protein 1 (p53AIP1), VEGF, HDM2, and MYC.

OUTLINE: This is a dose-escalation study of alisertib.

Patients receive alisertib orally (PO) twice daily (BID) on days 1-3 and mFOLFOX regimen comprising oxaliplatin intravenously (IV) over 2 hours on day 2, leucovorin calcium IV over 2 hours on day 2, and fluorouracil IV continuously over 46 hours on days 2-4. Courses repeat every 14 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up for 4 weeks.

Inclusion Criteria:

• Patients must have histologically confirmed gastrointestinal malignancy that is metastatic or unresectable and for which standard curative or palliative measures do not exist or are no longer effective, or for whom FOLFOX would be an appropriate therapy
• Patients are required to have evaluable disease
• Any number of prior treatment regimens is allowed
• Eastern Cooperative Oncology Group (ECOG) performance status =< 2 (Karnofsky >= 60%)
• Life expectancy of greater than 12 weeks
• Absolute neutrophil count >= 1,500/mcL
• Platelets >= 100,000/mcL
• Total bilirubin below institutional upper limit of normal
• Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT]) < 3 x institutional upper limit of normal
• Creatinine below institutional upper limit of normal OR creatinine clearance >= 60 mL/min/1.73 m^2 for patients with creatinine levels above institutional normal
• Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation, and 4 months after completion of MLN8237 (alisertib) administration; should a woman become pregnant or suspect she is pregnant while she is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of MLN8237 administration
• Ability to understand and the willingness to sign a written informed consent document
• Patients must be able to take oral medications

Exclusion Criteria:

• Patients who have had targeted therapy, chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier
• Patients who are receiving any other investigational agents
• Patients with known brain metastases should be excluded from this clinical trial
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to MLN8237, including but not limited to established allergic reaction to benzodiazepines, 5-FU (fluorouracil), leucovorin (leucovorin calcium) or oxaliplatin
• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with MLN8327
• Human immunodeficiency virus (HIV)-positive patients on combination antiretroviral therapy are ineligible
• Prior allogeneic bone marrow or organ transplantation
• Known history of uncontrolled sleep apnea syndrome and other conditions that could result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary disease; requirement for supplemental oxygen; or any conditions that could result in excessive toxicity associated with the benzodiazepine-like effects of MLN8237
• Requirement for constant administration of proton pump inhibitor, histamine (H2) antagonist, or pancreatic enzymes; intermittent uses of antacids or H2 antagonists are allowed as described
• Inability to swallow oral medication or to maintain a fast as required for 2 hours before and 1 hour after MLN8237 administration or any condition that would modify small bowel absorption of oral medications, including malabsorption, or resection of pancreas or upper bowel
• Patients requiring any medications or substances that are strong or moderate cytochrome P450, family 3, subfamily A (CYP3A) inhibitors or clinically significant enzyme inducers of CYP3A4 are ineligible
• Patients with grade 2 peripheral neuropathy or greater are excluded