This phase I trial studies the side effects and the best dose of alisertib when given together with vorinostat in treating patients with Hodgkin lymphoma, B-cell non-Hodgkin lymphoma, or peripheral T-cell lymphoma that has come back. Alisertib and vorinostat may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth.

PRIMARY OBJECTIVES:

I. To determine the maximum-tolerated dose (MTD) of MLN8237 (alisertib) when given in combination with vorinostat and to select a dose and schedule for further testing (recommended Phase 2 dose: RP2D) in patients with lymphoid malignancies.

II. To describe the toxicities of MLN8237 when given in combination with vorinostat on a 21-day schedule.

III. To determine any clinical responses with MLN8237 in combination with vorinostat.

IV. To compare the plasma pharmacokinetics of MLN8237 when given alone and in combination with vorinostat.

V. To perform immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH) analysis to determine aurora kinase A (AURKA) expression in archival formalin-fixed paraffin-embedded sections from the most recent available tumor specimens of patients.

VI. To perform correlative studies for apoptosis and proliferation on bone marrow and lymph node specimens, where available, obtained from patients in the expanded cohort at RP2D.

OUTLINE: This is a dose-escalation study of alisertib.

Patients receive alisertib orally (PO) twice daily (BID) on days 1-7 or days 1-3 and 8-10, and vorinostat PO BID on days 1-14 or days 1-5 and 8-12. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up for at least 30 days.

Inclusion Criteria:

• Patients must have a histologically or cytologically confirmed lymphoid malignancy (like Hodgkin lymphoma or one of the mature B- or T-cell non-Hodgkin lymphomas as classified by World Health Organization [WHO]) for which standard curative or palliative measures do not exist or are no longer effective

• Patients must have measurable disease in two dimensions and >= 2 cm is acceptable (or 1.5 cm if 0.5 slices are used, as in spiral computed tomography [CT] scans); lesions that are considered intrinsically non-measurable include the following:

  • Bone lesions
  • Leptomeningeal disease
  • Ascites
  • Pleural/pericardial effusion
  • Inflammatory breast disease
  • Lymphangitis cutis/pulmonis
  • Abdominal masses that are not confirmed and followed by imaging techniques
  • Cystic lesions
  • Lesions that are situated in a previously irradiated area

• Patients must have had at least 1 prior systemic chemotherapy (not just steroids or local radiation); last chemotherapy or radiation must be at least 4 weeks prior to enrollment on this study; patients who decline other potentially curative therapy may be eligible; prior radiation therapy must not have been to more than 25% of the bone marrow; whole pelvic radiation is considered to be over 25%

• Eastern Cooperative Oncology Group (ECOG) performance status =< 2 (Karnofsky >= 60%)

• Life expectancy of greater than 12 weeks

• Absolute neutrophil count >= 1,500/mcL

• Platelets >= 100,000/mcL

• Total/direct bilirubin < 1.5 X institutional upper limit of normal; patients with elevation of indirect (unconjugated) bilirubin alone, as in Gilbert's syndrome, are eligible

• Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT]) < 2.5 X institutional upper limit of normal

• Creatinine =< institutional upper limit of normal OR creatinine clearance >= 60 mL/min/1.73 m^2 for patients with creatinine levels above institutional normal

• Prior allogeneic stem cell transplant patients will be allowed to enroll if they are past day +100 of transplant, have no active graft-versus-host-disease, are not on any immunosuppressants and have been off immunosuppressants for at least 4 weeks; prior autologous stem cell transplant patients will also be allowed to enter this study if they are past their day +100 of transplant

• Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation, and 4 months after completion of MLN8237 administration; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of MLN8237 administration

• Ability to understand and the willingness to sign a written informed consent document

• According to current guidelines, patients must be able to take oral medication and to maintain a fast as required for 2 hours before and 1 hour after MLN8237 administration; these guidelines may change pending results from an ongoing food effects study

Exclusion Criteria:

• Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier
• Patients who are receiving any other investigational agents
• Patients with known brain metastases should be excluded from this clinical trial
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to MLN8237, including but not limited to established allergic reaction to benzodiazepines
• Treatment with valproic acid within 14 days prior to initiation of study and during the study
• Prior use of valproic acid or any other histone deacetylase (HDAC) inhibitor for lymphoma treatment
• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with MLN8237
• Human Immunodeficiency virus (HIV)-positive patients on combination antiretroviral therapy are ineligible; appropriate studies will be undertaken in patients receiving combination antiretroviral therapy when indicated
• Known history of uncontrolled sleep apnea syndrome and other conditions that could result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary disease; requirement for supplemental oxygen; or any conditions that could result in excessive toxicity associated with the benzodiazepine-like effects of MLN8237
• Requirement for constant administration of proton pump inhibitor, histamine (H2) antagonist, or pancreatic enzymes; intermittent uses of antacids or H2 antagonists are allowed
• Inability to swallow oral medication or to maintain a fast as required for 2 hours before and 1 hour after MLN8237 administration or any condition that would modify small bowel absorption of oral medications, including malabsorption, or resection of pancreas or upper bowel
• Treatment with clinically significant enzyme inducers, such as the enzyme-inducing antiepileptic drugs phenytoin, carbamazepine, oxcarbazepine, primidone or phenobarbital, or rifampin, rifabutin, rifapentine, or St. John's wort within 14 days prior to the first dose of MLN8237 and during the study
• Patients with New York Heart Association (NYHA) class II-IV heart failure