This phase II trial studies how well alisertib works in treating patients with sarcoma that has spread to other places in the body and usually cannot be cured or controlled with treatment (advanced) or has spread to other places in the body (metastatic). Alisertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth.

PRIMARY OBJECTIVES:

I. To determine the response rate (complete response [CR] + partial response [PR]) assessed for patients within each cohort: liposarcoma (cohort 1); leiomyosarcoma (non-uterine) (cohort 2); undifferentiated sarcoma (including pleomorphic undifferentiated sarcoma, formerly known as malignant fibrous histiocytoma, and myxofibrosarcoma) (cohort 3); malignant peripheral nerve sheath tumor (cohort 4); and other sarcomas (cohort 5).

SECONDARY OBJECTIVES:

I. To estimate the progression-free survival (PFS) and overall survival (OS) for patients treated with MLN8237 (alisertib) in each cohort.

II. To assess the adverse events associated with patients treated with MLN8237 in each cohort.

TERTIARY OBJECTIVES:

I. To correlate potential clinical benefit with markers of aurora kinase inhibition in pre- and post-treatment tumor biopsies.

II. To correlate clinical outcome with change in fluorine F 18 fluorothymidine (FLT)-positron emission tomography (PET) uptake at baseline versus after one week of treatment (ie, week 2 of cycle 1).

OUTLINE:

Patients receive alisertib orally (PO) twice daily (BID) on days 1-7. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up every 6 months for 18 months.

Inclusion Criteria:

• Patients must have histologically or cytologically confirmed sarcoma that is metastatic and/or locally advanced or locally recurrent and unresectable; confirmation of pathologic diagnosis will be performed at the registering site; patients will been rolled on one of five cohorts of the study:

  • Cohort 1: liposarcoma
  • Cohort 2: leiomyosarcoma (non-uterine)
  • Cohort 3: undifferentiated sarcoma (including malignant fibrous histiocytoma and myxofibrosarcoma)
  • Cohort 4: malignant peripheral nerve sheath tumor
  • Cohort 5: other sarcomas

• Patients must have measurable disease per Response Evaluation Criteria in Solid Tumors(RECIST) 1.1; note: defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as >= 2 cm with conventional techniques or as >= 1 cm with spiral computed tomography (CT) scan, magnetic resonance imaging (MRI), or calipers by clinical exam

• Any number of prior therapies is permitted; note: the last dose of systemic therapy (including tyrosine kinase inhibitors) must have been given >= 4 weeks prior to initiation of study therapy; patients receiving BCNU or mitomycin C must have received their last dose of such therapy at least 6 weeks prior to initiation of therapy

• Eastern Cooperative Oncology Group (ECOG) performance status (PS) =< 2

• Leukocytes >= 3,000/mcL

• Absolute neutrophil count >= 1,500/mcL

• Platelet count >= 100,000/mcL

• Total bilirubin =< 1.5 x institutional upper limit of normal

• Serum glutamic oxaloacetic transaminase (SGOT) (aspartate aminotransferase [AST]) and serum glutamate pyruvate transaminase (alanine aminotransferase [AST]) < 3 x institutional upper limit of normal if no liver metastases or < 5 x institutional upper limit of normal if liver metastases present

• Creatinine =< 1.5 x upper limit of normal (ULN) OR creatinine clearance >= 60 mL/min/1.73m^2 for patients with creatinine levels above institutional normal

• Women of childbearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of MLN8237 administration

• Ability to understand and the willingness to sign a written informed consent document

• According to current guidelines, patients must be able to take oral medication and to maintain a fast as required for approximately one hour before and two hours after MLN8237 administration

Exclusion Criteria:

• Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier; patients who have had radiation therapy to more than 25% of the bone marrow; whole pelvic radiation is considered to be over 25%
• Patients who are receiving any other investigational agents
• Patients with known brain metastases
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to MLN8237 including, but not limited to, established allergic reaction to benzodiazepines
• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, New York Heart Association (NYHA) class II-IV heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• Pregnant women; note: women of child-bearing potential must have a negative serum or urine pregnancy test within 7 days prior to registration; breastfeeding should be discontinued if the mother is treated with MLN8237
• Leiomyosarcoma of the uterus
• Patients known to be human immunodeficiency virus (HIV)-positive on antiretroviral therapy
• Prior allogeneic bone marrow or organ transplantation
• Known history of uncontrolled sleep apnea syndrome and other conditions that could result in excessive daytime sleepiness such as severe chronic obstructive pulmonary disease, requirement for supplemental oxygen, or any conditions that could result in excessive toxicity associated with the benzodiazepine-like effects of MLN8237
• Requirement for constant administration of proton pump inhibitor, histamine-2 (H2) antagonist, or pancreatic enzymes; note: intermittent uses of antacids or H2 antagonists are allowed
• Inability to swallow oral medication or to maintain a required fast for approximately one hour before and two hours after MLN8237 administration or any condition that would modify small bowel absorption of oral medications, including malabsorption or resection of pancreas or upper bowel
• Treatment with clinically significant enzyme inducers, such as the enzyme-inducing antiepileptic drugs phenytoin, carbamazepine, oxcarbazepine, primidone or phenobarbital, or rifampin, rifabutin, rifapentine, or St. John's wort within 14 days prior to the first dose of MLN8237 and during the study