This phase I trial studies the side effects and the best dose of MLN8237 (alisertib) when given together with paclitaxel albumin-stabilized nanoparticle formulation (nab-paclitaxel) in treating patients with solid malignancies that are metastatic or cannot be removed by surgery. Alisertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as nab-paclitaxel work by killing the cells or by stopping them from dividing. Giving alisertib together with nab-paclitaxel may provide a more effective anticancer treatment with fewer side effects.

Inclusion Criteria:

• Patient must have a histologically confirmed malignancy that is metastatic or unresectable and for which standard curative or palliative measures to not exist or are no longer effective (dose-escalation cohorts only).

• Patient must have a histologically or cytologically confirmed diagnosis of pancreatic cancer or poorly differentiated neuroendocrine tumor and must have been treated with a regimen with known benefit for pancreatic cancer or poorly differentiated neuroendocrine tumor (MTD expansion cohort only).

• Patient must have measurable disease defined as lesions that can be accurately measured in at least one dimension (longest diameter to be recorded) as ≥ 10 mm with CT scan, ≥ 20 mm by chest x-ray, or ≥ 10 mm with calipers by clinical exam.

• Patient must have disease that is easily accessible for a core biopsy as determined by the treating physician or study PI. Patient must agree to the mandatory biopsies at baseline and end of Cycle 2 (MTD expansion cohort only, and at the discretion of the PI/PI optional).

• Patient must be ≥ 18 years of age.

• Patient must have an ECOG performance status of 0 or 1

• Patient must have normal bone marrow and organ function as defined below:

  • Leukocytes ≥ 3,000/mcL
  • Absolute neutrophil count ≥ 1,500/mcL
  • Platelets ≥ 100,000/mcL
  • Hemoglobin > 9.0 g/dL
  • Total bilirubin ≤ institutional ULN
  • AST and ALT < 1.5 x ULN (or < 5 x ULN if known liver metastases)
  • Calculated creatinine clearance must be > 40 mL/min (by Cockcroft-Gault)

• If a female of childbearing potential (defined as a female who is non-menopausal or surgically sterilized), patient must be willing to use an acceptable method of birth control (i.e., hormonal contraceptive, intra-uterine device, diaphragm with spermicide, condom with spermicide, or abstinence) for the duration of the study. If a male with a partner who is a female of childbearing potential, patient must agree to use an acceptable method of contraception during the entire study treatment period through 4 months after the last dose of MLN8237. Should a woman become pregnant or suspect she is pregnant while participating in this study, she must inform her treating physician immediately.

• Patient must be able to take oral medication and to maintain a fast as required for 2 hours before and 1 hour after MLN8237 administration.

• Patient (or legally authorized representative if applicable) must be able to understand and willing to sign an IRB approved written informed consent document.

Exclusion Criteria:

• Patient must not have received chemotherapy or radiotherapy ≤ 4 weeks or 5 half-lives prior to study entry.
• Patient must not have had radiation therapy to more than 25% of the bone marrow. Whole pelvic radiation is considered to be over 25%.
• Patient must not have a history of other malignancy ≤ 2 years previous with the exception of basal cell or squamous cell carcinoma of the skin which was treated with local resection only, an in situ malignancy, or low-risk prostate cancer after curative therapy.
• Patient must not have had a prior allogeneic bone marrow or organ transplantation.
• Patient must not have previously received nab-paclitaxel.
• Patient must not have received any other investigational agents within 14 days prior to study enrollment.
• Patient must not have known brain metastases. Patients with known brain metastases must be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events.
• Patient must not have a history of allergic reactions attributed to compounds of similar chemical or biologic composition to either MLN8237 or nab-paclitaxel, or other agents used in the study.
• Patient must not have been treated with clinically significant enzyme inducers, such as the enzyme-inducing antiepileptic drugs phenytoin, carbamazepine, or Phenobarbital, or rifampin, rifabutin, rifapentine, or St. John's wort within 14 days prior to the first dose of MLN8237.
• Patient must not have received any previous treatment with any Aurora-kinase inhibitors (MTD expansion cohort only).
• Patient must not have a history of gastric resection (MTD expansion cohort only).
• Patient must not have an uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, uncontrolled diabetes, malabsorption, resection of the pancreas or upper small bowel, symptomatic congestive heart failure, or psychiatric illness/social situations that would limit compliance with study requirements.
• Patient must not have ≥ grade 2 peripheral neuropathy within 14 days before enrollment.
• Patient must not have a known history of uncontrolled sleep apnea syndrome and other conditions that could result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary disease.
• Patient must not have a requirement for supplemental oxygen.
• Patient must not require constant administration of a proton pump inhibitor, H2 antagonist, or pancreatic enzymes. Intermittent uses of antacids or H2 antagonists are allowed.
• Patient must not have QTc > 500ms within 14 days before enrollment.
• Patient must not have had a myocardial infarction within 6 months prior to enrollment or have New York Heart Association (NYHA) Class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities.
• Patient must not be pregnant and/or breastfeeding.
• Patient must not be known to be HIV-positive on combination antiretroviral because of the potential for pharmacokinetic interactions with MLN8237 or Abraxane. In addition, these patients are at increased risk of lethal infections when treated with marrow-suppressive therapy. Appropriate studies will be undertaken in patients receiving combination antiretroviral therapy when indicated.

Inclusion of Women and Minorities

Both men and women and members of all races and ethnic groups are eligible for this trial.