This phase I/II trial studies the side effects and best dose of alisertib when given together with abiraterone acetate and prednisone and to see how well it works in treating patients with hormone-resistant prostate cancer. Alisertib and abiraterone acetate may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Androgens can cause the growth of prostate cancer cells. Drugs, such as abiraterone acetate, may also lessen the amount of androgens made by the body. Drugs used in chemotherapy, such as prednisone, work in different ways to stop the growth of tumor cells, either by killing the cells or by stopping them from dividing. Giving alisertib, abiraterone acetate, and prednisone together may be an effective treatment for prostate cancer.

PRIMARY OBJECTIVES:

1. Phase I: To determine the safe dose of alisertib when given in combination with abiraterone (abiraterone acetate) and prednisone in metastatic castration resistant prostate cancer (mCRPC) patients with disease progression on abiraterone and prednisone.
2. Phase II: To determine the proportion of patients who have no disease progression after alisertib is added to abiraterone and prednisone.

SECONDARY OBJECTIVES:

1. Phase II: To determine the prostate specific antigen (PSA) kinetics after alisertib is added to abiraterone and prednisone regimen (this includes the proportion of patients with PSA progression free at 3 months, proportion of patients with 50% PSA reduction after study treatment, maximum PSA decline from baseline during the first 12 weeks).
2. Phase II: To compare baseline circulating tumor cells (CTCs) enumeration to 12 week post-therapy CTC enumeration.
3. Phase II: To compare baseline neuroendocrine marker (chromogranin A and neuron-specific enolase [NSE]) levels to 12 week post therapy neuroendocrine marker levels.
4. Phase II: To further assess overall safety of combination of alisertib with abiraterone and prednisone in the phase 1 portion of the study.

OUTLINE: This is a phase I, dose escalation study of alisertib followed by a phase II study.

Patients receive alisertib orally (PO) twice daily (BID) on days 1-7, abiraterone acetate PO daily, and prednisone PO BID. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up at 30 days.

Inclusion Criteria:

1. Age >/= 18 years and are capable of giving informed consent. Voluntary written informed consent before performance of any study-related procedure not part of normal medical care, with the understanding that consent may be withdrawn by the subject at any time without prejudice to future medical care.

2. Patients must have a pathologically confirmed diagnosis of prostate adenocarcinoma. Features of neuroendocrine phenotype are allowed.

3. Patients must have evidence of metastatic disease.

4. Patients are currently on Abiraterone treatment and the treatment dose has been stable for at least 4 weeks. There will be no further dose adjustment per treating physician.

5. Patients with evidence of any of the following disease progression based on PCWG2 criteria while on abiraterone acetate and prednisone:

   • Clinical progression (such as symptoms related to prostate cancer) with PSA progression (defined as 25% increase over baseline value with an increase in the absolute value of at least 2 ng/mL and a baseline PSA of 2 ng/ml or above).
   • Progression of one dimension measurable soft tissue (nodal or visceral metastasis) assessed within 30 days prior to registration by a CT scan or MRI of the abdomen and pelvis as per RECIST criteria.
   • Progression of bone disease (evaluable disease) or ( ≥ 2 new bone lesion(s)) by bone scan.

6. Patients must have and ECOG performance status of ≤ 2.

7. Patients must be on continuous LH-RH agonist or antagonist treatment or surgically castrated with castrate levels of testosterone (< 20 ng/dl).

8. For Phase I: any number of prior chemotherapy regimens are allowed, but patient needs to be on abiraterone acetate at the time of progression. Chemotherapy naïve patients are allowed only in the phase I part of the trial.

9. For Phase I: Patients must have either failed, are intolerant to, or have refused treatment with docetaxel.

10. For Phase II: Patients must have received 1 but no more than 2 prior chemotherapy regimen for prostate cancer.

11. Patients may have had androgen receptor targeted therapy (including second and third line antiandrogens) or other investigational drugs. Patient must have discontinued flutamide or nilutamide or other antiandrogens (including Enzalutamide) for at least 4 weeks and bicalutamide for at least 6 weeks prior to day1 treatment.

12. Patients receiving treatment with bisphosphonates or denosumab must remain on treatment during the study.

13. Patients must not require concurrent radiation or other chemotherapy while receiving protocol therapy. Patients may have received previous radiation but must have completed radiation at least 4weeks (8 weeks for radiation to the brain) prior to registration.

14. Patients must have recovered to grade ≤ 1 from all acute toxicity of previous radiation or hormonal or chemotherapy.

15. Patient agrees to use an acceptable method for contraception during the entire study treatment period through 4 months after the last dose of Alisertib. Adequate renal function as defined by serum creatinine ≤ 1.5 x ULN. If creatinine >1.5 x ULN, calculated or measured creatinine clearance must be ≥ 40 mL/minute (Cockcroft-Gault).

16. ANC > 1500/mm³, platelets > 100,000/mm³, Hgb > 9 g/dL. Values must be obtained without need for myeloid growth factor or platelet transfusion support within 14 days, however, erythrocyte growth factor is allowed as per published ASCO guidelines.

17. Total bilirubin ≤ ULN, SGOT (AST) and SGPT (ALT)< 1.5 x ULN. AST and/or ALT may be up to 5X ULN if with known liver mets

Exclusion Criteria:

1. Systemic infection requiring IV antibiotic therapy within 14 days preceding the first dose of study drug, or other severe infection.
2. Major surgery within 28 days or serious infection requiring IV antibiotics within 14 days preceding the first dose of study treatment.
3. Patient has received other investigational drugs within 14 days before enrollment.
4. Known GI disease or GI procedure that could impact drug absorption in the upper bowel, or tolerance of Alisertib. Examples include but are not limited to partial gastrectomy, small bowel resection, pancreatectomy, malabsorption or celiac disease.
5. Patient requires constant administration of proton pump inhibitor, H2 antagonist, or pancreatic enzymes. Intermittent uses of antacids or H2 antagonists are allowed (to manage gastric acidity or reflux) during the study day 8 - 20. [Histamine-2 (H2) receptor antagonists are not permitted from the day prior (Day -1) through to the end of Alisertib dosing (Day 7)]
6. Ongoing nausea or vomiting of any severity without improvement after appropriate treatment.
7. > Grade 1 diarrhea, not controlled with appropriate treatment.
8. History of uncontrolled sleep apnea syndrome and other conditions that could result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary disease requiring supplemental oxygen.
9. Clinical and/or radiographic evidence of cerebral metastases. However, patients who have a history of central nervous system (CNS) metastasis but who have no radiographic or clinical evidence of residual tumor (eg, following complete surgical resection or stereotactic radiosurgery) are not excluded from participation in this study.
10. Radiation therapy to more than 25% of the active bone marrow. Whole pelvic radiation is considered to be over 25%.
11. Myocardial infarction within 6 months prior to enrollment or has New York Heart Association (NYHA) Class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities. Prior to study entry, any ECG abnormality at Screening has to be documented by the investigator as not medically relevant.
12. Serious medical or psychiatric illness or laboratory abnormality that may increase the risk associated with study participation or investigational product administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for enrollment in this study.
13. Currently active other malignancy excluding controlled non-melanoma skin cancer. Patients are considered NOT to have "currently active" malignancy if they have completed any necessary therapy and are considered by their physician to be at less than 30% risk of relapse.
14. Treatment with clinically significant enzyme inducers, such as the enzyme-inducing antiepileptic drugs phenytoin, carbamazepine or phenobarbital, or rifampin, rifabutin, rifapentine or St. John's Wort within 14 days prior to the first dose of Alisertib and during the study.
15. Known history of human immunodeficiency virus (HIV) infection, hepatitis B, or hepatitis C. Testing is not required in the absence of clinical findings or suspicion.