This study incorporates alisertib, the small-molecule inhibitor of Aurora A activity, in the treatment of patients younger than 22 years of age. Patients with recurrent or refractory AT/RT or MRT will receive alisertib as a single agent. Patients with newly diagnosed AT/RT will receive alisertib as part of age- and risk-adapted chemotherapy. Radiation therapy will be given to children ≥12 months of age. Patients with AT/RT and concurrent extra-CNS MRT are eligible.

Alisertib will be administered as a single agent on days 1-7 of each 21-day cycle in all recurrent patients enrolled on Stratum A. For the patients on the newly diagnosed strata (B, C or D), alisertib will be administered in sequence with chemotherapy and radiotherapy.

This study has 3 primary strata: (A) children with recurrent/progressive AT/RT or extra-CNS MRT, (B) children < 36 months-old with newly diagnosed AT/RT, (C) children > 36 months old with newly diagnosed AT/RT. Children with concurrent MRT will be treated according to age and risk stratification schemes outlined for strata B and C and will have additional treatment for local control. Children with synchronous AT/RT will be treated with age and CNS risk-appropriate therapy, and also receive surgery and/or radiation therapy for local control of the non-CNS tumor.

PRIMARY OBJECTIVES

• To estimate the sustained objective response rate and disease stabilization in pediatric patients with recurrent or progressive AT/RT (atypical teratoid rhabdoid tumor in the CNS) (Stratum A1) treated with alisertib and to determine if the response is sufficient to merit continued investigation of alisertib in this population.
• To estimate the sustained objective response rate and disease stabilization in pediatric patients with recurrent or progressive extra-CNS MRT (malignant rhabdoid tumor outside the CNS) (Stratum A2) treated with alisertib and to determine if the response is sufficient to merit continued investigation of alisertib in this population.
• To estimate the 3-year PFS rate of patients with newly diagnosed AT/RT who are younger than 36 months of age at diagnosis with no metastatic disease (Stratum B1) treated with alisertib in sequence with induction and consolidation chemotherapy and radiation therapy (depending on age) and to determine if the rates are sufficient to merit continued investigation of alisertib in this population.
• To estimate the 1-year PFS rate of patients with newly diagnosed AT/RT who are younger than 36 months of age at diagnosis, with metastatic disease (Stratum B2) treated with alisertib in sequence with induction and consolidation chemotherapy and to determine if the rates are sufficient to merit continued investigation of alisertib in this population.
• To estimate the 3-year PFS rate of patients with newly diagnosed AT/RT who are 3 years of age or greater at diagnosis with no metastatic disease and gross total resection or near total resection (Stratum C1) treated with alisertib in sequence with radiation therapy and consolidation chemotherapy and to determine if the rates are sufficient to merit continued investigation of alisertib in this population.
• To estimate the 1-year PFS rate of patients with newly diagnosed AT/RT who are 3 years of age or greater at diagnosis with metastatic or residual disease (Stratum C2) treated with alisertib in sequence with radiation therapy and consolidation chemotherapy and to determine if the rates are sufficient to merit continued investigation of alisertib in this population.
• To characterize the pharmacokinetics and pharmacodynamics of alisertib in pediatric patients and to relate drug disposition to toxicity.

SECONDARY OBJECTIVES

• To estimate the duration of objective response and PFS in patients with recurrent/progressive AT/RT and MRT (Strata A1 and A2).
• To estimate PFS and OS distributions in patients with newly diagnosed AT/RT (Strata B1, B2, B3, C1 and C2).
• To describe toxicities experienced by patients treated on this trial, specifically any toxicities of alisertib when administered as a single agent or in combination with other therapy over multiple courses and toxicities related to proton or photon radiation therapy.
• To describe the patterns of local and distant failure in newly diagnosed patients (Strata B1, B2, B3, C1 and C2). Local control relative to primary-site radiation therapy, with criteria for infield, marginal, or distant failure will also be reported descriptively.

We propose a study with 3 primary treatment strata according to participant's previous treatment, age and presence of extra-CNS disease, with substrata for presence of focal or metastatic disease:

* STRATUM A - RECURRENT OR PROGRESSIVE DISEASE: Patients < 22 years of age at diagnosis with recurrent or progressive MRT (either CNS and/or extra-CNS) and measurable disease as defined in the protocol.

• Stratum A1: patients with AT/RT (CNS MRT).
• Stratum A2: patients with extra-CNS MRT (patients with concurrent progression of AT/RT and MRT are eligible for therapy, but their data will be analyzed separately).
• Stratum A3: patients with synchronous AT/RT and extra-CNS MRT

Stratum A is Closed to Accrual

* STRATUM B - NEWLY DIAGNOSED DISEASE IN YOUNG CHILDREN < 36 MONTHS: Patients < 36 months of age at diagnosis of CNS-AT/RT, no prior therapy:

• Stratum B1: Patients with no metastatic disease (M0).
• Stratum B2: Patients with metastatic disease (M+) regardless of degree of resection.
• Stratum B3: Patients for whom CSF by lumbar puncture was not obtained for clinical reasons and have no other evidence of metastatic disease (MX).

Stratum B is Closed to Accrual

* STRATUM C - NEWLY DIAGNOSED DISEASE IN CHILDREN > 3 YEARS: Patients > 3 years (36 months) of age at diagnosis of AT/RT, no prior therapy:

• Stratum C1: Patients with gross total (GTR) or near total resection (NTR) defined as <1.5 cm2 of residual tumor, and no metastatic disease.
• Stratum C2: Patients with metastatic disease (M+) and/or bulky residual tumor >1.5 cm^2.

STRATUM D - SYNCHRONOUS EXTRANEURAL AT/RT and EXTRA-CNS MRT: Treatment will be based on the extent of both CNS and extra-CNS disease. CNS-directed therapy will be given according to Strata B1, B2, C1 or C2 according to age and metastatic status. In addition, patients may receive irradiation according to best clinical management for local control of extra-CNS disease.

• Stratum D1: Patients < 36 months at time of diagnosis with synchronous AT/RT and extra-CNS MRT and no metastatic CNS disease (M0).
• Stratum D2: Patients < 36 months at time of diagnosis with synchronous AT/RT and extra- CNS MRT and metastatic CNS disease (M+).
• Stratum D3: Patients < 36 months at time of diagnosis with synchronous AT/RT and extra-CNS MRT for whom CSF by lumbar puncture was not obtained for clinical reasons and without other evidence of metastatic disease (MX)
• Stratum D4: Patients ≥ 36 months at time of diagnosis with synchronous extra-CNS MRT with or without metastatic CNS disease regardless of the degree of tumor resection.

Stratums D1, D2 and D3 are Closed to Accrual

Biological parents of participants with ATRT/MRT may consent to and provide a genomic blood specimen for DNA extraction and analysis.

OVERVIEW OF TREATMENT PLAN: Patients with recurrent disease (Stratum A) will receive alisertib as a single agent days 1-7 out of 21 days. Newly diagnosed patients (Strata B, C and D) will receive alisertib in sequence with chemo and radiotherapy. Patients on sub-strata B1 and D1 will receive focal RT once they are >12 months of age. Patients on sub-strata B2 and D2, with disseminated disease will not receive CNS radiation therapy (RT). Patients on sub-strata C1/C2/D4 will receive risk-stratified craniospinal irradiation (CSI) and boost to primary tumor site followed by adjuvant chemotherapy. Patients on sub-strata B3 and D3 will receive therapy similar to sub-strata B2 and D2 and will be considered for local radiotherapy depending on their age, response to therapy, and subsequent metastatic staging. Those patients with concurrent CNS and extra-CNS MRT may undergo irradiation of the extra-CNS MRT according to best clinical management in addition to CNS directed therapy. Alisertib will be administered only to eligible patients under the supervision of the investigator or identified sub-investigator(s).

INCLUSION CRITERIA for Patients on All Strata EXCEPT Stratum P

• Patients must be < 22 years of age at time of diagnosis (e.g., eligible until 22nd birthday).
• Histologic diagnosis of AT/RT or MRT as documented by the institutional pathologist with loss of INI1 or BRG1 expression in tumor cells confirmed by immunohistochemistry, or by molecular confirmation of tumor-specific biallelic SMARCB1/SMARCA4 loss/mutation if INI1/BRG1 immunohistochemistry is not available. For Stratum A participants, histologic confirmation of the diagnosis of AT/RT or MRT may be from the original diagnosis or at the time of recurrence/progression.
• Patients must have adequate organ function (bone marrow, renal, liver), as defined in the protocol.
• Female patients who are at least 10-years-old or are post-menarchal must have a negative serum or urine pregnancy test prior to enrollment.
• Patients of childbearing or child fathering potential must be willing to use a medically acceptable form of birth control, which includes abstinence during study treatment and 12 months after the last dose of alisertib.

Inclusion Criteria for Stratum A Participants:

• Patients with recurrent or progressive AT/RT/MRT (either CNS and/or extra-CNS) with radiographically measurable disease as defined by at least 1 lesion that can be measured in 2 dimensions or with tumor cells present in the CSF taken within 2 weeks prior to enrollment.

• Performance status defined by Karnofsky or Lansky > 60 (except for patients with posterior fossa syndrome). Use Karnofsky for patients > 16 years and Lansky for patients < 16 years. Note: Patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered to be ambulatory for the purpose of assessing the performance score.

• Patient has fully recovered from the acute toxic effects of chemotherapy, immunotherapy, or radiation therapy prior to entering this study:

  • Myelosuppressive chemotherapy: Patient has not received myelosuppressive chemotherapy within 3 weeks of enrollment onto this study (4 and 6 weeks if prior temozolomide and nitrosourea, respectively).
  • Hematopoietic growth factors: At least 7 days must have elapsed since the completion of therapy with a growth factor. At least 14 days must have elapsed after receiving pegfilgrastim.
  • Biologic (anti-neoplastic agent): At least 7 days must have elapsed since completion of therapy with a biologic agent. For agents that have known adverse events occurring beyond 7 days after administration, this period prior to enrollment must be extended beyond the time during which adverse events are known to occur.
  • Monoclonal antibodies: At least 3 half-lives must have elapsed since prior therapy that included a monoclonal antibody (see Appendix I).
  • Radiation therapy: at least 3 months must have elapsed since any irradiation unless measurable disease progression occurs at a site separate from the irradiated area and the patient has recovered from toxicities associated with radiation therapy.

• Patients with progressive synchronous/metachronous AT/RT and MRT will be eligible for stratum A3.

• Patients may not have previously received alisertib.

• Live expectancy >8 weeks.

• Stable neurologic deficits on a stable dose of corticosteroids (if applicable) for at least 1 week before study enrollment.

Inclusion Criteria for Strata B or C Participants:

• Patients with newly diagnosed AT/RT.
• Performance status defined by Karnofsky or Lansky > 30 (except for patients with posterior fossa syndrome). Other requirements of performance evaluation are the same as for Stratum A participants.
• No previous anticancer therapy (radiation therapy or chemotherapy) other than the use of corticosteroids.
• Patients must begin treatment as outlined in the protocol within 42 days of definitive surgery (day of surgery is day 0; definitive surgery includes last surgery to resect residual tumor).

Inclusion Criteria for Stratum D Participants:

• Patients with newly diagnosed AT/RT and synchronous extra-CNS MRT.
• Performance status defined by Karnofsky or Lansky > 30 (except for patients with posterior fossa syndrome). Other requirements of performance evaluation are the same as for Stratum A participants.
• No previous anticancer therapy (radiation therapy or chemotherapy) other than the use of corticosteroids.
• Patients must begin treatment within 42 days of definitive surgery (day of surgery is day 0; definitive surgery includes repeat surgeries to resect residual tumor).

Inclusion Criteria for Stratum P Participants:

• Biological parent of patient enrolling on the protocol (SJATRT) will be assigned to Stratum P.

EXCLUSION CRITERIA for All Strata Except Stratum P:

• Clinically significant medical disorders that could compromise the ability to tolerate protocol therapy or that would interfere with the study procedures or results history.
• Presence of an active, uncontrolled infection.
• Known history of uncontrolled sleep apnea syndrome or other conditions that could result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary disease or a requirement for supplemental oxygen.
• Requirement for constant administration of proton-pump inhibitor, H2 antagonist, or pancreatic enzymes. Intermittent uses of antacids or H2 antagonists are allowed while patients are on dexamethasone as described in the protocol.
• Inability to comply with the safety monitoring requirements of the study, as judged by the investigator.
• Female participants of childbearing potential cannot be pregnant or breast-feeding.
• Patients who are receiving other investigational drugs 14 or fewer days before enrollment.
• Treatment with clinically significant enzyme inducers, such as the enzyme-inducing antiepileptic drugs phenytoin, carbamazepine or phenobarbital, or with rifampin, rifabutin, rifapentine, or St. John's wort within 7 days prior to initiation of alisertib.
• Known gastrointestinal disease or procedures that could interfere with the oral absorption or tolerance of alisertib. Examples include, but are not limited to partial gastrectomy, history of small intestine surgery, and celiac disease.
• Myocardial infarction within 6 months prior to enrollment or New York Heart Association Class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities. If for some reason an electrocardiogram is obtained before study enrollment, any abnormalities detected should be documented as clinically irrelevant.
• Other severe acute or chronic medical or psychiatric condition, including uncontrolled diabetes, malabsorption, resection of the pancreas or upper small-bowel, or requirement for pancreatic enzymes, any condition that would modify the absorption of oral medications in the small bowel or any laboratory abnormality that may increase the risk associated with study participation or investigational product administration or that may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for enrollment in this study.