The purpose of this study is to test whether the study drug prexasertib is a safe and effective treatment for people with DSRCT or RMS when given in combination with the standard drugs irinotecan and temozolomide. The study will test different doses of prexasertib in combination with irinotecan and temozolomide to find the highest dose of prexasertib that causes few or mild side effects in participants.

Inclusion Criteria:

• Consent/Assent: all patients and/or their parents or legally authorized representatives must sign written informed consent; assent, when appropriate, will be obtained according to institutional guidelines

• Age: patients must be ≥12 months of age at the time of study enrollment

• Diagnosis: patients must have histologically documented locally advanced or metastatic desmoplastic small round cell tumor or rhabdomyosarcoma (confirmed at MSK)

• Therapeutic options: patient's current disease state must be one which has failed standard therapy and for which there is no known curative therapy

• Disease Status: patients must have measurable disease based on RECIST 1.1

• Performance level: Karnofsky ≥ 50% for patients > 16 years of age and Lansky ≥ 50 for patients ≤ 16 years of age

• Prior Therapy: patients may have had any number of prior therapies, but must have recovered from the acute toxic effects of all prior anti-cancer therapy (other than alopecia) as described below and must meet the following minimum duration from prior anti-cancer directed therapy prior to enrollment

  °patients who have previously received irinotecan and/or temozolomide will be allowed

  • 21 days must have elapsed after the last dose of cytotoxic or myelosuppressive chemotherapy
  • 7 days must have elapsed after the last dose of anti-cancer agents not known to be myelosuppressive
  • 14 days must have elapsed after radiation therapy, and toxicity related to prior radiation therapy must be recovered to grade ≤ 1
  • 21 days must have elapsed after the last dose of antibody therapy, and toxicity related to prior antibody therapy must be recovered to grade ≤ 1

• Organ Function Requirements: Adequate bone marrow function defined as:

  • absolute neutrophil count (ANC) ≥ 1500/mm^3
  • platelet count ≥ 100,000/ mm^3
  • hemoglobin ≥ 8 g/dl

• Adequate renal function defined as:

  • Creatinine clearance or radioisotope glomerular filtration rate ≥ 70 mL/min/1.73m2 OR
  • Serum creatinine based on age/gender derived from the Schwartz formula for estimating GFR53

• Adequate liver function defined as:

  • Bilirubin (sum of conjugated + unconjugated) ≤ 1.5 x upper limit of normal for age
  • AST or ALT ≤ 5 x upper limit of normal for patients with liver metastases
  • Serum albumin ≥ 2.5 g/dl

• Adequate cardiac function defined as:

  • echocardiogram with left ventricular ejection fraction (LVEF) >45%
  • QTc < 470 ms on screening 12 lead electrocardiogram

• Pregnancy/Contraception

  • post-menarchal females must have a negative urine or serum pregnancy test at screening and ≤ 24 hours prior to study treatment
  • males or females of reproductive potential must be willing to use a barrier method of contraception throughout the course of the study and for 6 months after participation

Exclusion Criteria:

• Patients for whom the investigator deems that irinotecan and temozolomide are not appropriate are not eligible.
• Patients who have an uncontrolled infection are not eligible.
• Patients who are pregnant or breast feeding are not eligible.
• Patients who have a history of Torsades de Pointes, carry a diagnosis of congestive heart failure, or have a family history of prolonged QT syndrome are not eligible.
• Patients who, in the opinion of the investigator, may not be able to comply with the safety monitoring requirements of the study are not eligible.
• Patients with known hypersensitivity to irinotecan or its excipients are not eligible.

Memorial Sloan Kettering Cancer Center supports the international committee of medical journal editors (ICMJE) and the ethical obligation of responsible sharing of data from clinical trials. The protocol summary, a statistical summary, and informed consent form will be made available on clinicaltrials.gov when required as a condition of Federal awards, other agreements supporting the research and/or as otherwise required. Requests for deidentified individual participant data can be made beginning 12 months after publication and for up to 36 months post publication. Deidentified individual participant data reported in the manuscript will be shared under the terms of a Data Use Agreement and may only be used for approved proposals. Requests may be made to: crdatashare@mskcc.org.