This phase I/II trial studies the side effects and best dose of decitabine followed by mitoxantrone hydrochloride, etoposide, and cytarabine and to see how well they work in treating patients with acute myeloid leukemia or high-risk myelodysplastic syndrome that has returned after a period of improvement or does not respond to treatment. Drugs used in chemotherapy, such as mitoxantrone hydrochloride, etoposide, cytarabine, and decitabine, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving more than one drug (combination chemotherapy) may kill more cancer cells.

PRIMARY OBJECTIVES:

I. Estimate the maximum tolerated dose (MTD) of decitabine priming followed by sequential mitoxantrone hydrochloride/etoposide/cytarabine (MEC) chemotherapy in adults with relapsed/refractory acute myeloid leukemia (AML).

SECONDARY OBJECTIVES:

I. Determine, within the limits of a Phase 1/2 study, disease response and duration of remission.

II. Identify biomarkers (e.g., deoxyribonucleic acid [DNA] methylation and/or gene expression changes) associated with treatment responses.

OUTLINE: This is a phase I, dose-escalation study of decitabine followed by a phase II study.

Patients receive decitabine intravenously (IV) on days -9 to -5 (dose level 1), days -11 to -5 (dose level 2), or days -14 to -5 (dose level 3).

INDUCTION THERAPY: Patients receive mitoxantrone hydrochloride IV on days 1-5, etoposide IV on days 1-5, and cytarabine IV on days 1-5. Patients achieving complete response (CR) or CR with incomplete platelet count recovery (CRp) may receive up to 2 courses of induction therapy and up to 2 courses of consolidation therapy.

After completion of study treatment, patients are followed up every 3 months for up to 5 years.

Inclusion Criteria:

• Prior diagnosis of "high-risk" myelodysplastic syndrome (MDS) (>= 10% blasts) or AML other than acute promyelocytic leukemia (APL) with t(15;17) (q22;q12) or variants according to the 2008 World Health Organization (WHO) classification; patients with biphenotypic AML are eligible
• Relapsed/persistent disease according to standard criteria requiring salvage therapy; outside diagnostic material is acceptable as long as peripheral blood and/or bone marrow slides are reviewed at the study institution; flow cytometric analysis of peripheral blood and/or bone marrow should be performed according to institutional practice guidelines
• Patients with prior autologous or allogeneic hematopoietic cell transplantation (HCT) are eligible if relapse occurs provided symptoms of graft-versus host disease are well controlled with stable use of immunosuppressive agents
• Treatment-related mortality (TRM) score =< 9.2 as calculated with simplified model
• Should be off any active therapy for AML with the exception of hydroxyurea for at least 14 days prior to study registration unless patient has rapidly progressive disease, and all grade 2-4 non-hematologic toxicities should have resolved
• May have previously received monotherapy with demethylating agents for MDS or AML
• May have previously received chemotherapy with MEC for MDS or AML
• Patients with symptoms/signs of hyperleukocytosis or white blood cells (WBC) > 100,000/uL can be treated with leukapheresis or may receive up to 2 doses of cytarabine (up to 500 mg/m^2/dose) prior to enrollment
• Bilirubin =< 2 x institutional upper limit of normal (IULN) unless elevation is thought to be due to hepatic infiltration by AML, Gilbert's syndrome, or hemolysis (assessed within 7 days prior to study day 1)
• Serum creatinine =< 1.5 x IULN (assessed within 7 days prior to study day 1)
• Left ventricular ejection fraction >= 40%, assessed within 3 months prior to study day 1, e.g. by multi gated acquisition (MUGA) scan or echocardiography, or other appropriate diagnostic modality and no clinical evidence of congestive heart failure; if the patient had anthracycline-based therapy since the most recent cardiac assessment, cardiac evaluation should be repeated if there is clinical or radiographical suspicion of cardiac dysfunction, or if the previous cardiac assessment was abnormal
• Women of childbearing potential and men must agree to use adequate contraception
• Provide written informed consent

Exclusion Criteria:

• Refractory/relapsing myeloid blast crisis of chronic myeloid leukemia (CML), unless patient is not considered candidate for tyrosine kinase inhibitor treatment
• Concomitant illness associated with a likely survival of < 1 year
• Active systemic fungal, bacterial, viral, or other infection, unless disease is under treatment with anti-microbials and/or controlled or stable (e.g. if specific, effective therapy is not available/feasible or desired [e.g. chronic viral hepatitis, human immunodeficiency virus (HIV)]); patient needs to be clinically stable as defined as being afebrile and hemodynamically stable for 24 hours; patients with fever thought to be likely secondary to leukemia are eligible
• Known hypersensitivity to any study drug
• Pregnancy or lactation
• Patients may not be receiving any other investigational agents