This phase I/II trial studies the side effects and best dose of decitabine when given together with filgrastim, cladribine, cytarabine, and mitoxantrone hydrochloride in treating patients with acute myeloid leukemia or myelodysplastic syndrome that is newly diagnosed, has come back or has not responded to treatment. Drugs used in chemotherapy, such as decitabine, cladribine, cytarabine, and mitoxantrone hydrochloride work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Colony-stimulating factors, such as filgrastim, may increase the production of blood cells and may help the immune system recover from the side effects of chemotherapy. Decitabine, filgrastim, cladribine, cytarabine, and mitoxantrone hydrochloride may work better in treating patients with acute myeloid leukemia and myelodysplastic syndrome.

PRIMARY OBJECTIVES:

I. Estimate the maximum tolerated dose (MTD) of decitabine when used concomitantly with filgrastim, cladribine, cytarabine, and mitoxantrone hydrochloride (G-CLAM) in patients with newly diagnosed acute myeloid leukemia (AML) and high-risk myelodysplastic syndrome (MDS).

II. Compare, within the limits of a phase 1/2 study, the rate of complete remission without measurable residual disease (minimal residual disease negative [MRDneg] complete remission [CR]) with decitabine + G-CLAM at the MTD compared to similar patients treated previously with G-CLAM alone.

SECONDARY OBJECTIVES:

I. Evaluate, within the limits of a phase 1/2 study, disease response (complete remission, overall response rate) relapse-free survival (RFS), event-free survival (EFS), and overall survival (OS) in patients with newly-diagnosed AML / high-risk MDS.

II. Describe, within the limits of a phase 1/2 study, the toxicity profile of the study regimen.

OUTLINE: This is a dose de-escalation study of decitabine.

INDUCTION: Patients receive decitabine intravenously (IV) over 1 hour on days 1-10. Patients also receive filgrastim subcutaneously (SC) on days 0-5, cladribine IV over 2 hours on days 1-5, cytarabine IV over 2-4 hours on days 1-5, and mitoxantrone hydrochloride IV over 60 minutes on days 1-3.

RE-INDUCTION: Patients who do not achieve MRDneg CR after first induction are eligible for re-induction. Patients receive the same treatment as during induction except that decitabine is omitted.

CONSOLIDATION THERAPY: Beginning 6 weeks after achieving MRDneg CR or CR/CR with incomplete count recovery (CRi) after induction and/or re-induction, patients are eligible to receive filgrastim, cladribine, and cytarabine as in Induction. Treatment may be repeated for up to 4 courses in the absence of disease progression or unacceptable toxicity. Subsequent consolidation cycles would be given after recovery from the previous cycle (roughly 4-6 weeks).

After completion of study treatment, patients are followed up at for 1 month and every 3 months for up to 5 years.

Inclusion Criteria:

• For patients with newly diagnosed disease: diagnosis of "high-grade" MDS (>= 10% blasts by morphology) or AML other than acute promyelocytic leukemia (APL) with t(15;17)(q22;q12) or variants according to the 2016 World Health Organization (WHO) classification; for patients with relapsed/refractory disease: prior diagnosis of "high-risk" MDS or non-APL AML, with relapsed/refractory disease according to 2003 recommendations of the International Working Group, requiring first or subsequent salvage therapy; patients with mixed phenotype acute leukemia (MPAL) are eligible
• Outside diagnostic material is acceptable as long as peripheral blood and/or bone marrow slides are reviewed at the study institution; flow cytometric analysis of peripheral blood and/or bone marrow should be performed according to institutional practice guidelines
• Patients with prior autologous or allogeneic hematopoietic cell transplantation (HCT) are eligible if relapse occurs provided symptoms of graft-versus host disease are well controlled with stable use of immunosuppressive agents
• Treatment-related mortality (TRM) score =< 9.2 as calculated with simplified model
• Should be off any active therapy for AML with the exception of hydroxyurea for at least 14 days prior to study registration unless patient has rapidly progressive disease, and all grade 2-4 non-hematologic toxicities should have resolved
• May have previously received monotherapy with demethylating agents for MDS or AML or treatment with a mitoxantrone- or cladribine-based regimen for MDS or AML, including G-CLAM, but not demethylating agent as priming for or in combination with chemotherapy
• Patients with symptoms/signs of hyperleukocytosis or white blood cells (WBC) > 100,000/uL can be treated with leukapheresis or may receive up to 2 doses of cytarabine (up to 500 mg/m^2/dose) prior to enrollment
• Bilirubin =< 2.5 x institutional upper limit of normal (IULN) unless elevation is thought to be due to hepatic infiltration by AML, Gilbert's syndrome, or hemolysis (assessed within 14 days prior to registration)
• Serum creatinine =< 2.0 mg/dL (assessed within 14 days prior to registration)
• Left ventricular ejection fraction >= 45%, assessed within 3 months prior to registration, e.g. by multigated acquisition scan (MUGA) scan or echocardiography, or other appropriate diagnostic modality and no clinical evidence of congestive heart failure; if the patient had anthracycline-based therapy since the most recent cardiac assessment, cardiac evaluation should be repeated if there is clinical or radiographic suspicion of cardiac dysfunction, or if the previous cardiac assessment was abnormal
• Women of childbearing potential and men must agree to use adequate contraception
• Ability to understand and willingness to sign a written consent

Exclusion Criteria:

• Myeloid blast crisis of chronic myeloid leukemia (CML), unless patient is not considered candidate for tyrosine kinase inhibitor treatment
• Concomitant illness associated with a likely survival of < 1 year
• Active systemic fungal, bacterial, viral, or other infection, unless disease is under treatment with anti-microbials and/or controlled or stable (e.g. if specific, effective therapy is not available/feasible or desired [e.g. chronic viral hepatitis, human immunodeficiency virus (HIV)]); patient needs to be clinically stable as defined as being afebrile and hemodynamically stable for 24-48 hours
• Known hypersensitivity to any study drug
• Pregnancy or lactation
• Patients may not be receiving any other investigational agents