This phase I/II trial studies how well quizartinib, decitabine, and venetoclax work in treating participants with acute myeloid leukemia or high risk myelodysplastic syndrome that is untreated or has come back (relapsed). Quizartinib may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as decitabine and venetoclax, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving quizartinib and decitabine may work better at treating acute myeloid leukemia and myelodysplastic syndrome.

PRIMARY OBJECTIVES:

I. To determine the overall response rate (ORR) including CR (complete remission) + CRp (complete remission with incomplete platelet recovery) + CRi (complete remission with incomplete count recovery) + partial remission (PR) within 3 months of treatment initiation of quizartinib and decitabine + venetoclax combination in patients with newly diagnosed or relapsed acute myeloid leukemia (AML) or high risk myelodysplastic syndrome (MDS) (> 10% blasts).

II. To determine the safety and maximum tolerable dose (MTD) of this combination.

SECONDARY OBJECTIVES:

I. To determine CR and CR+CRh rates within 3 months of treatment initiation of quizartinib and decitabine + venetoclax combination in patients with newly diagnosed or relapsed AML or high risk MDS (> 10% blast).

II. To determine the duration of response (DOR), event-free survival (EFS), overall survival (OS), MRD status at response and best MRD response attained by flow-cytometry (all patients) and by FLT3-PCR (if applicable) or variant allele frequency monitoring (if applicable) and number of patients bridged to hematopoietic stem cell transplant (HSCT) and median duration to HSCT from the initiation of the combination.

III. To investigate correlations of response to this combination with a pre-therapy, on-therapy, and progression 81-gene panel of gene mutations in AML.

EXPLORATORY OBJECTIVES:

I. To investigate possible relationships between response and non-response to the combination with pretherapy, on-therapy, and progression gene expression signatures.

II. To investigate the characterization of genetic heterogeneity in tumor cell populations, by performing targeted single-cell sequencing on longitudinally collected AML tumor populations from patients using a novel microfluidic approach that barcodes amplified genomic DNA from thousands of individual leukemia cells confined to droplets (single cell sequencing).

III. To identify individual treatment-resistant cell populations and how their signaling state in disease relates to clinical outcomes we will perform CyTOF (mass cytometry) on patients' bone marrow samples and peripheral blood at diagnosis, remission and relapse.

IV. To store and/or analyze surplus blood or tissue including bone marrow, if available, for potential future exploratory research into factors that may influence development of AML and/or response to the combination (where response is defined broadly to include efficacy, tolerability or safety).

OUTLINE: This is a phase I, dose escalation study followed by a phase II study.

Patients receive decitabine intravenously (IV) over 1 hour on days 1-10, quizartinib orally (PO) every day beginning on day 1 of cycle 1, and venetoclax PO on days 1-14 (days 1-21 if persistent leukemia). Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up every 3-6 months for up to 5 years.

Inclusion Criteria:

• Diagnosis of 1) AML (World Health Organization [WHO] classification definition of >= 20% blasts) excluding acute promyelocytic leukemia (APL) or 2) MDS with > 10% blasts (defined by the International Prognostic Scoring System [IPSS] classification).

• For frontline Cohort: Patients aged >= 60 years old who are not candidates for intensive induction therapy and agree to receive the proposed combination therapy will be enrolled.

• Not considered candidates for intensive remission induction chemotherapy at time of enrollment based on EITHER:

  • 75 years of age OR

  • < 75 years of age with at least 1 of the following:

    • Poor performance status (Eastern Cooperative Oncology Group [ECOG]) score of 2-3.

    • Clinically significant heart or lung comorbidities, as reflected by at least 1 of:

      • Left ventricular ejection fraction (LVEF) =< 50%.
      • Lung diffusing capacity for carbon monoxide (DLCO) =< 65% of expected.
      • Forced expiratory volume in 1 second (FEV1) =< 65% of expected.
      • Chronic stable angina or congestive heart failure controlled with medication.

  • Liver transaminases > 3 x upper limit of normal (ULN).

  • Other contraindication(s) to anthracycline therapy (must be documented).

  • Other comorbidity the investigator judges incompatible with intensive remission induction chemotherapy, which must be documented and approved by the principal investigator (PI).

• Patients with newly diagnosed AML with poor risk complex karyotype and/or TP53 deletions/mutations equal or younger than 60 year old

• For relapsed cohort: Patients aged >= 18 years old. (Patients who are candidates for relapse cohort will be enrolled into the study regardless of their fitness for intensive chemotherapy). (a) Patients with relapsed/refractory AML are eligible if they are not eligible for potentially curative therapy such as effective salvage therapy or hematopoietic stem cell transplantation or who refuse these options at the time of enrollment.

• Detection of FLT3-ITD mutation or FLT3-ITD/TKD co-mutations in bone marrow and/or peripheral blood samples within 30 days prior to study enrollment.

• For frontline cohort: Patients must be chemonaive, i.e. not have received any chemotherapy (except hydrea or 1-2 doses of ara-C for transient control of hyperleukocytosis) for AML or MDS. They may have received transfusions, hematopoietic growth factors or vitamins for an antecedent hematological disorder (AHD) or for AML. Temporary prior measures such as apheresis, all-trans-retinoic acid (ATRA), steroids or hydrea while diagnostic work-up is being performed are allowed and not counted as a prior salvage. Supportive care therapy for MDS (growth factors, transfusions) will not be considered as prior therapy for MDS/AML and these patients will be enrolled to the frontline cohort of the study if they are otherwise eligible.

• For relapsed cohort: Patients who have received at least one prior therapy for AML or for MDS with > 10% blasts will be eligible. Patients may have received up to 4 prior salvages for AML and/or MDS (defined by the IPSS classification). Prior therapy for AML or MDS will be counted as a prior salvage. Patients who receive MDS directed therapies considered not purely supportive such as HMAs, lenalidomide, investigational therapies, will be enrolled to the salvage cohort if they are otherwise eligible.

• In the absence of rapidly progressing disease, the interval from prior treatment to time of initiation of protocol therapy will be at least 2 weeks for cytotoxic agents or at least 5 half-lives for cytotoxic/noncytotoxic agents. The half-life for the therapy in question will be based on published pharmacokinetic literature (abstracts, manuscripts, investigator brochure's, or drug-administration manuals) and will be documented in the protocol eligibility document

• The use of chemotherapeutic or anti-leukemic agents is not permitted during the study with the following exceptions: (1) intrathecal (IT) therapy for patients with controlled central nervous system (CNS) leukemia at the discretion of the PI a (2) Use of one dose of cytarabine (up to 2 g/m^2) or hydroxyurea for patients with rapidly proliferative disease is allowed before the start of study therapy and for the first four weeks on therapy. These medications will be recorded in the case-report form.

• Eastern Cooperative Oncology Group (ECOG) performance status =< 2.

• Serum biochemical values with the following limits unless considered due to leukemia, hemolysis or congenital disorder (creatinine < 1.8 mg/dl, total bilirubin < 1.8 mg/dL, [serum glutamate pyruvate transaminase (SGPT)] < 2.5 x upper limit of normal).

• White blood cell count < 25 x 10^9/L

• Potassium, magnesium, and calcium (normalized for albumin) levels should be within institutional normal limits.

• Ability to take oral medication.

• Ability to understand and provide signed informed consent.

• Baseline left ventricular ejection fraction by echocardiogram (ECHO) or multigated acquisition (MUGA) >= 50%.

• Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test within 7 days. Men must agree not to father a child and agree to use a condom if his partner is of child bearing potential.

• WOCBP must use appropriate method(s) of contraception. WOCBP should use an adequate method to avoid pregnancy until at least 3 months after the last dose of investigational drug. Women who are not of childbearing potential (i.e., who are postmenopausal or surgically sterile) as well as men with azoospermia do not require contraception.

• Negative urine or serum pregnancy test.

• Investigational agents that are not used for treatment of the leukemia per se (e.g. anti-infective prophylaxis or therapy) will be allowed. Other supportive care studies are allowed, even if under an Investigational New Drug (IND).

Exclusion Criteria:

• Patients with known allergy or hypersensitivity to quizartinib, mannitol, decitabine or any of their components.
• Prior quizartinib use.
• Patients with known uncontrolled CNS leukemia.
• Only for frontline cohort: patients who are fit for intensive chemotherapy.
• Patients with electrolyte abnormalities at study entry defined as follows: Serum potassium < 3.5 mEq/L despite supplementation, or > 5.5 mEq/L Serum magnesium above or below the institutional normal limit despite adequate management. Serum calcium (corrected for albumin levels) above or below institutional normal limit despite adequate management.
• Patients with known significant impairment of gastrointestinal (GI) function or GI disease that may significantly alter the absorption of quizartinib.
• Patients with any other known concurrent severe and/or uncontrolled medical condition including but not limited to diabetes, cardiovascular disease including hypertension, renal disease, or active uncontrolled infection, which could compromise participation in the study. Patients on active antineoplastic or radiation therapy for a concurrent malignancy at the time of screening. Maintenance therapy, hormonal therapy, or steroid therapy for well-controlled malignancy is allowed.
• Patients with a known human immunodeficiency virus (HIV) infection.
• Patients with known positive hepatitis B or C infection by serology, with the exception of those with an undetectable viral load within 3 months. (Hepatitis B or C testing is not required prior to study entry). Subjects with serologic evidence of prior vaccination to HBV [i.e., hepatitis B surface antigen [HBs Ag]-, and anti-HBs+] may participate.
• Patients who have consumed grapefruit, grapefruit products, Seville oranges (including marmalade containing Seville oranges) or Starfruit within 3 days prior to the initiation of study treatment.
• Patients who have had any major surgical procedure within 14 days of day 1.
• Impaired cardiac function including any of the following: Screening ECG with a Fridericia's correction formula (QTcF) > 450 msec. The QTcF interval will be calculated by Fridericia's correction factor (QTcF). The QTcF will be derived from the average QTcF in triplicate. Patients are excluded if they have QTcF >= 450. Subjects with prolonged QTcF interval in the setting of RBBB (right bundle branch block) may participate upon review and approval by the medical monitor. RBBB for patients' triplicate EKGs can show false corrected QT (QTc) prolongation; therefore, the Cardiology collaborator for this study will manually review to provide an accurate reading of the QTc. Patients with congenital long QT syndrome. History or presence of sustained ventricular tachycardia requiring medical intervention. Any history of clinically significant ventricular fibrillation or torsades de pointes. Known history of second or third degree heart block (may be eligible if the patient currently has a pacemaker). Sustained heart rate of < 50/minute on pre-entry ECG. Right bundle branch block + left anterior hemiblock (bifascicular block). Complete left bundle branch block. Patients with myocardial infarction or unstable angina within 6 months prior to starting study drug. Congestive heart failure (CHF) New York (NY) Heart Association class III or IV. Atrial fibrillation documented within 2 weeks prior to first dose of study drug. Patients who are actively taking a strong CYP3A4 inducing medication.
• Patients who require treatment with concomitant drugs that prolong QT/QTc interval or strong CYP3A4 inhibitors with the exception of antibiotics, antifungals, and antivirals that are used as standard of care to prevent or treat infections and other such drugs that are considered absolutely essential for the care of the subject or if the Investigator believes that beginning therapy with a potentially QTc-prolonging medication (such as anti-emetic except for prochlorperazine) is vital to an individual subject's care while on study.
• Known family history of congenital long QT syndrome.
• Patients who are on strong CYP3A4 inhibitor will be excluded.