This phase II trial studies the effect of venetoclax together with busulfan, cladribine, and fludarabine in treating patients with high-risk acute myeloid leukemia or myelodysplastic syndrome who are undergoing stem cell transplant. Chemotherapy drugs, such as venetoclax, busulfan, cladribine, and fludarabine, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Adding venetoclax to the current standard of care stem cell transplant regimen of busulfan, fludarabine, and cladribine may help to control high-risk acute myeloid leukemia or myelodysplastic syndrome.

Phase 2 Portion

Primary Objective 1) To obtain preliminary evidence of efficacy as defined by 1-year progression free survival.

Secondary Objectives

To determine:

1. Safety of this regimen as per NCI toxicity criteria
2. Time to neutrophil and platelet engraftment
3. Incidence of acute and chronic GVHD
4. Relapse incidence
5. Non-relapse mortality
6. Overall survival
7. Graft versus host disease-relapse free survival (GRFS)

Phase 3 Portion

Primary Objective

1) To compare progression free survival between two arms Arm 1 Standard of Care: fludarabine + IV busulfan (FluBu) versus Arm 2 Experimental: Venetoclax + Fractionated busulfan, cladribine, and fludarabine

Secondary Objectives To compare following between two arms

1. Safety of this regimen as per NCI toxicity criteria
2. Time to neutrophil and platelet engraftment
3. Incidence of acute and chronic GVHD
4. Relapse incidence
5. Non-relapse mortality
6. Overall survival
7. Graft versus host disease-relapse free survival (GRFS)

Inclusion Criteria:

Phase II

1. Age ≥ 18 and ≤ 70 years. English and non-English speaking patients are eligible.

2. Patients with acute myeloid leukemia who have previously received induction therapy and one of the following high-risk features:

   1. ELN17 adverse risk prognostic group irrespective of remission status (see Appendix 2)
   2. Measurable residual disease positive (MRD +)
   3. Not in complete remission including complete remission without count recovery (Cri) and/or morphologic leukemia free state (MLFS), primary refractory, or relapsed disease. See Appendix 3 for details.
   4. AML secondary to MDS or MPD
   5. Therapy-related AML.
   6. Not in complete remission after one course of induction therapy

   Or

   Patients with myelodysplastic syndrome or CMML and one of the following high-risk features:

   1. Poor or Very poor cytogenetic risk group as per IPSS-R
   2. Mutated P53 or Ras pathway genes (CBL, NRAS, KRAS, NF1, PTPN1) or DNMT 3a or ASXL1 or RUNX1
   3. Maximum IPSS-R >3.5 between diagnosis and the start of the preparative regimen.
   4. ≥ 5% BM blasts at transplant
   5. Therapy-related MDS

3. HLA-identical sibling or a minimum of 7/8 matched unrelated donor, or a haploidentical related donor available

4. Subject must voluntarily sign an informed consent

5. Female subjects of childbearing potential must have negative results for pregnancy test

6. Adequate hepatic and renal function per local laboratory reference range as follows:

   • Aspartate transaminase (AST) and alanine transaminase (ALT) < 3.0X ULN
   • Bilirubin <1.5 x ULN (unless bilirubin rise is due to Gilbert's syndrome or of non-hepatic origin)
   • Subject must have adequate renal function as demonstrated by a creatinine clearance ≥ 50 mL/min; calculated by the Cockcroft Gault formula or measured by 24 hours urine collection.

Phase III

1. Age ≥ 18 and ≤ 65 years. English and non-English speaking patients are eligible.

2. Patients with acute myeloid leukemia who have previously received induction therapy and one of the following high-risk features:

   1. ELN22 adverse risk prognostic group irrespective of remission status (see Appendix
   2. Measurable residual disease positive (MRD +) including MRD + any time after induction therapy.
   3. Not in complete remission including complete remission without count recovery (Cri) and/or morphologic leukemia free state (MLFS), primary refractory, or relapsed disease. See Appendix 4 for details.
   4. AML secondary to MDS or MPD
   5. Therapy-related AML.
   6. Not in complete remission after one course of induction therapy
   7. Second or higher complete remission

   Or

   Patients with myelodysplastic syndrome and one of the following high-risk features:

   1. Poor or Very poor cytogenetic risk group as per IPSS-R
   2. Mutated P53 or Ras pathway genes (CBL, NRAS, KRAS, NF1, PTPN11) or ASXL1 or RUNX1 or moderate high, or high, or very high-risk group as per IPSS-M
   3. Maximum IPSS-R >3.5 between diagnosis and the start of the preparative regimen.
   4. ≥ 5% BM blasts at transplant
   5. Therapy-related MDS

   Or

   Patients with CMML

3. HLA-identical sibling or a minimum of 7/8 matched unrelated donor

4. Subject must voluntarily sign an informed consent

5. Female subjects of childbearing potential must have negative results for pregnancy test

6. Adequate hepatic and renal function per local laboratory reference range as follows:

   • Aspartate transaminase (AST) and alanine transaminase (ALT) < 3.0X ULN
   • Bilirubin <1.5 x ULN (unless bilirubin rise is due to Gilbert's syndrome or of non-hepatic origin)
   • Subject must have adequate renal function as demonstrated by a creatinine clearance ≥ 50 mL/min; calculated by the Cockcroft Gault formula or measured by 24 hours urine collection.

Exclusion criteria:

1. Subject is known to be positive for HIV.

2. Subject has cognitive impairments and/or is a prisoner.

3. Subject has acute promyelocytic leukemia

4. Subject has known active CNS involvement with AML.

5. Evidence of other clinically significant uncontrolled condition(s) including, but not limited to:

   1. Uncontrolled and/or active systemic infection (viral, bacterial or fungal)
   2. Chronic hepatitis B virus (HBV) or hepatitis C (HCV) requiring treatment. Note: subjects with serologic evidence of prior vaccination to HBV (i.e. hepatitis B surface (HBs) antigen negative-, anti-HBs antibody positive and anti-hepatitis B core (HBc) antibody negative) or positive anti-HBc antibody from intravenous immunoglobulins (IVIG) may participate.

6. Cardiac history of CHF requiring treatment or Ejection Fraction < 50% or unstable angina;

7. Corrected DLCO < 50% or FEV1 <65%.

8. Administration or consumption of any of the following within 3 days prior to the first dose of study drug:

   • grapefruit or grapefruit products
   • Seville oranges (including marmalade containing Seville oranges)
   • star fruit

9. Patients with cognitive impairments and/or any serious unstable pre-existing medical condition or psychiatric disorder that can interfere with safety or with obtaining informed consent or compliance with study procedures.

10. Prior allogeneic stem cell transplantation.