This randomized phase II trial studies how well venetoclax and sequential busulfan, cladribine, and fludarabine phosphate before donor stem cell transplant work in treating patients with acute myelogenous leukemia or myelodysplastic syndrome. Giving chemotherapy before a donor peripheral blood stem cell transplant helps kill cancer cells in the body and helps make room in the patient's bone marrow for new blood-forming cells (stem cells) to grow. When the healthy stem cells from a donor are infused into a patient, they may help the patient's bone marrow make more healthy cells and platelets and may help destroy any remaining cancer cells.

PRIMARY OBJECTIVE:

I. To compare progression free survival of two schedules of venetoclax, timed sequential busulfan, cladribine and fludarabine conditioning regimen in patients with acute myelogenous leukemia (AML) and myelodysplastic syndrome (MDS).

SECONDARY OBJECTIVES:

I. Compare overall survival between the two schedules. II. Compare non relapse mortality between the two schedules. III. Compare neutrophil and platelet engraftment between the two schedules. IV. Compare acute and chronic graft-versus-host disease (GVHD) between the two schedules.

V. Compare cumulative incidence of relapse between the two schedules. VI. Compare grade III/IV toxicity between the two schedules.

TERTIARY OBJECTIVES:

I. To study chemotherapy resistance. II. To study deoxyribonucleic acid (DNA) damage. III. To study immune recovery and cytokines (both in plasma and cells). IV. To study BCL-2 family expression, stem cell surface markers and intracellular signaling markers in AML cells at the time of relapse.

OUTLINE:

PREPARATIVE REGIMEN: Patients are randomized to 1 of 2 treatment arms.

ARM I: Patients receive venetoclax orally (PO) once daily (QD) on days -22 to -3 and busulfan intravenously (IV) over 3 hours on days -13 and -12. Patients then receive fludarabine phosphate IV over 1 hour, cladribine IV over 2 hours, and busulfan IV over 3 hours on days -6 to -3.

ARM II: Patients receive venetoclax PO QD on days -22 to -3 and busulfan IV over 3 hours on days -20 and -13. Patients then receive fludarabine phosphate IV over 1 hour, cladribine IV over 2 hours, and busulfan IV over 3 hours on days -6 to -3.

TRANSPLANT: Patients undergo allogeneic peripheral blood stem cell transplant (PBSCT) on day 0.

After completion of study treatment, patients are followed up for 2.5 years.

Inclusion Criteria:

• Patients with biopsy-proven acute myeloid leukemia or myelodysplastic syndrome with persistent disease or in remission
• Human leukocyte antigen (HLA)-identical sibling or 8/8 matched unrelated donor transplant
• Patients age 18 to 70 years old; eligibility for pediatric patients will be determined in conjunction with an MD Anderson Cancer Center (MDACC) pediatrician; patients age 2-17 years may be enrolled after at least 10 adults (ages 18-70 years old) have been assessed for safety at day 30
• Direct bilirubin < 1 mg/dl
• Alanine aminotransferase (ALT) < 3 times upper limit of normal
• Creatinine clearance > 50 ml/min (calculated creatinine clearance is permitted)
• Forced expiratory volume in 1 second (FEV1) >= 50% of expected corrected for hemoglobin and/or volume
• Forced vital capacity (FVC) >= 50% of expected corrected for hemoglobin and/or volume
• Diffusing capacity of the lungs for carbon monoxide (DLCO) >= 50% of expected corrected for hemoglobin and/or volume
• Children unable to perform pulmonary function tests (e.g., less than 7 years old) pulse oximetry of >= 92% on room air
• Left ventricular ejection fraction (LVEF) >= 40%
• Patient, legally authorized representative (LAR), or parent able to sign informed consent; able to give assent for patients age 7-17
• Negative beta human chorionic gonadotropin (HCG) test in a woman with child bearing potential, defined as not post-menopausal for 12 months or no previous surgical sterilization; women of child bearing potential must be willing to use an effective contraceptive measure while on study
• Performance score of >= 70 by Karnofsky/Lansky or performance status (PS) 0 or 1 (Eastern Cooperative Oncology Group [ECOG] =< 1)

Exclusion Criteria:

• Prior allogeneic or autologous transplantation
• Uncontrolled infections
• Human immunodeficiency virus (HIV) seropositivity
• Hematopoietic cell transplantation (HCT) co-morbidity index score > 3; the principal investigator is the final arbiter of eligibility for comorbidity score > 3
• Patients with prior coronary artery disease