This phase I trial studies the best dose of total body irradiation when given with cladribine, cytarabine, filgrastim, and mitoxantrone (CLAG-M) or idarubicin, fludarabine, cytarabine and filgrastim (FLAG-Ida) chemotherapy reduced-intensity conditioning regimen before stem cell transplant in treating patients with acute myeloid leukemia, myelodysplastic syndrome, or chronic myelomonocytic leukemia that has come back (relapsed) or does not respond to treatment (refractory). Giving chemotherapy and total body irradiation before a donor peripheral blood stem cell transplant helps kill cancer cells in the body and helps make room in the patient's bone marrow for new blood-forming cells (stem cells) to grow. When the healthy stem cells from a donor are infused into a patient, they may help the patient's bone marrow make more healthy cells and platelets and may help destroy any remaining cancer cells. Sometimes the transplanted cells from a donor can attack the body's normal cells called graft versus host disease. Giving cyclophosphamide, cyclosporine, and mycophenolate mofetil after the transplant may stop this from happening.

OUTLINE: This a dose-escalation study of TBI. Patients are assigned to 1 of 2 arms.

ARM I: Patients receive filgrastim (G-CSF) subcutaneously (SC) daily on days -9 to -4, cladribine intravenously (IV) over 2 hours daily on days -8 to -4, cytarabine IV over 2 hours daily on days -8 to -4, and mitoxantrone IV over 60 minutes daily on days -8 to -6. If white blood cell (WBC) > 20,000/uL, filgrastim on days -9 and -8 may be omitted at physician discretion. Patients undergo TBI on either day -1 or 0 and HCT on day 0.

GVHD PROPHYLAXIS: Patients receive cyclophosphamide IV over 1-2 hours daily on days 3-4, cyclosporine IV over 1-2 hours twice daily (BID) on days 5-60, and mycophenolate mofetil IV or orally (PO) BID on days 5-28 (transplant with related donors) or three times daily (TID) on days 5-35 (transplant with unrelated donors). After day 60, patients continue to receive cyclosporine tapered through day 180 at the discretion of the treating physician in the absence of GVHD.

ARM II: Patients receive G-CSF SC daily on days -9 to -4, fludarabine IV over 30 minutes daily on days -8 to -4, cytarabine IV over 2 hours daily on days -8 to -4, and idarubicin IV over 60 minutes daily on days -8 to -6. If white blood cell (WBC) > 20,000/uL, filgrastim on days -9 and -8 may be omitted at physician discretion. Patients undergo TBI on either day -1 or 0 and HCT on day 0. Patients receiving an unrelated cord blood (UCB) transplant also receive cyclophosphamide IV on day -2.

GVHD PROPHYLAXIS: Patients who received peripheral blood stem cell transplant receive cyclophosphamide IV over 1-2 hours daily on days 3-4, cyclosporine IV over 1-2 hours BID on days 5-60, and mycophenolate mofetil IV or PO BID on days 5-28 (transplant with related donors) or TID on days 5-35 (transplant with unrelated donors). After day 60, patients continue to receive cyclosporine tapered through day 180 at the discretion of the treating physician in the absence of GVHD. Patients who received an UCB transplant receive cyclosporine IV, over 1 hour, every 8 hours (Q8H) from day -3 until day +100 and mycophenolate mofetil IV or PO TID on days 5-35. After day 100 patients continue to receive cyclosporine tapered up to at least 6 months post-transplant.

Patients in both arms undergo multigated acquisition scan (MUGA) or echocardiography, and x-ray imaging during screening and as clinically indicated or per standard practice. Patients also undergo bone marrow biopsy and aspirate during screening, day 28, day 80 and at 1 year. Additionally, patients undergo blood sample collection throughout the study.

After completion of study treatment, patients are followed up at 100 days, at 6, 12, and 24 months post-transplant.

Inclusion Criteria:

• Age >= 18 years with an Hematopoietic Cell Transplantation-Comorbidity Index (HCT-CI) =< 5 for patients over 60 years -(Enrollment of patients >= 75 years of age will require case presentation at the transplant Patient Care Conference (PCC) and approval by consensus)

• Acute myeloid leukemia (AML) (2016 World Health Organization [WHO] criteria) that is either primary refractory (as defined by failure of 2 cycles of 7+3-like chemotherapy, 1 cycle of high-dose cytarabine-based chemotherapy, or at least 2 cycles of venetoclax in combination with other therapies), or is in untreated or unsuccessfully treated first or subsequent relapse. Patients in morphological remission (i.e. < 5% blasts in the bone marrow) but evidence of minimal residual disease (MRD) by multiparameter flow cytometry, cytogenetics/fluorescence in situ hybridization (FISH), or molecular means will be eligible for trial participation. Patients with relapsed or refractory acute leukemia of ambiguous lineage (acute undifferentiated leukemia, mixed phenotype acute leukemia) that is either primary refractory or is in untreated or unsuccessfully treated first or subsequent relapse are also eligible

• Subjects with previously treated myelodysplastic syndrome (MDS) and chronic myelomonocytic leukemia (CMML), defined as prior treatment with at least one hypomethylating agent (HMA; azacitidine and/or decitabine) whose disease progressed, relapsed, or was refractory to HMA treatment as follows: 1) patients who have failed at least 4 cycles of monotherapy with azacitidine or decitabine, 2) patients who received at least 2 cycles of HMA in combination with another therapeutic agent. Subjects with MDS and CMML who failed at least 1 cycle of induction chemotherapy will be also eligible

• The use of hydroxyurea prior to initiation of study treatment is allowed. Patients with symptoms/signs of hyperleukocytosis, WBC > 100,000/uL or with concern for other complications of high tumor burden (e.g. disseminated intravascular coagulation) can be treated with leukapheresis or may receive up to 2 doses of cytarabine (up to 500 mg/m^2 per dose) prior to start of study treatment

• Karnofsky score >= 70; Eastern Cooperative Oncology Group (ECOG) 0-1

• Adequate cardiac function defined as absence of decompensated congestive heart failure and/or uncontrolled arrhythmia and left ventricular ejection fraction >= 45%

• Bilirubin =< 2.5 x institutional upper limit of normal unless elevation is thought to be due to hepatic infiltration by AML, Gilbert's syndrome, or hemolysis

• Adequate pulmonary function defined as absence of oxygen (O2) requirements and either carbon monoxide diffusing capability test (DLCO) correct >= 70% mmHg or DLCO corrected 60-69% mmHg and partial pressure of oxygen (pO2) >= 70 mmHg

• Serum creatinine =< 1.5 mg/dL

• Prior autologous HCT is permissible if relapse occurred > 3 months but =< 6 months after HCT

• Prior TBI-containing allogeneic HCT up to 3 Gy is permissible if > 6 months after HCT

• A human leukocyte antigen (HLA)-matched or near-matched related, unrelated or haploidentical donor for collection of stimulated peripheral blood stem cells or HLA-matched or near-matched cord blood unit must be identified and readily available

• Women of childbearing potential and men must agree to use adequate contraception beginning at the signing of the consent until at least 12 months post-transplant

• Patients may have previously received chemotherapy with a mitoxantrone, idarubicin- or cladribine/fludarabine-based regimen for MDS or AML. If the patient has received CLAG-M or FLAG-Ida before and has been sensitive to this regimen, eligibility will be determined on a case-by-case basis by the study principal investigator (PI)

• Ability to understand and sign a written informed consent document (or legal representative)

• DONOR: Patients must have an HLA-matched related donor or an HLA-matched unrelated donor, or haploidentical donor who meets standard FHCC and/or National Marrow Donor Program (NMDP) or other donor center criteria for peripheral blood stem cell (PBSC) donation as follows:

  • Related donor: related to the patient and genotypically or phenotypically identical for HLA-A, B, C, DRB1 and DQB1. Phenotypic identity must be confirmed by high-resolution typing

  • Unrelated donor:

    • Matched for HLA-A, B, C, DRB1 and DQB1 by high resolution typing; OR
    • Mismatched for a single allele without antigen mismatching at HLA-A, B, or C as defined by high resolution typing but otherwise matched for HLA-A, B, C, DRB1 and DQB1 by high resolution typing
    • Donors are excluded when preexisting immunoreactivity is identified that would jeopardize donor hematopoietic cell engraftment. The recommended procedure for patients with 10 of 10 HLA allele level (phenotypic) match is to obtain panel reactive antibody (PRA) screens to class I and class II antigens for all patients before HCT. If the PRA shows > 10% activity, then flow cytometric or B and T cell cytotoxic cross matches should be obtained. The donor should be excluded if any of the cytotoxic cross match assays are positive. For those patients with an HLA class I allele mismatch, flow cytometric or B and T cell cytotoxic cross matches should be obtained regardless of the PRA results. A positive anti-donor cytotoxic crossmatch is an absolute donor exclusion
    • Patient and donor pairs homozygous at a mismatched allele in the graft rejection vector are considered a two-allele mismatch, i.e., the patient is A*0101 and the donor is A*0102, and this type of mismatch is not allowed

• Haploidentical donor:

  • Donors must be haploidentical relatives of the patients. Donor-recipient compatibility will be tested through HLA typing at high resolution for the HLA loci (-A, -B, -C, -DRB1, -DQB1). Donor and recipient should share at least 5/10 HLA loci.

  • Age ≥ 12 years

  • Weight ≥ 40 kg.

  • Ability of donors younger than 18 years of age to undergo apheresis without use of a vascular access device. Vein check must be performed and verified by an apheresis nurse prior to arrival.

  • Donor must meet the selection criteria as defined by the Foundation of the Accreditation of Cell Therapy (FACT) and will be screened per the American Association of Blood Banks (AABB) guidelines.

  • In case of more available haploidentical donors, selection criteria should include, in this order:

    • For cytomegalovirus (CMV) seronegative recipients, a CMV seronegative donor

    • Red Blood Cell compatibility

      • i. RBC cross match compatible
      • ii. Minor ABO incompatibility
      • iii. Major ABO incompatibility

Exclusion Criteria:

• Patients >= 18 years being treated at Seattle Children's Hospital

• Active central nervous system (CNS) disease

• Concomitant illness associated with a likely survival of < 1 year

• Active systemic fungal, bacterial, viral, or other infection, unless disease is under treatment with anti-microbials and/or controlled or stable. Patients with fever thought to be likely secondary to myeloid malignancy are eligible

• Known hypersensitivity or contraindication to any study drug used in this trial

• Pregnancy or lactation

• Concurrent treatment with any other approved or investigational anti-leukemia agent

• Haploidentical donor exclusion criteria:

  • Since detection of anti-donor-specific antibodies (anti-DSA) is associated with higher graft rejection rate, patients will be screened for anti-DSA pre-transplant. Patient with DSA mean fluorescent intensity (MFI) <5000 after desensitization treatment, will be considered eligible to participate in the study.