This phase II trial tests the effect of total marrow and lymphoid irradiation (TMLI) in combination with fludarabine and melphalan as conditioning regimen in older patients with acute myeloid leukemia or high-risk myelodysplastic syndrome that has not responded to previous treatment (refractory) and that has come back after a period of improvement (relapsed) and are undergoing a donor (allogeneic) peripheral blood stem cell (PBSC) hematopoietic cell transplant (HCT) from a matched related or unrelated donor. HCT is the only curative treatment for high-risk patients, but the side effects related to the current conditioning treatments limit the use to younger and more fit patients. TMLI is a targeted form of total body radiation that uses intensity-modulated radiation therapy to target marrow, lymph node chains, and the spleen. It is designed to reduce radiation-associated side effects and maximize the radiation therapeutic effect. Fludarabine blocks cells from making deoxyribonucleic acid (DNA) and may kill cancer cells. It is a type of purine antagonist and a type of ribonucleotide reductase inhibitor. Melphalan is in a class of medications called alkylating agents. It may kill cancer cells by damaging their DNA and stopping them from dividing. Giving chemotherapy, such as fludarabine and melphalan, and TMLI before an allogeneic transplant helps kill cancer cells in the body and helps make room in the patient's bone marrow for new blood-forming cells (stem cells to grow. When healthy stem cells from a related or unrelated donor, such as PBSC HCT, that closely match the patient's blood, are infused into a patient, they may help the patient's bone marrow make more healthy cells and platelets, an may help destroy any remaining cancer cells. Giving TMLI in combination with fludarabine and melphalan as conditioning treatment for an allogeneic PBSC HCT from a matched related or unrelated donor may be safe, tolerable, and/or effective in treating high-risk older patients with relapsed and refractory acute myeloid leukemia or high-risk myelodysplastic syndrome.

PRIMARY OBJECTIVE:

I. Assess the efficacy of the total marrow and lymphoid irradiation (TMLI)-based therapy at the recommended phase 2 dose (RP2D) of 1600 cGy prior to hematopoietic cell transplant (HCT) for older patients (≥ 50 years of age) with refractory and relapsed acute myeloid leukemia (AML) or high-risk myelodysplastic syndrome (MDS) undergoing peripheral blood stem cell (PBSC) HCT from matched related/unrelated donor, as measured by 2-years leukemia-free survival (LFS).

SECONDARY OBJECTIVES:

I. Further evaluation of safety of the TMLI-based conditioning regimen, by assessing the following: type, frequency, severity, attribution, time course and duration of adverse events, including acute/chronic GVHD, infection and delayed engraftment.

II. Estimate overall survival (OS: at 1 and 2 years post-HCT), cumulative incidence (CI) of relapse/progression (at 1 and 2 years post-HCT), and non-relapse mortality (NRM) at 100 days, 1 year and 2 years post-HCT.

III. Estimate the cumulative incidence and severity of acute graft-versus-host disease (GVHD) by day 180 using Malignant Germ Cell International Consortium (MAGIC) grading and chronic GVHD by 1 and 2 years post-HCT using National Institutes of Health (NIH) consensus criteria.

IV. Estimate the cumulative incidence of GVHD-free and relapse-free survival (GRFS) at 1-year post-HCT.

EXPLORATORY OBJECTIVES:

I. Collect longitudinal blood samples for immune analysis II. Collect longitudinal blood samples to assess presence and levels of GVHD biomarkers and inflammatory cytokines III. Collect longitudinal bone marrow samples to assess changes in the bone marrow environment after TMLI.

IV. Collect longitudinal blood samples for circulating tumor DNA (ctDNA) profiling.

V. Collect longitudinal stool samples to explore the potential effects of lower gastrointestinal (GI) tract radiation exposure on microbiome composition and HCT outcomes.

VI. In patients ≥ 50 years old, evaluate physical function and quality of life and cognitive impairment using Cancer Health Assessments Reaching Many (CHARM) assessments at baseline then frailty assessments, Patient Reported Outcomes Measurement Information Systems (PROMIS) Physical Function and Montreal cognitive assessment on day 100 and 180-, and 1-year post-HCT.

OUTLINE:

Patients receive palifermin intravenously (IV) on days -11, -10, -9, 0, 1 and 2, fludarabine IV on days -4 to -2 and melphalan IV on day -2 and undergo TMLI twice daily (BID) for 8 fractions on days -8 to -5. Patients receive allogeneic PBSC-HCT on day 0. Starting on day -1, patients also receive tacrolimus IV or orally (PO) once daily (QD) and sirolimus QD per standard of care.

Additionally, patients undergo echocardiography or multigated acquisition scan (MUGA), computed tomography (CT), urine and blood sample collection, and bone marrow biopsy throughout the study.

After completion of study treatment, patients are followed up at 30, 60, 100 and 180 days and at 1 and 2 years.

Inclusion Criteria:

• Documented informed consent of the participant and/or legally authorized representative

  • Assent, when appropriate, will be obtained per institutional guidelines

• Agreement to allow the use of archival tissue from diagnostic tumor biopsies

  • If unavailable, exceptions may be granted with study principal investigator (PI) approval

• Age: ≥ 50 years (no upper age limit)

  • Note: Patients ≥ 18 years and < 50 years are also included if they are not candidates for myeloablative conditioning regimens due to comorbidities or active disease

• Karnofsky or Lansky performance status ≥ 70

• Eligible patients will have a histopathological confirmed diagnosis of hematologic malignancy in one of the following categories:

  • Acute myeloid leukemia (AML):

    • Patients with de novo or secondary disease in unfavorable risk group including poor risk cytogenetics according to National Comprehensive Cancer Network (NCCN) guidelines for AML i.e., monosomal karyotype, -5,5q-,-7,7q-,11q23-non t(9;11), inv (3), t(3;3), t(6;9), t(9;22) and complex karyotypes (≥ 3 unrelated abnormalities), or all patient in intermediate risk groups

    • Patients with active disease:

      • Morphologically
      • Minimal residual disease (MRD) testing (MRD+ through flow cytometry, cytogenetics, or molecular assays)

  • Myelodysplastic syndrome/chronic myelomonocytic leukemia (CMML) (MDS) with ≥ 10% blast

• Patients must have an human leukocyte antigen (HLA) (A, B, C, and DRB1) identical sibling or a 8/8 (A, B, C, and DR) allele matched unrelated donor who is willing to donate primed blood stem cells

• Serum direct bilirubin ≤ 2.0 x upper limit of normal (ULN) (unless has Gilbert's disease) (within 35 days prior to day 1 of protocol therapy unless otherwise stated)

• Aspartate aminotransferase (AST) ≤ 2.5 x ULN (within 35 days prior to day 1 of protocol therapy unless otherwise stated)

• Alanine aminotransferase (ALT) ≤ 2.5 x ULN (within 35 days prior to day 1 of protocol therapy unless otherwise stated)

• Creatinine clearance of ≥ 60 mL/min per 24 hour urine test or the Cockcroft-Gault formula (within 35 days prior to day 1 of protocol therapy unless otherwise stated)

• Left ventricular ejection fraction (LVEF) ≥ 50%

  • Note: To be performed within 35 days prior to day 1 of protocol therapy

• If able to perform pulmonary function tests: Forced expiratory volume in 1 second (FEV1), forced vital capacity (FVC) and DLCO (diffusion capacity) ≥ 50% of predicted (corrected for hemoglobin)

  • Note: To be performed within 35 days prior to day 1 of protocol therapy

• If unable to perform pulmonary function tests: Oxygen (O2) saturation > 92% on room air

  • Note: To be performed within 35 days prior to day 1 of protocol therapy

• Seronegative for HIV antigen (Ag)/antibody (Ab) combo, hepatitis C virus (HCV), active hepatitis B virus (HBV) (surface antigen negative) (within 35 days prior to day 1 of protocol therapy unless otherwise stated)

  • If seropositive for HIV, HCV or HBV, nucleic acid quantitation must be performed. Viral load must be undetectable
  • HIV-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial

• Meets institutional and federal requirements for infectious disease titer requirements

  • Note: Infectious disease testing to be performed within 28 days prior to day 1 of protocol therapy

• Women of childbearing potential (WOCBP): Negative urine or serum pregnancy test (within 35 days prior to day 1 of protocol therapy unless otherwise stated)

  • If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required

• Agreement by females and males of childbearing potential to use an effective method of birth control or abstain from heterosexual activity for the course of the study through at least 6 months after the last dose of protocol therapy

  • Childbearing potential defined as not being surgically sterilized (men and women) or have not been free from menses for > 1 year (women only)

Exclusion Criteria:

• Allogeneic stem cell transplant or autologous HCT within 1 year prior to day 1 of protocol therapy

• Chemotherapy, radiation therapy, biological therapy, immunotherapy within 14 days of day 1 of protocol therapy

  • Note: Low dose chemotherapy or maintenance chemotherapy given within 7 days of planned study enrollment is permitted. These include hydroxyurea, 6-meraptopurine, oral methotrexate, vincristine, oral etoposide, and tyrosine kinase inhibitors (TKIs). TKIs can also be given up to 3-5 days before conditioning regimen

• More than three previous lines of intensive chemotherapy, where the regimen intent was to induce remission

• Co-enrollment in other clinical trials involving post-HCT maintenance interventions or any study with potential to affect disease-free survival is not allowed

• History of allergic reactions attributed to compounds of similar chemical or biologic composition to study agent

• Clinically significant uncontrolled illness

• Active infection not responding to antibiotics

• Other active malignancy. Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial

• Females only: Pregnant or breastfeeding

• Any other condition that would, in the investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures

• Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)