This randomized phase II trial studies how well treosulfan and fludarabine phosphate, with or without total body irradiation before donor stem cell transplant works in treating patients with myelodysplastic syndrome or acute myeloid leukemia. Giving chemotherapy, such as treosulfan and fludarabine phosphate, and total-body irradiation before a donor stem cell transplant helps stop the growth of cancer cells. It may also stop the patient's immune system from rejecting the donor's stem cells. The donated stem cells may replace the patient's immune cells and help destroy any remaining cancer cells (graft-versus-tumor effect). Sometimes the transplanted cells from a donor can also make an immune response against the body's normal cells. Giving tacrolimus before and mycophenolate mofetil after the transplant may stop this from happening.

PRIMARY OBJECTIVES:

I. To determine the better of two treosulfan-based conditioning regimens in patients with myelodysplastic syndrome (MDS) and acute myeloid leukemia (AML), by comparing 6-month progression-free survival.

SECONDARY OBJECTIVES:

I. Determine the effects of two conditioning regimens on changes in gene expression profiles, and evaluate the association of gene expression profiles and disease relapse.

II. Determine the incidence of progression-free survival at 1 year and 2 years after hematopoietic cell transplantation (HCT).

III. Evaluate overall survival (OS) at 6 months, at 1 year and at 2 years after HCT.

IV. Determine the incidence of grades II-IV acute graft-versus-host disease (GVHD).

V. Determine the incidence of chronic GVHD.

VI. Determine donor chimerism around days +28 and +84.

CONDITIONING REGIMEN:

Arm A: Patients receive treosulfan intravenously (IV) over 2 hours on days -6 to -4 and fludarabine phosphate IV over 30 minutes on days -6 to -2.

Arm B: Patients receive treosulfan and fludarabine phosphate as in Arm A and undergo low-dose total-body irradiation (TBI) on day 0.

TRANSPLANT: Patients in both arms undergo allogeneic peripheral blood stem cell (PBSC) transplant or bone marrow transplant on day 0.

GVHD PROPHYLAXIS: Patients with a related donor receive tacrolimus orally (PO) every 8 or 12 hours on days -3 to 56 with taper to day 180. Beginning 4-6 hours after PBSC infusion, patients also receive mycophenolate mofetil PO every 12 hours to day 28. Patients with an unrelated donor receive tacrolimus PO every 8 or 12 hours on days -3 to 100 with taper to day 180. Beginning 4-6 hours after PBSC infusion, patients also receive mycophenolate mofetil PO every 8 hours to day 40 with taper to day 96.

NOTE: Patients with related donors eligible for FHCRC protocol 2545 may receive cyclosporine IV, instead of tacrolimus, beginning on day -3 to day 50 with a taper to day 180.

After completion of study treatment, patients are followed up periodically.

Inclusion Criteria:

• MDS, myelodysplastic syndrome/myeloproliferative neoplasia overlap disorders (including chronic myelomonocytic leukemia [CMML], and MDS/myeloproliferative neoplasm [MPN] unclassifiable syndromes)
• AML, other than acute promyelocytic leukemia (APL), in first or second remission or with minimal residual disease
• With Karnofsky index or Lansky Play-Performance scale > 70% on pre-transplant evaluation
• Able to give informed consent (if > 18 years), or with a legal guardian capable of giving informed consent (if < 18 years)
• Patients with previous autologous or allogeneic HCT are allowed to enroll
• DONOR: Human leukocyte antigen (HLA)-identical related donors or
• DONOR: Unrelated donors matched for HLA-A, B, C, DRB1, and DQB1 as defined by high resolution deoxyribonucleic acid (DNA) typing; mismatch for one HLA allele is allowed
• DONOR: Donors able to undergo peripheral blood stem cell collection or bone marrow harvest
• DONOR: Donors in good general health, with a Karnofsky or Lansky play performance score > 90%
• DONOR: Donors able to give informed consent (if > 18 years), or with a legal guardian capable of giving informed consent (if < 18 years)

Exclusion Criteria:

• Receiving umbilical cord blood
• With impaired cardiac function as evidenced by ejection fraction < 35% (or, if unable to obtain ejection fraction, shortening fraction of < 26%) or cardiac insufficiency requiring treatment or symptomatic coronary artery disease; patients with a shortening fraction < 26% may be enrolled if approved by a cardiologist
• With impaired pulmonary function as evidenced by partial pressure of oxygen (pO2) < 70 mm Hg and carbon monoxide diffusing capability test (DLCO) < 70% of predicted or pO2 < 80 mm Hg and DLCO < 60% of predicted; (or, for pediatric patients unable to perform pulmonary function tests, then oxygen (O2) saturation < 92% on room air), or receiving supplementary continuous oxygen
• With impaired renal function as evidenced by creatinine-clearance < 50% for age, weight, height or serum creatinine > 2 x upper limit of normal or dialysis-dependent
• With hepatic dysfunction as evidenced by total bilirubin > 2.0 x upper limit of normal or evidence of synthetic dysfunction or severe cirrhosis
• With hepatic dysfunction as evidenced by aspartate aminotransferase (AST) > 2.0 x upper limit of normal or evidence of synthetic dysfunction or severe cirrhosis
• With active infectious disease requiring deferral of conditioning, as recommended by an infectious disease specialist
• With human immunodeficiency virus (HIV)-positivity or active infectious hepatitis
• With central nervous system (CNS) leukemic involvement not clearing with intrathecal chemotherapy, cranial irradiation or both prior to initiating conditioning (day -6)
• Patients with active non-hematological malignancies (except non-melanoma skin cancers) or those with non-hematological malignancies who have been rendered with no evidence of disease, but have a greater than 20% chance of having disease recurrence within 5 years; this exclusion does not apply to patients with non-hematologic malignancies that do not require therapy
• With life expectancy severely limited by diseases other than malignancy
• Women who are pregnant or lactating
• With known hypersensitivity to treosulfan or fludarabine (fludarabine phosphate)
• Receiving another experimental drug within 4 weeks before initiation of conditioning (day -6)
• Unable to give informed consent (if > 18 years) or with a legal guardian (if < 18 years) unable to give informed consent
• DONOR: Individuals deemed unable to undergo marrow harvesting or PBSC mobilization and leukapheresis
• DONOR: Individuals who are HIV-positive
• DONOR: Individuals with active infectious hepatitis
• DONOR: Females with a positive pregnancy test
• DONOR: Persons unable to give informed consent (if > 18 years) or with a legal guardian (if < 18 years) unable to give informed consent