This phase I trial studies the side effects and the best dose of cytarabine when given together with decitabine and vorinostat in treating patients with acute myeloid leukemia or myelodysplastic syndrome that has returned or has not responded to treatment. Drugs used in chemotherapy, such as cytarabine and decitabine, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Vorinostat may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Giving cytarabine together with decitabine and vorinostat may kill more cancer cells.

PRIMARY OBJECTIVES:

I. To determine the maximum tolerated dose (MTD) of the combination of decitabine, vorinostat, and cytarabine in patients with relapsed/refractory acute myeloid leukemia (AML) and select subsets of high risk leukemia/myelodysplastic syndromes (MDS).

II. To define the specific toxicities and the dose limiting toxicity (DLT) of the combination.

SECONDARY OBJECTIVES:

I. To develop a platform for specifically targeting mixed-lineage leukemia partial-tandem duplication (MLL PTD), for future efficacy studies.

II. To determine the overall response rate (ORR) of this regimen in relapsed/ refractory AML.

III. To examine the role of decitabine and vorinostat in re-expression of mixed-lineage leukemia wild type (MLL- WT) in patients with MLL PTD via correlative studies specific to patients with MLL PTD and the preliminary relationship of this to clinical response in patients with MLL PTD+ AML.

IV. To correlate the biological activity of decitabine as demethylating agent (changes in target gene methylation and gene expression, deoxyribonucleic acid [DNA] [cytosine-5-]-methyltransferase 1 [DNMT1] protein expression, global methylation) with clinical endpoints.

V. To explore the biologic role of microribonucleic acids (RNAs) in determining clinical response to the combination and achievement of the other pharmacodynamic endpoints.

OUTLINE: This is a dose-escalation study of cytarabine.

INDUCTION THERAPY: Patients receive decitabine intravenously (IV) over 1 hour on days 1-10; vorinostat orally (PO) on days 5-10; and high-dose cytarabine IV over 2 hours on days 12, 14, and 16 in the absence of disease progression or unacceptable toxicity. Patients who achieve complete remission (CR) proceed to maintenance therapy. Patients who achieve CR with incomplete blood count recovery undergo bone marrow aspiration and biopsy at count recovery or day 42 before proceeding to maintenance therapy.

MAINTENANCE THERAPY: Patients receive decitabine IV over 1 hour on days 1-5 and vorinostat PO on days 5-10. Treatment repeats every 28 days for up to 11 courses in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up for 30 days.

Inclusion Criteria:

• Patients with relapsed or refractory acute myeloid leukemia (AML)
• Patients aged between 55-59.9 with previously untreated AML will also be eligible, but these patients must be screened for AML-core binding factor (CBF)+ AML is NOT eligible in this subset of previously untreated AML patients
• Patients with relapsed or refractory high risk MDS (defined as International Prognostic Scoring System [IPSS] score >= 1.5) will also be eligible; IPSS score can be calculated any time from myelodysplastic syndrome (MDS) diagnosis at relapse/treatment failure for the purposes of trial eligibility
• Patients with secondary AML or therapy related disease (t-AML) are eligible; patients who received decitabine or 5-azacytidine as prior treatment for MDS (or AML) are eligible; patients who previously received high dose cytarabine (>= 1 gm/m^2/dose) are eligible
• If the patient has co-morbid medical illness, life expectancy attributed to this must be greater than 6 months
• Eastern Cooperative Oncology Group (ECOG) performance status =< 2
• Total bilirubin < 2.0 mg/dL
• Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT]) < 2.5 X institutional upper limit of normal
• Creatinine < 2.0 mg/dL
• New York Heart Association (NYHA) congestive heart failure (CHF) class II or better
• Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; if the patient does not agree, the patient is not eligible; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately
• Ability to understand and willingness to sign the written informed consent document
• Patients with known human immunodeficiency virus (HIV) infection without a history of acquired immune deficiency syndrome (AIDS) and with sufficiently high cluster of differentiation (CD)4 cells (> 400/mm^3) and low HIV viral loads (< 30,000 copies/ml plasma) not requiring anti-HIV therapy are eligible
• Patients must have recovered from the toxicity of prior therapy to less than grade 2

Exclusion Criteria:

• Patients who have had chemotherapy or radiotherapy within 2 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study, or those who have not recovered from adverse events (to less than grade 2) due to agents administered more than 4 weeks earlier
• Patients may not have taken valproic acid, or any other histone deacetylase inhibitor, for at least 2 weeks prior to study enrollment
• Patients receiving any other investigational agents or patients that have received other investigational agents within 14 days of enrollment
• Patients with active central nervous system disease or with granulocytic sarcoma as sole site of disease
• Patients with history of medically serious allergic reactions attributed to decitabine, vorinostat, or cytarabine or compounds of similar chemical or biologic composition that are not easily managed
• Patients with the following will be excluded: uncontrolled intercurrent illness including, but not limited to, symptomatic congestive heart failure, unstable angina pectoris, serious cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements, myocardial infarction within 6 months prior to enrollment, New York Heart Association (NYHA) class III or IV heart failure, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities; patients with medical comorbidities that will preclude safety evaluation of the combination should not be enrolled
• Patients with serious medical or psychiatric illness likely to interfere with participation in this clinical study
• Pregnant women or women who are breastfeeding; breastfeeding should be discontinued; confirmation that the subject is not pregnant must be established by a negative serum beta-human chorionic gonadotropin (beta-hCG) pregnancy test result obtained during screening; pregnancy testing is not required for post-menopausal or surgically sterilized women
• Patients with advanced malignant solid tumors are excluded; patients with active additional hematologic malignancies are excluded
• Patients with a history of neurologic toxicity with cytarabine or vorinostat are excluded
• Patients with active infection are permitted to enroll provided that the infection is under control; patients with uncontrolled infection shall not be enrolled until infection is treated and brought under control
• Patients who are unable to swallow pills are excluded
• Patients requiring warfarin are excluded