This phase I trial studies the side effects and determine the best dose of prexasertib (LY2606368) when given together with cytarabine and fludarabine in patients with acute myeloid leukemia or high-risk myelodysplastic syndrome that has returned after a period of improvement or no longer responds to treatment. Prexasertib (LY2606368) may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as cytarabine and fludarabine, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving prexasertib (LY2606368) together with cytarabine and fludarabine may work better in treating patients with acute myeloid leukemia or myelodysplastic syndrome.

PRIMARY OBJECTIVES:

I. To determine the safety and toxicity profile of combination therapy of prexasertib (LY2606368), fludarabine and cytarabine in patients with relapsed or refractory (first or second salvage) acute myeloid leukemia (AML) or high-risk myelodysplastic syndrome (HRMDS).

SECONDARY OBJECTIVES:

I. To determine the complete response (CR), CR with incomplete count recovery (CRi), morphologic leukemia free state (MLFS) and partial response (PR) rates.

II. To determine the duration of response. III. To determine the disease-free survival. IV. To determine the overall survival with combination therapy of LY2606368, fludarabine and cytarabine in patients with relapsed or refractory AML/HRMDS.

V. To determine the correlative studies including but not limited to total and phosphor histone (H2AX), checkpoint kinase-2 (Chk1), checkpoint kinase-2 (Chk2), ras protein specific guanine nucleotide releasing factor 1 (Cdc25), retinoblastoma-associated protein (Rb), cyclin-dependent kinase (CDK), protein kinase B (AKT) in AML cells by flow cytometry and/or Western blot.

OUTLINE: This is a dose-escalation study of prexasertib (LY2606368).

Patients =/< 65 years of age: receive fludarabine intravenously (IV) over approximately 2 hours on days 1-4, cytarabine IV over 4 hours on days 1-4, and prexasertib (LY2606368) IV over approximately 2 hours on days 1, 3, and 4 or on days 1-4.

Patients > 65 years of age: receive fludarabine IV over approximately 2 hours on days 1-3, cytarabine IV over 4 hours on days 1-3, and prexasertib (LY2606368) IV over approximately 2 hours on days 1, 3, and 4 or on days 1-4. Treatment for both age groups repeats every 28 days for up to 5 courses in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients in remission who do not start new treatment are followed up every 2-3 months for up to 2 years.

Inclusion Criteria:

• All patients with histologically or cytologically confirmed relapsed or refractory acute myeloid leukemia (AML) (except acute promyelocytic leukemia) or relapsed or refractory high-risk myelodysplastic syndrome (HRMDS) (intermediate 2 [Int-2] or higher risk by International Prognostic Scoring System [IPSS]); patients with chronic myelomonocytic leukemia (CMML) can be enrolled if they can be classified as HRMDS using MDS criteria; patients should not have received more than one salvage therapy; second induction regimen or stem cell transplant in remission will be considered salvage therapy; refractory subjects, up to second consecutive salvage
• Patients must have a performance status of 0-2 (Eastern Cooperative Oncology Group [ECOG] scale)
• Serum creatinine less than or equal to 1.3 mg/dL and/or creatinine clearance > 40 mL/min
• Bilirubin less than or equal to 1.5 mg/dl (unless due to Gilbert's syndrome)
• Serum glutamic-oxaloacetic transaminase (SGOT)/aspartate transaminase (AST) or serum glutamate pyruvate transaminase (SGPT)/alanine transferase (ALT) less than or equal to 2.5 X the upper limit of normal (ULN) for the reference lab
• Patients must have normal cardiac ejection fraction (left ventricular ejection fraction [LVEF] >/= 45%)
• Corrected QT (QTc) interval =/< 470 msecs, no familial history of QTc prolongation or ventricular arrhythmias
• Female patients must not be pregnant or lactating; female patients of childbearing potential (including those < 1 year post-menopausal) and male patients must agree to use contraception
• Patients who have received prior stem cell transplantation will be allowed to enroll as long as prior transplantation has been at least 3 months before enrollment in the trial and any transplant related toxicities have subsided to grade 1 or less

Exclusion Criteria:

• Patients must not have untreated or uncontrolled life-threatening infection
• Patients must not have been treated with CHK1/2 inhibitors
• Patients must not have received chemotherapy and/or radiation therapy within 2 weeks of start of protocol treatment; hydroxyurea is allowed up to 48 hours prior to starting therapy in the setting of rapidly proliferating disease
• Patients must not have received an investigational anti-cancer drug within two weeks of start of protocol treatment
• Patients must not have active central nervous system leukemia; patients with history of central nervous system (CNS) leukemia with no evidence of active CNS disease may be enrolled; maintenance intrathecal chemotherapy for adequately treated CNS involvement with leukemia is allowed with approval from the study supporter
• Patients must not have significant cardiac co-morbidity including: history of acute coronary syndromes (including myocardial infarction and unstable angina) within 12 months; coronary angioplasty or stenting within 6 months; history or evidence of current >/= class III congestive heart failure as defined by the New York Heart Association (NYHA); patients with intra-cardiac defibrillators or permanent pacemakers