This phase Ib/II trial studies the best dose and side effects of venetoclax and how well it works when given with combination chemotherapy in treating patients with newly diagnosed acute myeloid leukemia or acute myeloid leukemia that has come back or does not respond to treatment. Venetoclax may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as fludarabine, cytarabine, filgrastim and idarubicin, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving venetoclax together with combination chemotherapy may work better in treating patients with acute myeloid leukemia.

PRIMARY OBJECTIVES:

I. To evaluate the safety and tolerability and to determine the dose-limiting toxicity and the maximum tolerated dose MTD of the combination of fludarabine, cytarabine, filgrastim (GCSF), idarubicin (FLAG-IDA) + venetoclax for patients with acute myeloid leukemia (AML) (Phase 1b).

II. To determine the overall activity of this combination in patients newly diagnosed or relapsed/refractory (AML) (Phase 2).

SECONDARY OBJECTIVES:

I. Determine the preliminary assessment of efficacy by response to revised International Working Group (IWG) criteria and time to response variables including overall survival (OS), event-free survival (EFS) and duration of response (DOR).

II. Determine biomarkers that may be predictive of venetoclax activity.

OUTLINE: This is a phase I, dose-escalation study of venetoclax followed by a phase II study.

INDUCTION THERAPY: Patients receive venetoclax orally (PO) on days 1-14, fludarabine intravenously (IV) over 30 minutes on days 2-6, cytarabine IV over 4 hours on days 2-6, idarubicin IV over 15-30 minutes on days 4 and 5, filgrastim subcutaneously (SC) on days 1-7, or pegfilgrastim SC after day 5. Treatment repeats every 28 days for up to 2 cycles in the absence of disease progression or unacceptable toxicity.

CONSOLIDATION THERAPY: Patients receive venetoclax PO on days 1-7, fludarabine IV over 30 minutes on days 2-4, cytarabine IV over 4 hours on days 2-4, filgrastim SC on days 1-7, or pegfilgrastim SC after days 3. Treatment repeats every 28 days for up to 6 cycles in the absence of disease progression or unacceptable toxicity. Patients may also receive idarubicin as in Induction Therapy during 1 cycle of Consolidation Therapy per the treating physician.

MAINTENANCE THERAPY: Patients receive venetoclax PO on days 1-28. Cycles repeat every 28 days for up to 1 year in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up within 30 days.

Inclusion Criteria:

• Diagnosis of AML by World Health Organization (WHO) criteria. Patients with high risk myelodysplastic syndrome (MDS) as defined by the presence of >= 10% blasts are also eligible at the discretion of the principal investigator
• Patients older than 65 who are deemed fit to receive intensive chemotherapy by the treating physician will be eligible after discussion with the principal investigator (PI).
• Eastern Cooperative Oncology Group (ECOG) performance status of =< 2
• Creatinine clearance >= 30 mL/min based on the Cockcroft-Gault equation
• Total bilirubin < 1.5 x upper limit of normal (ULN) unless increase is due to Gilbert's disease or leukemic involvement
• Aspartate aminotransferase (AST) and/or alanine aminotransferase (ALT) < 3 x ULN unless considered due to leukemic involvement
• Ability to understand and provide signed informed consent
• Male subjects must agree to refrain from unprotected sex and sperm donation from initial study drug administration until 90 days after the last dose of study drug
• Only patients who are relapsed, refractory, or intolerant of standard AML therapy will be eligible for Part 1 (minimum of 1 prior line of AML-directed therapy)

Exclusion Criteria:

• Patients with t(15;17) karyotypic abnormality or acute promyelocytic leukemia (French-American-British [FAB] class M3-AML)
• Patients having received any prior BCL2 inhibitor therapy
• Subject has known active central nervous system (CNS) involvement with AML
• Patients with New York Heart Association (NYHA) class III or IV congestive heart failure or left ventricular ejection fraction (LVEF) < 40% by echocardiogram or multi-gated acquisition (MUGA) scan
• Patients with a history of myocardial infarction within the last 6 months or unstable / uncontrolled angina pectoris or history of severe and/or uncontrolled ventricular arrhythmias
• Patients with known infection with human immunodeficiency virus (HIV) or active hepatitis B or C
• Patients with known dysphagia, short-gut syndrome, or other conditions that would affect the ingestion or gastrointestinal absorption of drugs administered orally
• Subject has any other significant medical or psychiatric history that in the opinion of the investigator would adversely affect participation in this study
• Subject has a white blood cell count > 25 x 10{9}/L. (Note: hydroxyurea is permitted to meet this criterion)
• Nursing women, women of childbearing potential (WOCBP) with positive urine pregnancy test, or women of childbearing potential who are not willing to maintain adequate contraception (a) appropriate method(s) of contraception include oral or injectable hormonal birth control, intrauterine device (IUD), and double barrier methods (for example a condom in combination with a spermicide)