This will be a phase II, open-label, non-randomized study with a safety lead-in phase. There are 3 Arms in this study each with 2 parts. If you are eligible, you will be assigned to an Arm and a part when you join the study. In each arm, you will receive a different combination of study drugs: Arm 1: nivolumab and azacitidine, Ih Arm 2: nivolumab, azacitidine, and ipilimumab, Arm 3: nivolumab, azacitidine, and venetoclax.

There are 2 parts in each arm: Part A (dose escalation) and Part B (dose expansion).

The goal of Part A of this clinical research study is to find the highest tolerable dose of the study drugs (nivolumab, azacitidine, ipilimumab, and/or venetoclax) that can be given to patients with AML. The goal of Part B of this study is to learn if the dose found in Part A can help to control AML.

Arm 1 Up to 5 groups of up to 6 participants will be enrolled in Part A of the study, and up to 110 participants will be enrolled in Part B of the study. If you are enrolled in Part A of the study, the dose of nivolumab and azacitidine you receive will depend on when you joined this study. The first group of participants will receive the starting dose levels of nivolumab and azacitidine. If intolerable side effects are seen, the next group may receive a lower dose level of nivolumab and/or azacitidine. This will continue until the highest tolerable combination dose is found.

If you are enrolled in Part B, you will receive nivolumab and azacitidine at the highest dose that was tolerated in Part A.

Arm 2 Up to 4 groups of up to 6 participants will be enrolled in Part A of the study, and up to 84 participants will be enrolled in Part B of the study. If you are enrolled in Part A of the study, the dose of nivolumab, azacitidine, and ipilimumab you receive will depend on when you joined this study. The first group of participants will receive the starting dose levels of nivolumab, azacitidine and ipilimumab. If intolerable side effects are seen, the next group may receive a lower dose level of nivolumab, azacitidine and/or ipilimumab. This will continue until the highest tolerable combination dose is found. If you are enrolled in Part B, you will receive nivolumab, azacitidine, and ipilimumab at the highest dose that was tolerated in Part A.

Arm 3 About 6-18 participants will be enrolled in Part A and up to 60 participants will be enrolled in Part B. If you are enrolled in Part A of the study, the dose of nivolumab, azacitidine, and venetoclax you receive will depend on when you joined this study. The first group of participants will receive the starting dose levels of nivolumab, azacitidine and venetoclax. If intolerable side effects are seen, the next group may receive a lower dose level of nivolumab, azacitidine and/or venetoclax. This will continue until the highest tolerable combination dose is found. If you are enrolled in Part B, you will receive nivolumab, azacitidine, and venetoclax at the highest dose that was tolerated in Part A.

Inclusion Criteria:

• ARM 1 SALVAGE COHORT: Patients with AML or biphenotypic or bilineage leukemia who have failed prior therapy; patients with AML should have failed prior therapy or have relapsed after prior therapy will be eligible for Arm 1;

• ARM 2 SALVAGE COHORT: Patients with AML who have failed up to one prior salvage therapy (i.e. salvage 1 or 2 status) will be eligible for Arm 2 relapse cohort; allogeneic stem cell transplant for patients in remission at the time of stem cell transplant will not be considered a salvage regimen; similarly, hydroxyurea if used alone will not be considered a salvage regimen

• ARM 1 AND 2 FRONT LINE OLDER COHORT: Patients age 65 years and above with previously untreated AML who are unfit for or decline standard induction therapy; prior therapy with hydroxyurea, biological or targeted therapy (e.g. FLT3 inhibitors, other kinase inhibitors), or hematopoietic growth factors is allowed, however prior therapy with chemotherapy agents for the disease under study is not allowed; patients may have received one dose of cytarabine (up to 2 g/m2 administered at presentation for control) of hyperleucocytosis

• Patients with myelodysplastic syndrome (MDS) or chronic myelomonocytic leukemia (CMML) who received therapy for the MDS or CMML and progress to AML are eligible at the time of diagnosis of AML regardless any prior therapy for MDS or CMML; the World Health Organization (WHO) classification will be used for AML; prior therapy for MDS or CMML will not be considered as a prior therapy for AML, hence such patients will be considered as frontline AML and eligible for the frontline elderly cohort

• Prior therapy with hydroxyurea, chemotherapy, biological or targeted therapy (e.g. FLT3 inhibitors, other kinase inhibitors), or hematopoietic growth factors is allowed

• Eastern Cooperative Oncology Group (ECOG) performance status =< 2

• Total bilirubin =< 2 times upper limit of normal (x ULN) (=< 3 x ULN if considered to be due to leukemic involvement or Gilbert's syndrome)

• Aspartate aminotransferase or alanine aminotransferase =< 2.5 x ULN (=< 5.0 x ULN if considered to be due to leukemic involvement)

• Serum creatinine =< 2 x ULN or glomerular filtration rate (GFR) >= 50

• Patients must provide written informed consent

• In the absence of rapidly progressing disease, the interval from prior treatment to time of initiation of 5-azacytidine and nivolumab will be at least 2 weeks OR at least 5 half-lives for cytotoxic/noncytotoxic agents; the half-life for the therapy in question will be based on published pharmacokinetic literature (abstracts, manuscripts, investigator brochure's, or drug-administration manuals) and will be documented in the protocol eligibility document; since the effect of both nivolumab and 5-azacytidine may be delayed; use of one dose of cytarabine (up to 2 g/m2) or hydroxyurea for patients with rapidly proliferative disease is allowed before the start of study therapy and during the study treatment; concurrent therapy for central nervous system (CNS) prophylaxis or continuation of therapy for controlled CNS disease is permitted

• Females must be surgically or biologically sterile or postmenopausal (amenorrheic for at least 12 months) or if of childbearing potential, must have a negative serum or urine pregnancy test within 72 hours before the start of the treatment

• Women of childbearing potential must agree to use an adequate method of contraception during the study and until 3 months after the last treatment; males must be surgically or biologically sterile or agree to use an adequate method of contraception during the study until 3 months after the last treatment; adequate methods of contraception include: total abstinence when this is in line with the preferred and usual lifestyle of the patient; periodic abstinence (e.g., calendar, ovulation, symptothermal, post-ovulation methods) and withdrawal are not acceptable methods of contraception; female sterilization (have had surgical bilateral oophorectomy with or without hysterectomy) or tubal ligation at least six weeks before taking study treatment; in case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow up hormone level assessment; male sterilization (at least 6 months prior to screening); for female patients on the study, the vasectomized male partner should be the sole partner for that patient

• Combination of any of the two following

  • Use of oral, injected or implanted hormonal methods of contraception or other forms of hormonal contraception that have comparable efficacy (failure rate < 1%), for example hormone vaginal ring or transdermal hormone contraception
  • Placement of an intrauterine device (IUD) or intrauterine system (IUS)
  • Barrier methods of contraception: condom or occlusive cap (diaphragm or cervical/vault caps) with spermicidal foam/gel/film/cream/ vaginal suppository; in case of use of oral contraception, women should have been stable on the same pill before taking study treatment; note: oral contraceptives are allowed but should be used in conjunction with a barrier method of contraception

• Women are considered post-menopausal and not of child bearing potential if they have had 12 months of natural (spontaneous) amenorrhea with an appropriate clinical profile (e.g. age appropriate, history of vasomotor symptoms) or have had surgical bilateral oophorectomy (with or without hysterectomy) or tubal ligation at least six weeks ago; in the case of oophorectomy alone, only when the reproductive status of the woman has been confirmed by follow up hormone level assessment is she considered not of child bearing potential

• Patients with graft versus host disease (GVHD) active < grade 2 who are on a stable dose of immunosuppressive therapy (tacrolimus, cyclosporine, or other) for > 2 weeks will be included; Note: subjects may be using systemic corticosteroids or topical or inhaled corticosteroids

Exclusion Criteria:

• Patients with known allergy or hypersensitivity to nivolumab, ipilimumab, 5-azacytidine, or any of their components

• Patients with a known history of severe interstitial lung disease or severe pneumonitis or active pneumonitis that is uncontrolled in the opinion of the treating physician

• Patients who have previously been treated with nivolumab and/or ipilimumab in combination with 5-azacytidine will be excluded

• Patients with a known history of any of the following autoimmune diseases are excluded:

  • Patients with a history of inflammatory bowel disease (including Crohn's disease and ulcerative colitis)
  • Patients with a history of rheumatoid arthritis, systemic progressive sclerosis (scleroderma), systemic lupus erythematosus, autoimmune vasculitis (e.g., Wegener's granulomatosis)

• Patients with organ allografts (such as renal transplant) are excluded

• Patients with active GVHD > grade 2 will be excluded; patients with recent increase in the immunosuppressive medication dose within last 2 weeks to control GVHD will not be included; patients with grade 1 or lower GVHD on =< 10 mg prednisone without any additional immunosuppressive therapies (tacrolimus, prograf, etc) will be eligible

• Patients with symptomatic CNS leukemia or patients with poorly controlled CNS leukemia

• Active and uncontrolled disease/(active uncontrolled infection, uncontrolled hypertension despite adequate medical therapy, active and uncontrolled congestive heart failure New York Heart Association [NYHA] class III/IV, clinically significant and uncontrolled arrhythmia) as judged by the treating physician

• Patients with known human immunodeficiency virus seropositivity will be excluded

• Known to be positive for hepatitis B by surface antigen expression; known to have active hepatitis C infection (positive by polymerase chain reaction or on antiviral therapy for hepatitis C within the last 6 months)

• Any other medical, psychological, or social condition that may interfere with study participation or compliance, or compromise patient safety in the opinion of the investigator

• Patients unwilling or unable to comply with the protocol

• Pregnant or breastfeeding

• Acute promyelocytic leukemia (APL)