This phase I trial tests the safety, side effects, and best dose of eltanexor in combination with venetoclax for the treatment of patients with myelodysplastic syndrome (MDS) or acute myeloid leukemia (AML) that has come back after a period of improvement (relapsed) or that has not responded to previous treatment (refractory). Eltanexor works by trapping "tumor suppressing proteins" within the cell, thus causing the cancer cells to die or stop growing. Venetoclax is in a class of medications called B-cell lymphoma-2 (BCL-2) inhibitors. It may stop the growth of cancer cells by blocking Bcl-2, a protein needed for cancer cell survival. Giving eltanexor together with venetoclax may be safe, tolerable and/or effective in treating patients with relapsed or refractory MDS or AML.

Primary objective:

• To establish the safe and biologically effective dose (BED) of eltanexor in combination with venetoclax in patients with R/R MDS and/or AML

Secondary objectives:

• To estimate the complete remission (CR) rate with eltanexor and venetoclax in patients with R/R MDS and/or AML
• To assess the overall response rate (ORR) following treatment with eltanexor/venetoclax
• To assess the overall survival of patients
• To assess the progression free survival (PFS) and duration of response (DOR) in patients treated with eltanexor/venetoclax

Exploratory objectives:

• To assess differential response between MDS and AML cohorts
• To develop and evaluate a phenotypic flow-based assay to predict response to eltanexor/venetoclax
• To assess the effect of mutational changes on response to eltanexor/venetoclax
• To measure the rates of measurable residual disease with eltanexor/venetoclax

OUTLINE: This is a dose-escalation study of eltanexor in combination with venetoclax.

Patients receive eltanexor orally (PO) once per day (QD) for 5 days per week for 14, 21, or 28 days every cycle, and venetoclax PO QD on days 1-14 of each cycle. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients undergo bone marrow aspiration and biopsy and blood sample collection throughout the study.

After completion of study treatment, patients are followed up every 3 months for up to 24 months.

Inclusion Criteria:

- Age >/= 18 years at the time of signing the Informed Consent Form (ICF); must voluntarily sign an ICF; and must be able to meet all study requirements.

For Myelodysplastic Syndrome (MDS):

Morphologically confirmed diagnosis of MDS with increased blasts (>/= 5%), with a prior DNA methyltransferase inhibitor (DNMTi) treatment and progression after 2 cycles or stable disease after 4 cycles

For Acute Myeloid Leukemia (AML):

Morphologically confirmed diagnosis of AML in accordance with WHO diagnostic criteria that is relapsed or refractory following >/= 1 line(s) of therapy.

• WBC must be less than 25,000/ul prior to study start (hydroxyurea allowed).
• A bone marrow aspirate must be performed, and tissue collected for entrance to the trial unless circulating blasts >/= 5% in which case, peripheral blood can be used.
• Eastern Cooperative Oncology Group Performance Status of 0 - 2.
• Must have adequate hepatic and renal function as demonstrated by the following:

ALT(SGPT) and/or AST (SGOT) </= 3x upper limit of normal (ULN); Direct bilirubin </= 1.5 x ULN; or Total bilirubin </= 2.5x ULN (known Gilbert's Syndrome as cause of elevated bilirubin is allowed); Calculated creatinine clearance > 50 ml/min (per the Cockroft-Gault formula).

- Willingness to abide by all study requirements, including contraception, maintenance of a pill diary, and acceptance of recommended supportive care medications.

Exclusion Criteria:

• Anticancer therapy, including investigational agents </= 2 weeks or </= 5 half-lives of the drug, whichever is shorter, prior to C1D1. (Use of hydroxyurea is permitted).
• Inadequate recovery from toxicity attributed to prior anti-cancer therapy to </= Grade 1 (NCI CTCAE v5.0), excluding alopecia or fatigue.
• Prior treatment with SINE compounds or other inhibitors of XPO1.
• History of allogeneic hematopoietic stem cell transplant (HCT), or other cellular therapy product, within 3 months.
• Active acute or chronic GVHD requiring calcineurin inhibitors or steroid dosing >/= 10mg/day or patients within 4 weeks of stopping calcineurin inhibitors for GVHD.
• Radiation therapy or major surgery within 3 weeks.
• Active, uncontrolled infection. Patients with infection under active treatment and controlled with antibiotics are eligible. Prophylaxis, even if parenteral, is acceptable.
• Inability to swallow oral medications.
• Active documented central nervous system leukemia.
• Second active malignancy within past 2 years except for basal or squamous cell carcinoma of the skin, ductal carcinoma of breast in situ or cervical carcinoma in situ.
• Women of childbearing age or potential must have negative pregnancy test and must not be actively breastfeeding to enroll on the study
• Clinically significant cardiovascular disease with major event or cardiac intervention within the past 6 months (e.g. percutaneous intervention, coronary artery bypass graft, documented cardiac heart failure) as determined by the investigator.
• Any condition not listed but deemed by the investigator to make the patient a poor candidate for clinical trial and/or treatment with investigational agents.