This phase I/II trial studies the side effects and best dose of venetoclax when given together with carmustine, etoposide, cytarabine, and melphalan before stem cell transplant in treating participants with non-Hodgkin lymphoma that has come back or does not respond to treatment. Drugs used in chemotherapy, such as venetoclax, carmustine, etoposide, cytarabine, and melphalan, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving chemotherapy before a stem cell transplant helps kill any cancer cells that are in the body and helps make room in the patient?s bone marrow for new blood-forming cells (stem cells) to grow.

PRIMARY OBJECTIVES:

I. Determine the maximum tolerated dose (MTD) of venetoclax that can be safely combined with carmustine, etoposide, cytarabine, and melphalan (BEAM) prior to autologous stem cell transplant which will the recommended phase II dose (RP2D).

II. Determine the safety and efficacy of venetoclax as measured by overall response rate (ORR) at day 100, 12-month survival and freedom from relapse (FFR-12).

SECONDARY OBJECTIVES:

I. Long term effects (progression-free survival [PFS] and overall survival [OS]) of addition of venetoclax to BEAM.

II. Correlation of response and survival with expression of BCL-2, BCL-XL, and MCL-1 as measured by immunohistochemistry (IHC).

OUTLINE: This is a dose-escalation study of venetoclax.

Participants receive venetoclax orally (PO) once daily (QD) on days -10 to -1, carmustine intravenously (IV) on day -6, etoposide IV twice daily (BID) on days -5 to -2, cytarabine IV BID on days -5 to -2, and melphalan IV on day -1. Participants then undergo hematopoietic stem cell transplantation on day 0.

After completion of study treatment, participants are followed up for 2 years.

Inclusion Criteria:

• Subjects must have histologically confirmed diagnosis of non-Hodgkin?s lymphoma that has relapsed, or is refractory, after upfront induction therapy. Excluded histologies are T-cell lymphomas, post-transplant lymphoproliferative disorder, Burkitt lymphoma, lymphoblastic lymphoma, chronic lymphocytic leukemia/small lymphocytic lymphoma. All other histologies are eligible that include but not limited to: diffuse-large B-cell lymphoma, follicular lymphoma (grades I, II, and III), marginal zone lymphoma, transformed indolent lymphoma, grey zone lymphoma, and undifferentiated B-cell lymphoma. Patients with non-Hodgkin's lymphoma (NHL) who are at high risk of relapse can be enrolled in sustained partial response (PR) after induction chemotherapy (PR1)
• Expected survival of more than six months
• Karnofsky performance status >= 80%
• Within 1 week prior to initiation of treatment: Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) < 3 x upper limits of normal (ULN) unless due to disease
• Within 1 week prior to initiation of treatment: Total bilirubin < 2 x ULN unless due to disease
• Within 1 week prior to initiation of treatment: Calculated glomerular filtration rate (GFR) 30 ml/min
• Within 1 week prior to initiation of treatment: Absolute neutrophil count (ANC) > 500 cells/mm^3
• Within 1 week prior to initiation of treatment: Platelet count > 50 mm^3
• Left ventricular ejection fraction >= 40%
• Diffusion capacity of carbon monoxide (DLCO) >= 50% predicted
• Ability to collect 2 x 10^6/kg CD34+ cells for transplantation
• Patient must be otherwise eligible for autologous stem cell transplantation (ASCT) per local institutional guidelines
• No serious disease, or condition, that, in the opinion of the investigator, would compromise the patient?s ability to participate in the study
• Subjects must have the ability to understand and the willingness to sign a written informed consent document

Exclusion Criteria:

• Subjects who sustained a complete metabolic response (CMR) by positron emission tomography (PET)-computed tomography (CT) (Deauville score of =< 3) after salvage chemotherapy unless lymphoma relapsed less than 12 months from the first day of last cycle of induction chemotherapy OR patient required more than 2 lines of salvage chemotherapy to sustain a CMR
• Subjects receiving any other investigational agents
• Prior treatment with venetoclax
• Patients with central nervous system (CNS) involved by lymphoma can be included if CNS disease is deemed controlled prior to enrollment as determined by the investigator. Patients with uncontrolled CNS disease will be excluded
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to venetoclax or other agents used in this study
• Subjects with uncontrolled intercurrent illness including, but not limited to ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• Patients who are human immunodeficiency virus (HIV) positive and receiving combination antiretroviral therapy will be excluded; because of the potential for pharmacokinetic interactions with venetoclax
• Female patients who are pregnant or breast-feeding. Confirmation that the subject is not pregnant must be established by a negative serum beta-human chorionic gonadotropin (beta-hCG) pregnancy test result obtained during screening. Pregnancy testing is not required for post-menopausal or surgically sterilized women. Male or female patients, who are sexually active and of the child bearing age, must be willing to practice accepted birth control measures