This phase I trial studies the side effects and the best dose of veliparib when given together with pegylated liposomal doxorubicin hydrochloride, carboplatin, and bevacizumab in treating patients with ovarian cancer, primary peritoneal cancer, or fallopian tube cancer that has returned after previous treatment. Veliparib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as pegylated liposomal doxorubicin hydrochloride and carboplatin, may stop the growth of tumor cells by, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Monoclonal antibodies, such as bevacizumab, can block tumor growth by blocking the ability of tumor cells to grow and spread. Bevacizumab may also stop the growth of tumors by blocking the growth of new blood vessels necessary for tumor growth. Giving veliparib together with pegylated liposomal doxorubicin hydrochloride, carboplatin, and bevacizumab may kill more tumor cells.

PRIMARY OBJECTIVES:

I. To determine the maximum-tolerated doses (MTD) and dose-limiting toxicities of two different regimens of ABT-888 (veliparib) when administered with carboplatin and PLD (Doxil, Lipodox) (pegylated liposomal doxorubicin hydrochloride) in recurrent, platinum-sensitive epithelial ovarian, fallopian tube, or primary peritoneal cancer.

II. To assess the toxicity of these regimens using the Cancer Therapy Evaluation Program (CTEP) National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) version 4.0.

III. To examine the tolerability of these treatment regimens in combination with bevacizumab at the MTD.

SECONDARY OBJECTIVES:

I. To estimate the objective response rate (complete and partial) in patients with measurable disease.

TERTIARY OBJECTIVES:

I. To examine the relationships between platinum-free interval, activity of ABT-888 (objective response rate) and measures of breast cancer susceptibility gene 1/2 (BRCA1/2) status including mutations, alterations, rearrangements, promoter methylation, and immunohistochemical expression).

OUTLINE: This is a dose-escalation study of veliparib. Patients are assigned to 1 of 2 treatment arms.

REGIMEN I: Patients receive veliparib orally (PO) twice daily (BID) on days 1-7, and pegylated liposomal doxorubicin hydrochloride intravenously (IV) over 1 hour and carboplatin IV over 30 minutes on day 1.

REGIMEN II: Patients receive veliparib PO BID on days 1-28, and pegylated liposomal doxorubicin hydrochloride and carboplatin as in Regimen I.

BEVACIZUMAB: Once the MTD for veliparib has been determined, patients also receive bevacizumab IV over 30-90 minutes on days 1 and 15.

In both arms, treatment repeats every 28 days for up to 10 courses in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up quarterly for 1 year.

Inclusion Criteria:

• Patients must have histologic diagnosis of epithelial ovarian, fallopian tube, or primary peritoneal carcinoma which is now recurrent; histologic documentation of the original primary tumor is required via the pathology report

• Patients with the following histologic epithelial cell types are eligible: high-grade serous adenocarcinoma, endometrioid adenocarcinoma, undifferentiated carcinoma, clear cell adenocarcinoma, mixed epithelial carcinoma or adenocarcinoma not otherwise specified (N.O.S.)

• Patients must have recurrence documented by elevated cancer antigen (CA)-125 (biochemical recurrence) or clinically evident measurable or non-measurable recurrent disease

  • Biochemical recurrence is defined as a CA-125 greater than or equal to two times the upper normal limit; patients whose CA-125 is less than 100 U/mL must undergo a second confirmatory value within a period of not more than 4 weeks; patients with a level greater than or equal to 100 U/mL may be entered without confirmatory measurement; the CA-125 assessment for eligibility must be done at least 4 weeks after paracentesis or other surgical procedures
  • Detectable (non-measurable) disease is defined as symptomatic ascites or pleural effusions, solid and/or cystic abnormalities on radiographic imaging that do not meet Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 definitions for target lesions and/or biopsy proven recurrence
  • Measurable disease will be defined by RECIST 1.1; measurable disease is defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded); each lesion must be >= 10 mm when measured by computed tomography (CT), magnetic resonance imaging (MRI) or caliper measurement by clinical exam; or >= 20 mm when measured by chest x-ray; lymph nodes must be >= 15 mm in short axis when measured by CT or MRI
  • Patients with measurable disease must have had at least one "target lesion" to be used to assess response on this protocol as defined by RECIST 1.1; tumors within a previously irradiated field will be designated as "non-target" lesions unless progression is documented or a biopsy is obtained to confirm persistence at least 90 days following completion of radiation therapy

• Absolute neutrophil count (ANC) greater than or equal to 1,500/mcl

  • This ANC cannot have been induced or supported by granulocyte colony-stimulating factors

• Platelets greater than or equal to 100,000/mcl

• Creatinine =< 1.5 times institutional upper limit of normal (ULN)

• Bilirubin < 1.2 times ULN

• Serum glutamic oxaloacetic transaminase (SGOT) (aspartate aminotransferase [AST]) and serum glutamate pyruvate transaminase (SGPT) (alanine aminotransferase [ALT]) =< 3.0 times ULN

• Alkaline phosphatase =< 2.5 times ULN

• Left ventricular ejection fraction (LVEF) greater than or equal to institutional lower limit of normal (LLN) as determined by gated cardiac radionucleotide scan (MUGA) or echocardiogram

• Neuropathy (sensory and motor) less than or equal to grade 1

• Patients must have a platinum-free interval following initial platinum-based chemotherapy of at least 6 months at first recurrence; front-line therapy may have included a biologic/targeted agent (e.g., bevacizumab)

  • NOTE: Front-line treatment may have included maintenance therapy; patients receiving maintenance therapy (biological therapy, hormonal, or taxane therapy) are ELIGIBLE provided their platinum-free interval is at least 6 months from initial chemotherapy AND a minimum of 4 weeks has elapsed since their last dose of biologic/targeted or taxane therapy or a minimum of 1 week has elapsed since their last dose of hormonal therapy

• Patients must have a Gynecologic Oncology Group (GOG) performance status of 0 or 1

• Patients of childbearing potential must have a negative pregnancy test prior to the study entry and be practicing an effective form of contraception; if applicable, patients must discontinue breastfeeding prior to study entry

• Patients who have met the pre-entry requirements

• Patients must have signed an Institutional Review Board (IRB)-approved informed consent and authorization permitting release of personal health information

• ADDITIONAL CRITERIA FOR PATIENTS BEING TREATED ON BEVACIZUMAB COHORT

• Prothrombin time (PT) such that international normalized ratio (INR) is =< 1.5 X ULN (or an in-range INR, usually between 2 and 3, if a patient is on a stable dose of therapeutic warfarin) and a partial thromboplastin time (PTT) < 1.5 X ULN

• Urine protein should be screened by urine analysis; if protein is 2+ or higher, 24-hour urine protein should be obtained and the level should be < 1000 mg for patient enrollment

• Patients treating with enoxaparin are eligible for inclusion in the study

• Fertile women must agree to use adequate contraceptive measures during study therapy and for at least 6 months after the completion of bevacizumab therapy; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, the patient should inform the treating physician immediately

Exclusion Criteria:

• Patients who have received more than one previous regimen of chemotherapy (maintenance is not considered a second regimen)

• Patients who have received prior ABT-888 or any other poly-adenosine diphosphate (ADP)--ribose polymerase (PARP) inhibitor

• Patients who have received prior PLD

• History of allergic reactions attributed to compounds of similar chemical or biologic composition to ABT-888 or other agents used in this study

• Patients with other invasive malignancies, with the exception of non-melanoma skin cancer and other specific malignancies are excluded if there is any evidence of other malignancy being present within the last three years; patients are also excluded if their previous cancer treatment contraindicates this protocol therapy

• Patients who have received prior radiotherapy to any portion of the abdominal cavity or pelvis are excluded; prior radiation for localized cancer of the breast, head and neck, or skin is permitted, provided that it was completed more than three years prior to registration, and the patient remains free of recurrent or metastatic disease

• Patients who have received prior chemotherapy for any abdominal or pelvic tumor (other than ovarian, fallopian tube and primary peritoneal) are excluded; patients may have received prior adjuvant chemotherapy for localized breast cancer, provided that it was completed more than three years prior to registration, and that the patient remains free of recurrent or metastatic disease

• Patients with synchronous primary endometrial cancer or a history of endometrial cancer, unless all of the following conditions are met:

  • Stage not greater than IB
  • No more than superficial myometrial invasion
  • No vascular or lymphatic invasion
  • No poorly differentiated subtypes, including papillary serous, clear cell, or other International Federation of Gynecology and Obstetrics (FIGO) grade 3 lesions

• Patients with known chronic or active hepatitis or ongoing or active infection that requires parenteral antibiotics

• Patients with concurrent severe medical problems unrelated to the malignancy that would significantly limit full compliance with the study or expose the patient to extreme risk or decreased life expectancy

• Patients of childbearing potential, not practicing adequate contraception, patients who are pregnant or patients who are breastfeeding are not eligible for this trial

• Patients with seizures or a history of seizures are ineligible

• Patients with history or evidence upon physical examination of central nervous system (CNS) disease, including primary brain tumor, any CNS metastases, or history of cerebrovascular accident (CVA, stroke), transient ischemic attack (TIA) or subarachnoid hemorrhage within six months of the first date of treatment on this study; patients with CNS metastases must be stable for > 3 months after treatment and off steroid treatment prior to study enrollment

• Patients who cannot swallow pills

• ADDITIONAL CRITERIA FOR PATIENTS BEING TREATED ON BEVACIZUMAB COHORT:

• History of allergic reactions attributed to compounds of similar chemical or biologic composition to bevacizumab

• Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with bevacizumab

• Human immunodeficiency virus (HIV)-positive patients on combination antiretroviral therapy are ineligible

• Urine protein should be screened by urine analysis; if protein is 2 + or higher, 24-hour urine protein should be obtained and the level should be < 1000 mg for patient enrollment

• Serious non-healing wound, ulcer, or bone fracture

• History of abdominal fistula, gastrointestinal perforation or intra-abdominal abscess within 6 months to day 1

• Invasive procedures defined as follows:

  • Major surgical procedure, open biopsy or significant traumatic injury within 28 days prior to day 1 therapy
  • Anticipation of need for major surgical procedures during the course of the study
  • Core biopsy within 7 days prior to day 1 (D1) therapy

• Significant vascular disease (e.g., aortic aneurysm, requiring surgical repair or recent peripheral arterial thrombosis) within 6 months prior to day 1

• Known CNS disease except for treated brain metastases; treated brain metastases are defined as having no ongoing requirement for steroids and no evidence of progression or hemorrhage after treatment for at least 3 months, as ascertained by clinical examination and brain imaging (magnetic resonance imaging [MRI] or computed tomography [CT]); (stable dose of anticonvulsants are allowed); treatment for brain metastases may include whole brain radiotherapy (WBRT), radiosurgery (RS; Gamma Knife, LINAC, or equivalent) or a combination as deemed appropriate by the treating physician; patients with CNS metastases treated by neurosurgical resection or brain biopsy performed within 3 months prior to Day 1 will be excluded

• Patients with clinically significant cardiovascular disease are excluded

  • Inadequately controlled hypertension (HTN) (systolic blood pressure [SBP] > 160 mm Hg and/or diastolic blood pressure [DBP] > 90 mm Hg despite antihypertensive medication)
  • History of cerebrovascular accident (CVA) within 6 months
  • History of myocardial infraction or unstable angina within 6 months
  • New York Heart Association grade II or greater congestive heart failure
  • Serious and inadequately controlled cardiac arrhythmia
  • Significant vascular disease (e.g. aortic aneurysm, history of aortic dissection)
  • Clinically significant peripheral vascular disease

• Evidence of bleeding diathesis or coagulopathy

• Patients with known hypersensitivity to Chinese hamster ovary cell products or other recombinant human antibodies