This phase I trial studies the side effects and the best dose of veliparib when given together with pegylated liposomal doxorubicin hydrochloride in treating patients with ovarian cancer, fallopian tube cancer, or primary peritoneal cancer that has come back after a period of improvement, or breast cancer that has spread to other parts of the body. Veliparib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as liposomal doxorubicin hydrochloride, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving veliparib together with liposomal doxorubicin hydrochloride may kill more tumor cells.

PRIMARY OBJECTIVES:

I. To determine the recommended Phase II dose of ABT-888 (veliparib) given in combination with pegylated liposomal doxorubicin (pegylated liposomal doxorubicin hydrochloride) (PLD - 40 mg/m2 every 4 weeks) in patients with ovarian or breast cancer.

SECONDARY OBJECTIVES:

I. To determine the toxicity profile of the ABT-888 plus PLD combination. II. To determine the effects of ABT-888 on the pharmacokinetics of PLD. III. To determine the efficacy of ABT-888 plus PLD in ovarian and breast cancer.

OUTLINE: This is a dose-escalation study of veliparib.

Patients receive veliparib orally (PO) twice daily (BID) on days 1-14 and pegylated liposomal doxorubicin hydrochloride intravenously (IV) over 60 minutes on day 1. Courses repeat every 28 days in the absence of disease progression or unacceptable toxicity.

After completion of study therapy, patients are followed up every 3 months for 1 year and then every 6 months for 2 years.

Inclusion Criteria:

• Histologically confirmed diagnosis of recurrent or residual epithelial ovarian cancer, primary peritoneal carcinoma or fallopian tube carcinoma, OR histologically confirmed metastatic breast cancer, that is estrogen receptor (ER)-negative, progesterone receptor (PR)-negative, and human epidermal growth factor receptor 2 (HER2)/neu negative (as determined by local pathology laboratory)

• Prior chemotherapy:

  • Ovarian cancer: patients with no prior PLD exposure are eligible after failure of platinum-containing chemotherapy; no more than 2 prior platinum containing regimens is permitted; dose escalating cohorts only: patients already on PLD are also eligible if they are receiving PLD beyond 3 cycles without prohibitive (i.e. no grade 3 or 4) skin or mucosal toxicities, and showing no progressive disease compared to a computed tomography (CT) scan obtained 2 or more months earlier; these patients are eligible in spite of any progression from baseline determined prematurely (i.e., applicable to those patients who are deemed in their best interest to continue to receive PLD after a CT obtained at 2 or 3 months has shown progression from baseline)

    • Breast cancer: patients may have received 0-2 prior chemotherapy regimens for metastatic disease; breast cancer patients may not have received prior PLD, and will not be eligible for the expanded cohort A
    • Interval between prior chemotherapy and registration for breast and ovarian cancer; there should be at least a 3 week interval between the last chemotherapy regimen and registration, and the patient should have recovered from acute toxicity related to prior therapy (6 weeks if the last regiment included BCNU or mitomycin C)
    • Dose escalating cohorts only: patients will be categorized in the following strata based upon prior PLD exposure: Stratum A -patients with ovarian cancer who have had prior PLD exposure and received at least 3 cycles of PLD without prohibitive (i.e. no grade 3 or 4 skin toxicity) and have not had progressive disease; Stratum B: patients with ovarian or breast cancer who have had no prior PLD exposure

• Eastern Cooperative Oncology Group (ECOG) performance score 0-2

• All potential subjects should be evaluated for whether breast cancer (BRCA)1-2 testing is medically appropriate; individuals who have a 10% or higher risk of having a BRCA1-2 mutation (Myriad tables at www.myriad.com) are encouraged (but not required) to have mutation testing and results known; information regarding mutation status (positive [including specific mutation], negative, or unknown) and projected risk of having a mutation (as determined by Myriad tables) will be collected at the time of diagnosis

• Non-measurable and/or measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) criteria, or abnormal cancer antigen (CA)-125 to levels (in patients with ovarian cancer) at least 1.5 x normal documented by two independent measurements at least 4 weeks apart

• Ability to give voluntary informed consent and to comply with treatment and required tests

• Ability to tolerate oral medications

• Female subjects age >= 18 years (males with breast cancer are eligible)

• Absolute neutrophil count >= 1500/mL

• Platelets >= 100,000/mL

• Creatinine =< 1.5 mg/dL

• Total bilirubin =< 1.5 x institutional upper limit of normal

• Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =< 3 x institutional upper limit of normal (=< 5 x institutional upper limit of normal if evidence of liver metastasis)

• Left ventricular ejection fraction at or above institutional lower limit of normal (obtained within 8 weeks of registration by multigated acquisition [MUGA] scan or echocardiogram; the same test performed at baseline should be repeated after every 3 cycles of therapy)

• Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately

Exclusion Criteria:

• Known central nervous system (CNS) metastases with active symptoms, or requiring anticonvulsive medications, or steroids
• Prior chemotherapy (except PLD in Dose Escalation Cohorts only) or any investigational agent within 3 weeks prior to registration
• Prior radiation therapy to whole pelvis or greater amount of marrow-forming bone
• Prior or current non-gynecologic or non-breast malignancy within 5 years except non-melanoma skin cancer
• Patients with active severe infection; known infection with human immunodeficiency virus (HIV), hepatitis B virus, hepatitis C virus, or severe concurrent illness
• Patients with any non-malignant intercurrent illness (e.g. cardiovascular, pulmonary, central nervous system disease) which is either poorly controlled with currently available treatment, or which is of such severity that the investigators deem it unwise to enter the patient on protocol
• Patients with history of seizure disorder requiring antiepileptics who have had a seizure episode within the last 6 months
• Pregnant (positive pregnancy test) or lactating; unwillingness to use effective means of contraception in subjects with child-bearing potential
• Evidence of complete or partial bowel obstruction or other unable to take oral medications