This phase I/II trial studies the side effects and best dose of veliparib and topotecan hydrochloride and to see how well they work in treating patients with solid tumors, ovarian cancer that has come back or does not respond to treatment, or primary peritoneal cancer. Veliparib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as topotecan hydrochloride, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving veliparib with chemotherapy may kill more tumor cells.

PRIMARY OBJECTIVES:

I. To determine the maximum tolerated dose of the combination of veliparib (ABT-888) and weekly topotecan (topotecan hydrochloride) in adult patients with advanced solid tumors. (Phase I) II. To identify any anti-tumor activity of this treatment combination, as assessed by objective response in patients with advanced solid tumors. (Phase I) III. To assess the confirmed response rate for patients with epithelial ovarian cancer, fallopian tube cancer or primary peritoneal carcinoma treated with the combination of ABT-888 and weekly topotecan.

IV. To assess the progression free response (PFS) for patients with epithelial ovarian cancer, fallopian tube cancer or primary peritoneal carcinoma treated with the combination of ABT-888 and weekly topotecan. (Phase II)

SECONDARY OBJECTIVES:

I. To identify any pharmacokinetic interactions between ABT-888 and topotecan. (Phase I) II. To determine whether topotecan stimulates adenosine diphosphate (ADP)-ribose polymer formation in circulating peripheral blood mononuclear cells. (Phase I) III. To determine whether ABT-888 inhibits basal or topotecan-stimulated ADP-ribose polymer formation. (Phase I) IV. To assess differences in the toxicity and/or efficacy of this regimen based on BRCA 1/2 mutational status. (Phase II) V. To determine whether pretreatment tumor cell levels of topoisomerase I, poly ADP-ribose polymerase (PARP), BRCA1, BRCA2, XRCC1, tyrosyl-deoxyribonucleic acid (DNA) phosphodiesterase 1 (TDP1), P-glycoprotein or breast cancer resistance protein (BCRP) predict response to this regimen. (Phase II) VI. To identify, in an exploratory manner, any transcriptional profiles that may predict response to this regimen. (Phase II)

OUTLINE: This is a phase I, dose-escalation study of veliparib and topotecan hydrochloride followed by a phase II study. (PHASE I DOSE-ESCALATION PART IS COMPLETED)

Patients receive veliparib orally (PO) on days 1-3, 8-10, and 15-17 (veliparib is omitted on days 1-3 of course 2) and topotecan hydrochloride intravenously (IV) over 30 minutes on days 2, 9, and 16. Courses repeat every 28 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up at 3 months (Phase I) or every 3 or 6 months for 5 years (Phase II).

Inclusion Criteria:

• PHASE I: Adult patients with histologically confirmed solid tumor malignancy that is metastatic or unresectable and for which standard curative measures or other therapy definitely capable of extending life expectancy does not exist

• PHASE II: All patients enrolled in the Phase II portion of this trial must have a history of biopsy-proven ovarian, fallopian tube or primary peritoneal cancer

• Patients must have received < 3 lines of prior therapy and have relapsed less than a year from their last platinum regimen; regimens that are used twice (for example carboplatin and paclitaxel) can be counted as one; if a regimen is changed during the course of treatment due to side effect profile or allergy, the course of therapy is counted as one regimen; (for example, if docetaxel is substituted for paclitaxel due to a reaction during the initial course of adjuvant therapy, this is considered one regimen)

• Patients must have measurable disease with at least one lesion whose longest diameter can be accurately measured as >= 2.0 cm with conventional techniques or as >= 1.0 cm with spiral computed topography (CT); if spiral CT is used, it must be used for both pre- and post- treatment tumor assessments

• Absolute neutrophil count >= 1500/mcL

• Hemoglobin >= 9.0 g/dL

• Platelets >= 100,000/mcL

• Total bilirubin =< 1.5 x the upper limit of normal (ULN)

• Serum glutamic pyruvic transaminase (SGPT) (alanine aminotransferase [ALT]) or serum glutamic oxaloacetic transaminase (SGOT) (aspartate aminotransferase [AST]) =< 2.5 x ULN in the absence of hepatic metastasis; SGPT (ALT) =< 3 x ULN or SGOT (AST) =< 5 x ULN in the presences of hepatic metastasis

• Creatinine =< 1.5 x ULN

• International normalized ratio (INR) =< 1.4 unless receiving therapeutic doses of coumadin

• Partial thromboplastin time (PTT) =< 48 seconds (1.25 x ULN)

• Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1, or 2

• Ability to provide informed consent

• Willingness to return to enrolling institution for follow up

• Life expectancy >= 12 weeks

• Willingness to provide the biologic specimens is required by the protocol; this is part of the mandatory correlative research component; these specimens include:

  • PHASE I: peripheral blood for plasma pharmacokinetic analysis and peripheral blood mononuclear cell (PBMC) polymer assessment from 0-24 h after drug administration on days 1 and 2 of cycle 1 as well as day 2 of cycle 2; urine for assessment of ABT-888 renal clearance for 24 h after administration of drugs on days 1 and 2 of cycle 1 as well as day 2 of cycle 2; and a pretreatment peripheral blood sample for possible sequencing of the BRCA1, BRCA2 loci as well as possible pharmacogenomic analysis

• Negative urine or serum pregnancy test done =< 7 days prior to registration for females of child bearing potential only

• Able to swallow and absorb the medication

Exclusion Criteria:

• Known standard therapy for the patient's disease that is potentially curative or definitely capable of extending life expectancy

• Prior treatment with a PARP inhibitor or topotecan

• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements

• Any of the following prior therapies:

  • Chemotherapy =< 4 weeks prior to registration
  • Mitomycin C/nitrosoureas =< 6 weeks prior to registration
  • Immunotherapy =< 4 weeks prior to registration
  • Biologic therapy =< 4 weeks prior to registration
  • Radiation therapy =< 4 weeks prior to registration
  • Radiation to > 25% of bone marrow
  • Investigational therapy or any ancillary therapy considered investigational (utilized for a non-Food and Drug Administration [FDA]-approved indication and in the context of a research investigation) =< 4 weeks prior to registration; subjects with prostate cancer will be permitted to continue hormone therapy

• Failure to fully recover from acute, reversible effects of prior chemotherapy regardless of interval since last treatment

• New York Heart Association classification III or IV

• Known central nervous system (CNS) metastases or seizure disorder; patients with known brain metastases that have been successfully treated and stable for >= 6 months without requirement for corticosteroids and without seizure activity will be eligible

• Any of the following:

  • Pregnant women
  • Nursing women
  • Men or women of childbearing potential who are unwilling to employ adequate contraception

• Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens

• Immunocompromised patients (other than that related to the use of corticosteroids) including patients known to be human immunodeficiency virus (HIV) positive

• Receiving any other investigational agent which would be considered as a treatment for the primary neoplasm

• Other active malignancy, except non-melanotic skin cancer or carcinoma-in-situ of the cervix

  • Note: If there is a history of prior malignancy, they must not be receiving other specific treatment (other than hormonal therapy) for their cancer

• History of myocardial infarction =< 6 months, or congestive heart failure requiring use of ongoing maintenance therapy for life-threatening ventricular arrhythmias

• More than 2 prior chemotherapy regimens for the current malignancy; full dose chemotherapy used in conjunction with concurrent radiation therapy will be included as prior therapy

  • Note: Prior hormonal therapy (e.g. leuprolide, aromatase inhibitors, tamoxifen) and immunotherapy will be allowed and not included as a prior chemotherapy; if the chemotherapy regimen is altered during the course due to issues with tolerability or safety, the regimen will be counted as one; using the same regimen at recurrence is counted as one regimen; the addition of bevacizumab to a prior regimen is considered one regimen