This phase I trial studies the side effects and the best dose of veliparib and dinaciclib in treating patients with solid tumors that have spread to other places in the body and usually cannot be cured or controlled with treatment. Veliparib and dinaciclib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth.

PRIMARY OBJECTIVES:

I. To evaluate the safety and tolerability of veliparib (ABT-888) and dinaciclib (SCH727965) in patients with advanced solid tumors.

II. To determine the recommended phase 2 dose (RP2D) for ABT-888 in combination with SCH727965, determined by evaluating the feasibility, safety, dose-limiting toxicities, and the maximally tolerated dose(s).

SECONDARY OBJECTIVES:

I. To confirm the safety of the combination of ABT-888 and SCH727965 in patients with known BRCA1 or BRCA2 germline mutation.

II. To characterize the pharmacokinetic parameters of ABT-888 both alone and in combination with SCH727965.

III. To assess the pharmacodynamic effects of ABT-888 in combination with SCH727965, both in surrogate tissues and in tumor.

IV. To assess preliminary antitumor activity of the ABT-888/SCH727965 combinations in subjects with solid tumors.

OUTLINE: This is a dose-escalation study of veliparib and dinaciclib followed by expanded doublet cohort studies of non-breast BRCA-proficient patients and BRCA-proficient triple negative breast cancer (TNBC) patients, a safety doublet cohort for BRCA-deficient patients, and cohorts of BRCA-associated TNBC (PARP inhibitor-naive and PARP inhibitor-resistant).

PART 1A: Patients receive veliparib orally (PO) twice daily (BID) on days 1-28 and dinaciclib intravenously (IV) over 2 hours on days 8 and 22. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity.

PART 1B: Patients receive veliparib and dinaciclib as patients in Part 1A. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity.

PART 1C: Patients receive veliparib PO BID on days 1-7 of cycle 0. Patients then receive veliparib PO BID on days 1-21 and dinaciclib IV over 2 hours on days 1, 4, 8, and 11 or days 1 and 8. Cycles repeat every 21 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up every 3 months.

Inclusion Criteria:

• Participants must have histologically confirmed diagnosis of a solid tumor for which no curative therapy exists
• Participants must have measurable or evaluable disease
• Eastern Cooperative Oncology Group (ECOG) performance status =< 2
• Prior chemotherapy is allowed; patients must not have received chemotherapy for 3 weeks prior to the initiation of study treatment and must have full recovery from any acute effects of any prior chemotherapy; patients must not have had nitrosoureas or mitomycin C for 6 weeks prior to the initiation of study treatment
• Prior exposure to approved receptor tyrosine kinase inhibitors is permitted; at least 5 half-lives must have elapsed since the completion of the kinase inhibitor and the initiation of study treatment
• Prior radiation therapy is allowed; patients must not have received any radiation within 3 weeks prior to the initiation of study treatment; patients may not have areas of irradiated marrow exceeding 40% of bone marrow volume
• Prior experimental (non-Food and Drug Administration [FDA] approved) therapies and immunotherapies are allowed; patients must not have received these therapies for 3 weeks prior to the initiation of study treatment and must have full recovery from any acute effects of these therapies
• Prior exposure to ABT888 or other PARP inhibitors is permitted in all cohorts; prior exposure to cyclin-dependent kinase inhibitors other than SCH727965 is permitted
• Absolute neutrophil count >= 1,500/mm^3
• Hemoglobin (Hgb) > 10.0 g/dL with no blood transfusion in the past 28 days
• Platelets >= 100,000/mm^3
• Total bilirubin < 1.5 mg/dl
• Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT]) =< 2.5 times the institutional upper limit of normal; for subjects with known liver metastases, AST and ALT =< 5 times institutional upper limit of normal
• Creatinine =< 1.5 times institutional upper limit of normal or creatinine clearance >= 60 mL/min/1.73 m^2
• Prothrombin time (PT)/international normalized ratio (INR) and partial thromboplastin time (PTT) =< 1.5 times institutional upper limit of normal
• The effects of ABT-888 and SCH727965 on the developing human fetus are unknown; for this reason, women of childbearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately; additionally, if a man suspects that he has fathered a child while taking study agents, he should also inform his treating physician immediately
• Eligible patients in the dose escalation phases of the trial must agree to biopsies of normal skin, unless they undergo optional tumor biopsies; the mandatory biopsy requirement can be waived at the discretion of principal investigator in the event of any medical contraindication (e.g. lidocaine allergy); patients enrolled to the expanded cohorts must agree to tumor sampling; patients on anticoagulation must be able to hold warfarin or low molecular weight heparin for a sufficient amount of time to make skin and tumor biopsies safe to perform; PT/INR and PTT should be =< 1.5 times the institutional upper limit of normal prior to performance of skin or tumor biopsies, with values re-checked after the eligibility screen as medically indicated
• Patients must be able to swallow pills
• Patients enrolling in the BRCA-deficient cohorts must have a documented BRCA1 or BRCA2 germline mutation; alternatively, patients with tumors harboring somatic BRCA mutations may also enroll after discussion with the principal investigator
• All patients must agree to provide an archival tissue block or paraffin sample from archival tissue block (approximately 10 sections) for use in pharmacodynamic correlative studies; however, patients are not considered ineligible if archival tumor is not available
• Ability to understand and the willingness to sign a written informed consent document; subjects must be willing to adhere to dose and visit schedules

Exclusion Criteria:

• Patients must not receive any other anti-cancer therapy (cytotoxic, biologic, radiation, or hormonal other than for replacement) while on this study except for medications that are prescribed for supportive care but may potentially have an anti-cancer effect (i.e. megestrol acetate, bisphosphonates); in addition, men receiving treatment for prostate cancer will be maintained at castrate levels of testosterone by continuation of luteinizing-releasing hormone agonists; palliative radiation therapy (XRT) can be administered on study after documented discussion with the principal investigator; for patients in expansion cohorts, this must not involve target lesions
• Patients with known active brain metastases are excluded; patients with a history of central nervous system (CNS) metastases that have been treated must be stable with no symptoms for > 3 months after completion of that treatment and off steroid treatment, with image documentation required prior to study enrollment
• Any patient requiring chronic maintenance of white blood cell counts or granulocyte counts through the use of growth factor support (e.g. Neulasta, Neupogen)
• Patients who have previously received SCH727965
• Patients with other medical conditions judged by the investigator to be clinically relevant in the setting of this study, which may include active infectious processes, intractable emesis, or chronic diarrheal disease
• Uncontrolled intercurrent illness including, but not limited to ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, or psychiatric illness/social situations that would limit compliance with study requirements
• Pregnant women are excluded from this study because ABT-888 and SCH727965 are anti-proliferative agents with the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk of adverse events in nursing infants secondary to treatment of the mother with ABT-888 and SCH727965, breastfeeding should be discontinued if the mother is treated with ABT-888 and SCH727965; these potential risks may also apply to other agents used in this study
• Patients with prior seizure history who have experienced a seizure within the three months prior to enrollment are excluded
• Subjects with a known allergy to lidocaine
• Subjects on a potent CYP3A4 inhibitor or CPY3A4 inducer who cannot be changed to another medication are excluded
• Subjects with treatment-related acute myeloid leukemia (AML) (t-AML)/myelodysplastic syndrome (MDS) or with features suggestive of AML/MDS
• Prior allogeneic bone marrow transplant or double umbilical cord blood transplantation