This phase I trial is studying the side effects and best dose of veliparib when given together with topotecan hydrochloride with or without carboplatin in treating patients with relapsed or refractory acute leukemia, high-risk myelodysplasia, or aggressive myeloproliferative disorders. Veliparib may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as topotecan hydrochloride and carboplatin, work in different ways to stop the growth of cancer cells, either by killing the cells or by stopping them from dividing. Giving veliparib together with topotecan hydrochloride and carboplatin may kill more cancer cells.

PRIMARY OBJECTIVES:

I. To determine the feasibility, tolerability, and toxicities of ABT-888 (veliparib) when administered alone and in combination with topotecan hydrochloride with or without carboplatin in patients with relapsed or refractory acute leukemia, high-risk myelodysplasia, or aggressive myeloproliferative disorders.

II. To determine the maximum tolerated dose of ABT-888 when administered with topotecan hydrochloride and carboplatin in these patients.

III. To determine if ABT-888 when administered with topotecan hydrochloride and carboplatin can induce clinical responses in these patients.

SECONDARY OBJECTIVES:

I. To determine the pharmacokinetics of ABT-888 when administered alone and in combination with topotecan hydrochloride with or without carboplatin in these patients.

II. To obtain pharmacodynamic data regarding the ability of ABT-888 to inhibit poly (ADP-ribose) levels in leukemic blasts.

III. To obtain descriptive data regarding the mutational status and/or methylation status of key genes in selected DNA repair pathways (Fanconi complementation groups A-F, Blooms, and ataxia-telangiectasia) in leukemic blasts.

OUTLINE: This is a multicenter, dose-escalation study of veliparib.

Patients receive veliparib orally twice daily on days 1-8, 1-14, or 1-21 and topotecan hydrochloride with or without carboplatin IV continuously over 120 hours on days 3-7. Treatment repeats every 28-63 days for up to 6 courses in the absence of disease progression or unacceptable toxicity.

Patients undergo blood sample collection periodically for pharmacokinetic studies.

After completion of study therapy, patients are followed for 30 days.

Inclusion Criteria:

• Pathologically confirmed diagnosis of 1 of aggressive MPD or AML out of MPD

• Aggressive phase high-risk myeloproliferative disorders (i.e., polycythemia vera, essential thrombocythemia, or Ph-negative chronic myelogenous leukemia) meeting ≥ 1 of the following criteria:

  • Marrow blasts > 5%
  • Peripheral blood blasts plus progranulocytes > 10%
  • New onset or increasing myelofibrosis OR;

• New onset or > 25% increase in hepatomegaly or splenomegaly

• New onset constitutional symptoms (i.e., fever, weight loss, splenic pain, or bone pain)

• Patients who failed primary induction therapy or relapsed after achieving complete remission are eligible

• No active CNS leukemia; patients with a history of CNS disease must be stable for > 3 months after treatment and off steroid treatment prior to study enrollment

• Chronic myelomonocytic leukemia meeting either of the following criteria:

  • 5-19% bone marrow blasts (aggressive)
  • At least 20% marrow blasts (transformation)

• ECOG performance status 0-2

• No hyperleukocytosis with >= 50,000 blasts/uL

• AST, ALT, and alkaline phosphatase =< 5 times upper limit of normal

• Bilirubin =< 2.0 mg/dL

• Creatinine normal OR creatinine clearance >= 60 mL/min

• LVEF >= 45% by MUGA or ECHO

• Not pregnant or nursing

• Negative pregnancy test

• Fertile patients must use effective contraception during and for 30 days after completion of study therapy

• No active disseminated intravascular coagulation

• No active uncontrolled infection

• Patients with infection that is under active treatment and controlled with antibiotics are eligible

• No other life-threatening illness

• No mental deficits and/or psychiatric history that would preclude giving informed consent or following protocol

• No prior or current seizure disorder or a history of seizure

• No more than 3 prior cytotoxic regimens

• At least 3 weeks since prior cytotoxic chemotherapy

• At least 2 weeks since prior radiotherapy

• At least 4 weeks since prior autologous or allogeneic stem cell transplantation

• No active graft-versus-host disease

• At least 1 week since prior biologic therapies, including hematopoietic growth factors

• At least 24 hours since prior hydroxyurea, steroids, imatinib mesylate, arsenic trioxide, interferon, or other noncytotoxic agents for blast count control

• No prior ABT-888

• No other concurrent chemotherapy, radiotherapy, or immunotherapy

• No concurrent antiretroviral therapy for HIV-positive patients

• No other concurrent investigational or commercial agents or therapies for this cancer