RATIONALE: Enzastaurin may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth and by blocking blood flow to the tumor.

PURPOSE: This phase II trial is studying how well enzastaurin works in treating patients with persistent or recurrent ovarian epithelial cancer or primary peritoneal cancer.

OBJECTIVES:

Primary

• Assess the efficacy of enzastaurin hydrochloride, in terms of 6-month progression-free survival or objective tumor response, in patients with recurrent or persistent ovarian epithelial or primary peritoneal cancer.
• Determine the nature and degree of toxicity of this regimen in these patients.

Secondary

• Determine the duration of progression-free and overall survival of patients treated with this regimen.
• Determine the effects of prognostic variables, including platinum sensitivity, initial performance status, and age, in patients treated with this regimen.

OUTLINE: This is a multicenter study.

Patients receive oral enzastaurin hydrochloride 3 times on day 1 and then once daily on days 2-28 of course 1. For all subsequent courses, patients receive enzastaurin hydrochloride once daily on days 1-28. Courses repeat every 28 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed every 3 months for 2 years and then every 6 months for 3 years.

PROJECTED ACCRUAL: A total of 68 patients will be accrued for this study.

DISEASE CHARACTERISTICS:

• Histologically confirmed ovarian epithelial or primary peritoneal carcinoma

• Recurrent or persistent disease

• Measurable disease, defined as ≥ 1 unidimensionally measurable lesion ≥ 20 mm by conventional techniques OR ≥ 10 mm by spiral CT scan

  • Must have ≥ 1 target lesion to assess response

    • Tumors within a previously irradiated field are designated as "nontarget" lesions unless progression is documented or a biopsy is obtained to confirm persistence ≥ 90 days after completion of radiotherapy

• Must have received 1 prior platinum-based chemotherapy regimen containing carboplatin, cisplatin, or another organoplatinum compound for management of primary disease

  • Initial treatment may have included high-dose therapy, consolidation therapy, or extended therapy administered after surgical or nonsurgical assessment

  • Must meet any 1 of the following criteria for platinum-based therapy:

    • Disease progression during therapy
    • Treatment-free interval after completion of treatment < 12 months
    • Disease persistence after completion of therapy

• Ineligible for a higher priority GOG clinical trial

PATIENT CHARACTERISTICS:

• GOG performance status 0-1 (for patients who received 2 prior treatment regimens) OR 0-2 (for patients who received 1 prior treatment regimen)
• Absolute neutrophil count ≥ 1,500/mm³
• Platelet count ≥ 100,000/mm³
• Hemoglobin ≥ 9 g/dL (transfusions allowed)
• Creatinine < 1.5 times upper limit of normal (ULN)
• Bilirubin ≤ 2 times ULN
• Alkaline phosphatase ≤ 3 times ULN (5 times ULN if liver metastases are present)
• AST and ALT ≤ 3 times ULN (5 times ULN if liver metastases are present)
• Not pregnant or nursing
• Negative pregnancy test
• Fertile patients must use effective contraception during and for 3 months after completion of study treatment
• Able to swallow tablets
• No sensory or motor neuropathy > grade 1
• No active infection requiring antibiotics
• No other invasive malignancies or evidence of cancer within the past 5 years except nonmelanoma skin cancer
• No serious systemic disorders that would preclude study compliance, including an abnormal ECG indicative of cardiac disease

PRIOR CONCURRENT THERAPY:

• See Disease Characteristics
• Recovered from prior surgery, radiotherapy, or chemotherapy
• At least 1 week since prior anticancer hormonal therapy
• No more than 1 additional cytotoxic regimen for management of recurrent or persistent disease
• At least 4 weeks since other prior anticancer therapy, including immunotherapy
• At least 30 days since prior investigational drugs
• No prior enzastaurin hydrochloride
• No prior radiotherapy to > 25% of marrow-bearing areas
• No prior noncytotoxic therapy, including bevacizumab, for recurrent or persistent disease
• No prior treatment that would preclude treatment on this protocol
• No concurrent chemotherapy, immunotherapy, or other experimental medications
• No concurrent enzyme-inducing antiepileptic drugs, including carbamazepine, phenobarbital, or phenytoin
• No other concurrent systemic anticancer therapy
• No concurrent radiotherapy, including palliative radiotherapy
• No concurrent agents that stimulate thrombopoiesis
• No concurrent amifostine or other protective reagents
• Concurrent hormone replacement therapy allowed
• Concurrent bisphosphonates allowed provided bony metastases are present