RATIONALE: Androgens can cause the growth of prostate cancer cells. Antihormone therapy, such as bicalutamide, may lessen the amount of androgens made by the body. Enzastaurin may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. It is not yet known whether giving bicalutamide together with enzastaurin is more effective than bicalutamide alone in treating prostate cancer.

PURPOSE: This randomized phase II trial is studying bicalutamide to see how well it works compared with giving bicalutamide together with enzastaurin in treating patients with prostate cancer.

OBJECTIVES:

Primary

• To compare the two regimens on the proportion of patients with undetectable prostate-specific antigen (PSA) level (< 0.2 ng/mL) at 44 weeks.

Secondary

• To assess the proportion of patients with PSA decline > 85% at 44 weeks on the combination therapy arm compared to that of bicalutamide monotherapy arm.
• To assess the distribution of best PSA response in each study arm.
• To assess the time to PSA progression and the time to PSA nadir in each arm of the study.
• To assess the duration of PSA response in each arm of the study.
• To characterize the PSA slope before, during, and after treatment.
• To evaluate the safety and tolerability of enzastaurin hydrochloride in this patient population.
• To determine whether Gleason score or prior hormonal therapy has any effect on PSA response to treatment.

OUTLINE: This is a multicenter study. Patients are stratified according to Gleason score (≤ 6 vs 7 vs 8-10) and prior hormonal therapy (yes vs no). Patients are randomized to 1 of 2 treatment arms.

• Arm A:

  • Weeks 1-12: Patients are observed without treatment. Patients with a prostate-specific antigen (PSA) rise of > 50% above baseline or nadir (whichever is lowest) and a rise of at least 5 ng/mL, confirmed by a repeat PSA at least 2 weeks later, may be started on bicalutamide before the end of week 12 at the discretion of the treating physician.
  • Weeks 13-44: Patients with a rise PSA ≥ 50% above baseline or nadir, and a PSA rise of at least 5 ng/mL confirmed by a repeat PSA at least 2 weeks later, are removed from study. Patients receive oral bicalutamide once daily. Patients achieving a PSA decline ≥ 50% in the absence of toxicity may continue to receive bicalutamide up to 72 weeks.

• Arm B:

  • Weeks 1-12: Patients receive oral enzastaurin hydrochloride twice daily. Patients with a PSA rise of > 50% above baseline or nadir, and a rise of at least 5 ng/mL, confirmed by a repeat PSA at least 2 weeks later, may be started on bicalutamide before the end of week 12 at the discretion of the treating physician.
  • Weeks 13-44: Patients with a PSA rise of ≥ 50% above baseline or nadir, and a rise of at least 5 ng/mL, confirmed by a repeat PSA at least 2 weeks later, are removed from study. Patients receive oral enzastaurin twice daily and oral bicalutamide once daily. Patients achieving a PSA decline ≥ 50% in the absence of toxicity may continue on this combination therapy up to 72 weeks.

After completion of study treatment, patients are followed every 3 months for 5 years, and then every 6 months for up to 10 years.

DISEASE CHARACTERISTICS:

• Histologically confirmed prostate cancer

  • Hormone-sensitive disease, as evidenced by a serum total testosterone level > 150 ng/dL
  • No evidence of metastatic disease on physical exam, CT abdomen/pelvis (or MRI), chest x-ray or CT scan and bone scan within 6 weeks prior to randomization

• Underwent prior definitive surgery or radiotherapy

• Must have evidence of biochemical failure after primary therapy and subsequent progression as determined by 1 of the following:

  • Prostate-specific antigen (PSA) ≥ 0.4 ng/mL (in case of radical prostatectomy)
  • PSA rise ≥ 2 ng/mL above the nadir PSA (in case of radiotherapy)

• Baseline PSA must be at least 2 ng/mL and no greater than 50 ng/mL

• PSA doubling time (PSADT) < 12 months

PATIENT CHARACTERISTICS:

• ECOG performance status 0 - 1

• Granulocytes ≥ 1,500/mm^3

• Platelet count ≥ 75,000/mm^3

• Serum creatinine normal or creatinine clearance ≥ 60 mL/min

• Serum total bilirubin ≤ 1.5 times upper limit of normal (ULN)

• Alkaline phosphatase ≤ 2.5 times ULN

• SGOT and SGPT < 2.5 times ULN

• PT/INR normal

• Fertile patients must use effective barrier contraception during and for at least 3 months after completion of study treatment

• No gastrointestinal (GI) tract disease resulting in: inability to take oral medication, malabsorption syndrome, a requirement for IV alimentation, prior surgical procedures affecting absorption, or uncontrolled inflammatory GI disease (e.g., Crohn's, ulcerative colitis)

• No history of allergic reactions attributed to compounds of similar chemical or biologic composition to enzastaurin hydrochloride or bicalutamide

• No uncontrolled intercurrent illness including, but not limited to, any of the following:

  • Ongoing or active infection
  • Symptomatic congestive heart failure
  • Unstable angina pectoris
  • Cardiac arrhythmia
  • Psychiatric illness/social situations that would limit compliance with study requirements

• A history of other malignancy is permitted if the patient is predicted to be disease-free for 2 years

PRIOR CONCURRENT THERAPY:

• See Disease Characteristics

• More than 4 weeks since prior salvage therapy with intent to cure (i.e., surgery, radiotherapy, or other local ablative procedures)

• More than 4 weeks since prior prophylactic radiotherapy to prevent gynecomastia

• More than 1 year since prior therapy modulating testosterone levels (such as luteinizing-hormone releasing-hormone agonists/antagonists and antiandrogens) unless in the neoadjuvant or adjuvant setting

• No 5 alpha reductase inhibitors, ketoconazole, megestrol acetate, systemic steroids, or herbal supplements during PSA value collection

• At least 14 days since prior enzyme-inducing anti-epileptic drugs (EIAEDs)

  • Patients who must begin EIAED therapy while on study are allowed to remain

• No concurrent combination antiretroviral therapy for HIV-positive patients

• No concurrent anticoagulant therapy

  • Low dosage acetyl salicylic acid ≤ 325 mg/day allowed

• No other concurrent investigational agents or anticancer therapy (i.e., chemotherapy, immunotherapy, radiotherapy, surgery for cancer, or experimental medications)

• No history of allergic reactions attributed to compounds of similar chemical or biologic composition to enzastaurin or bicalutamide

• Prior neoadjuvant and/or adjuvant therapy ≤ 4 weeks prior to randomization (i.e., hormones, chemotherapy, vaccines, or experimental agents) allowed if PSA rise and PSADT were documented after testosterone level was > 150 ng/dL