RATIONALE: Giving colony-stimulating factors, such as G-CSF help stem cells move from the patient's bone marrow to the blood so they can be collected and stored. Combination chemotherapy is then given to prepare the bone marrow for the stem cell transplant. The stem cells are returned to the patient to replace the blood-forming cells that were destroyed by the chemotherapy.

PURPOSE: This randomized trial is studying the side effects and best dose of topotecan when given together with high-dose cyclophosphamide, and carboplatin followed by an autologous peripheral blood stem cell transplant in treating patients with recurrent ovarian cancer or primary peritoneal cancer.

OBJECTIVES:

• To determine the maximum tolerated dose of topotecan hydrochloride combined with high-dose cyclophosphamide and carboplatin in the setting of autologous peripheral blood stem cell transplantation for relapsed, recurrent, or persistent ovarian epithelial or primary peritoneal cavity cancer.
• To assess the toxicity of this regimen.

OUTLINE: This is a dose escalation study of topotecan.

• Autologous hematopoietic stem cell collection: Patients receive filgrastim subcutaneously (SC) daily for 5 days. Patients undergo leukapheresis per standard practice until a minimum of 2 x10^6 CD34+ cells/kg are collected and cryopreserved.
• High-dose chemotherapy: Patients receive topotecan hydrochloride IV, cyclophosphamide IV, and carboplatin IV over 8 hours on days -6 to -3.
• Autologous peripheral stem cell reinfusion: Patients undergo autologous peripheral blood stem cell transplantation on day 0. Patients also receive sargramostim SC daily beginning on day 5 and continuing until blood counts recover.

After completion of study therapy, patients are followed monthly for 3 months, every 3 months for 2 years, and then every 6 months for 5 years.

DISEASE CHARACTERISTICS:

• Histologically confirmed ovarian epithelial or primary peritoneal cavity cancer

  • Recurrent, relapsed, or persistent disease meeting 1 or more of the following criteria:

    • Patients with a positive second-look laparotomy who are not candidates for higher priority GOG protocols
    • Largest mass of recurrent disease ≤ 0.2 cm achieved by surgery or chemotherapy
    • Achievement of complete response to 1 prior regimen of platinum-based chemotherapy with relapse > 6 months from last chemotherapy
    • Achievement of partial response to 1 platinum-based chemotherapy regimen prior to study

  • Histological proof of disease recurrence with or without a rising serum CA-125 level (relapsed or recurrent disease)

• The following histological cell types are allowed:

  • Clear-cell adenocarcinoma
  • Endometrioid adenocarcinoma
  • Mixed epithelial carcinoma
  • Mucinous adenocarcinoma
  • Serous adenocarcinoma
  • Undifferentiated carcinoma

• Must have unilateral bone marrow aspirate and biopsy with cytogenetics without evidence of metastatic ovarian carcinoma by conventional morphology within 1 month of registration

• Not eligible for GOG-164

PATIENT CHARACTERISTICS:

• ECOG performance status 0-2

• Creatinine ≤ 1.5 mg/dL

• Total bilirubin ≤ 2.0 mg/dL (≤ 5.0 mg/dL with metastatic disease)

• AST ≤ 2 times upper limit of normal (ULN) (≤ 600 units/mL with metastatic disease)

• Alkaline phosphatase ≤ 2 times ULN (unless related to metastatic disease)

• ANC ≥ 1,000/mm^3

• Platelets ≥ 100,000/mm^3

• Cardiac ejection fraction ≥ 45% by rest ECHO or MUGA

• FEV_1 ≥ 50% of predicted

• HIV negative

• No uncontrolled infection

• No severe medical or psychiatric illness, including any of the following:

  • Renal failure
  • Brittle insulin dependent diabetes mellitus
  • Congestive heart failure
  • History of myocardial infarction within the past 3 months
  • Significant arrhythmia requiring medication
  • Poorly controlled hypertension (diastolic blood pressure >100 mm Hg)
  • History of hospitalization for severe depression or psychosis
  • Significant non-neoplastic pulmonary disease
  • Current alcohol or drug abuse.
  • Active infection
  • Active peptic ulcer disease

• No prior malignancy within the past 5 years except adequately treated basal cell or squamous cell carcinoma of the skin or in situ cervical carcinoma

PRIOR CONCURRENT THERAPY:

• See Disease Characteristics
• No more than 1 prior treatment regimen for this cancer
• More than 3 weeks since surgery
• No prior topotecan hydrochloride