RATIONALE: Drugs used in chemotherapy use different ways to stop tumor cells from dividing so they stop growing or die. Combining chemotherapy with peripheral stem cell transplantation may allow the doctor to give higher doses of chemotherapy drugs and kill more tumor cells.

PURPOSE: Phase I/II trial to study the effectiveness of combination chemotherapy plus peripheral stem cell transplantation in treating patients with stage III or stage IV ovarian cancer that has not recurred or that has not responded to previous chemotherapy.

OBJECTIVES: I. Determine the maximum tolerated dose and the dose limiting toxicities of topotecan, mitoxantrone, and thiotepa given in combination followed by autologous peripheral blood stem cell transplantation in patients with recurrent or refractory platinum resistant epithelial ovarian cancer. II. Determine the progression-free survival and overall survival of these patients after this therapy.

OUTLINE: This is a dose escalation study of topotecan. All patients have peripheral stem cells collected. Patients then receive topotecan according to an escalating dose schedule, and mitoxantrone and thiotepa on a fixed dose schedule. Patients receive topotecan by continuous infusion for 24 hours on days 1-3, mitoxantrone intravenously over 1 hour on days 1-3, and thiotepa intravenously over 4 hours on days 1-3, followed 48 hours later by infusion of their peripheral stem cells. Patients may receive a second course of chemotherapy and peripheral stem cell transplantation in the absence of disease progression and unacceptable toxicity. Dose escalation of topotecan continues in cohorts of 3-6 patients each until the maximum tolerated dose (MTD) is determined. The MTD is defined as the dose preceding that at which 2 or more patients experience dose limiting toxicity. Patients are followed every week for the first month, then every month for 6 months, every 3 months for 1 year, and then every 6 months.

PROJECTED ACCRUAL: A total of 21-50 patients will be accrued for this study.

DISEASE CHARACTERISTICS: Histologically confirmed stage III or IV epithelial ovarian cancer that is refractory to platinum therapy or has relapsed within 12 months after platinum therapy Minimal residual disease by laparotomy Must have adequate number of peripheral stem cells collected No intraabdominal, pelvic disease, or other disease greater than 1 cm No brain metastasis

PATIENT CHARACTERISTICS: Age: 18 and over Performance status: Zubrod 0-2 Life expectancy: At least 12 weeks Hematopoietic: WBC greater than 3,500/mm3 Platelet count greater than 100,000/mm3 Hepatic: Bilirubin less than 2 times the upper limit of normal (ULN) Serum transaminases less than 2 times ULN Renal: Creatinine clearance greater than 60 mL/min Cardiovascular: Cardiac ejection fraction at least 45% No active angina No uncontrolled hypertension Pulmonary: FEV1, vital capacity, and diffusion capacity greater than 50% of predicted Other: Not HIV positive No active hepatitis B or C infection Not pregnant No concurrent malignancy except basal cell or squamous cell carcinoma of the skin No serious medical conditions such as uncontrolled diabetes mellitus

PRIOR CONCURRENT THERAPY: Biologic therapy: No immunotherapy within the past 4 weeks No concurrent immunotherapy Chemotherapy: No chemotherapy within the past 4 weeks No mitomycin within the past 6 weeks No other concurrent chemotherapy No prior anthracycline therapy greater than 200 mg/m2 Endocrine therapy: Not specified Radiotherapy: No radiotherapy within the past 4 weeks No concurrent radiotherapy No prior radiotherapy to the whole abdomen Surgery: Prior surgery allowed