RATIONALE: Drugs used in chemotherapy use different ways to stop cancer cells from dividing so they stop growing or die. Peripheral stem cell transplantation may allow the doctor to give higher doses of chemotherapy drugs and kill more cancer cells. Isotretinoin may help cancer cells develop into normal white blood cells.

PURPOSE: Phase I/II trial of topotecan, fludarabine, cytarabine, and filgrastim followed by peripheral stem cell transplantation or isotretinoin in treating patients who have acute myeloid leukemia, myelodysplastic syndrome, or recurrent or refractory acute lymphocytic leukemia.

OBJECTIVES: I. Determine the safety of topotecan in combination with fludarabine, cytarabine, and filgrastim (FLAG) in patients with poor prognosis acute myeloid leukemia, myelodysplastic syndrome, or recurrent or refractory acute lymphocytic leukemia. II. Determine the maximum tolerated dose of topotecan in the FLAG regimen in these patients. III. Assess the complete remission rates in patients treated with this regimen.

OUTLINE: Patients with complete response proceed to further therapy according to age. 65 and under: Patients receive etoposide by IV continuously for 5 days, cytarabine IV over 2 hours every 12 hours for 4 days, and filgrastim (G-CSF) subcutaneously. Peripheral blood stem cells (PBSC) are then harvested. Patients then receive oral busulfan every 6 hours on days -7 to -4 and etoposide IV over 10 hours on day -3. PBSC are reinfused on day 0. Over 65: Patients receive oral isotretinoin and vitamin E daily. Treatment continues in the absence of disease progression or unacceptable toxicity. Patients are followed every month for 1 year and then every 3 months thereafter.

PROJECTED ACCRUAL: A total of 15 patients will be accrued for this study within 3-4 years.

DISEASE CHARACTERISTICS: Histologically proven poor prognosis acute myeloid leukemia (AML), myelodysplastic syndrome (MDS), or recurrent/refractory acute lymphocytic leukemia, including: Myelodysplastic syndrome (MDS) stages B and C Refractory anemia, refractory anemia with ringed sideroblasts, or refractory anemia with excess blasts (between 5% and 20% myeloblasts) MDS with increased erythroblasts or monocytoblasts of no greater than 20% MDS in transformation (between 20% to 30% myeloblasts) or acute nonlymphoblastic leukemia (at least 30% myeloblasts) Chronic myelomonocytic leukemia Poor prognosis refractory or recurrent acute myeloid leukemia after complete response Secondary or therapy related AML or MDS AML blastic crisis of chronic myelogenous leukemia or other myeloproliferative disorders such as polycythemia vera, essential thrombocytopenia, or agnogenic myeloid metaplasia

PATIENT CHARACTERISTICS: Age: 19 to 90 Performance status: ECOG 0-2 Life expectancy: Not specified Hematopoietic: Not specified Hepatic: Bilirubin no greater than 2.0 mg/dL SGOT or SGPT less than 2.0 times upper limit of normal Renal: Normal serum creatinine Cardiovascular: No congestive heart failure No symptomatic ischemic heart disease Other: Not pregnant or nursing Fertile patients must use effective contraception HIV negative No uncontrolled infection No poorly controlled diseases (e.g., diabetes, systemic lupus erythematosus) No history of psychiatric disorders No other malignancies within the past 5 years except adequately treated basal or squamous cell skin cancer or carcinoma in situ of the cervix No concurrent severe medical problems No history of allergic reaction to topotecan and its derivatives

PRIOR CONCURRENT THERAPY: Biologic therapy: No other concurrent immunotherapy Chemotherapy: No prior topotecan At least 4 weeks since prior chemotherapy No other concurrent chemotherapy Endocrine therapy: Not specified Radiotherapy: No concurrent radiotherapy Surgery: Not specified Other: No other investigational drugs within 30 days of study No other concurrent investigational therapy except for basal cell skin cancer