RATIONALE: Drugs used in chemotherapy use different ways to stop tumor cells from dividing so they stop growing or die. Combining chemotherapy with peripheral stem cell transplantation may allow the doctor to give higher doses of chemotherapy drugs and kill more tumor cells. Radiation therapy uses high-energy x-rays to damage tumor cells.

PURPOSE: Phase I trial to study the effectiveness of combination chemotherapy, peripheral stem cell transplantation, and radiation therapy in treating patients with recurrent metastatic Ewing's sarcoma, peripheral primitive neuroectodermal tumor, or rhabdomyosarcoma.

OBJECTIVES: I. Estimate the maximum tolerated dose of total bone marrow irradiation (TMI) that can be administered as planned consolidation utilizing autologous peripheral blood stem cell support following local radiotherapy (if indicated) and prior busulfan, melphalan, and thiotepa. II. Examine the efficacy of this dual transplant approach for high-risk patients with Ewing's sarcoma, peripheral primitive neuroectodermal tumor, or rhabdomyosarcoma in first complete remission or greater.

OUTLINE: This is a two part, radiation dose escalation study. Peripheral blood stem cells (PBSC) are collected after 5-6 daily injections of G-CSF. The PBSC are infused in two halves. One half is given after chemotherapy and the other half after total marrow irradiation (TMI). Transplant #1 (part one) consists of chemotherapy and PBSC infusion. Busulfan (BU) is administered orally every 6 hours for 3 days for a total of 12 doses on days -8, -7 and -6. Melphalan is intravenously infused over 30 minutes for 2 days on days -5 and -4. Thiotepa is intravenously infused over 2 hours on days -3 and -2. PBSC are infused on day 0, 36-48 hours after completion of chemotherapy. Patients are considered for local irradiation therapy between transplant #1 and #2 if tissue limiting irradiation doses to bulk tumor site have not previously been administered. The local irradiation is given immediately prior to TMI administration. Transplant #2 starts sometime between day 60 and 120 after transplant #1. For transplant #2, cohorts of 4 patients are treated with TMI twice a day for 5 days at initial dose level on days -5 through -1. TMI is administered over 30-40 minutes. The second half of the PBSC is infused 1-24 hours following the last dose of TMI. After treatment of at least 4 patients at the initial TMI dose level, dose levels escalate in the absence of toxicity. If there is no dose limiting toxicity (DLT) in the current group of 4 patients, the next cohort is treated at the next higher dose level. If 1 of the 4 patients experiences DLT, the next cohort is treated at the same dose. If 1 DLT is seen among 8 patients treated at a dose, then the next cohort is treated at the next higher dose level. If 2 patients out of 8 experience DLT, this dose is identified as the maximum tolerated dose (MTD). If 1 out of 4 or 3 out 8 patients experience DLT at a dose level, the next lower dose level is identified as the MTD. Each patient in a cohort is observed for a minimum of 28 days prior to escalation to the higher dose level. Tumor restaging occurs approximately 9 months after initial transplant, then at 12 months and annually thereafter.

PROJECTED ACCRUAL: An expected 12-16 patients are required to complete this study. Accrual should last 3-4 years at 4-5 patients per year.

DISEASE CHARACTERISTICS: Histologically confirmed recurrent metastatic Ewing's sarcoma, peripheral primitive neuroectodermal tumor (PNET) or Clinical Group IV rhabdomyosarcoma Complete remission or very good partial remission (at least 50% reduction in measurable tumor burden) following initial chemotherapy with or without surgery No primary CNS PNET

PATIENT CHARACTERISTICS: Age: Under 50 Performance status: Karnofsky 70-100% Life expectancy: Greater than 3 months Hematopoietic: Granulocyte count at least 1,000/mm3 Platelet count at least 100,000/mm3 Hepatic: Bilirubin less than 2.0 mg/dL Renal: Creatinine clearance greater than 50% of normal Pulmonary: LVEF greater than 41% Other: HIV negative Not pregnant

PRIOR CONCURRENT THERAPY: Biologic therapy: Not specified Chemotherapy: See Disease Characteristics Endocrine therapy: Not specified Radiotherapy: No prior dose limiting irradiation to any organ site Surgery: See Disease Characteristics