RATIONALE: Talabostat may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as temozolomide and carboplatin, work in different ways to stop the growth of tumor cells, either by killing the cells or by stopping them from dividing. Giving talabostat together with temozolomide or carboplatin may kill more tumor cells.

PURPOSE: This phase I trial is studying the side effects and best dose of talabostat when given together with temozolomide or carboplatin in treating young patients with relapsed or refractory brain tumors or other solid tumors.

OBJECTIVES:

Primary

• Determine the dose of talabostat, when used in combination with either temozolomide or carboplatin, at which maximum plasma dipeptidyl peptidase IV enzyme inhibition is achieved (in the absence of talabostat-related dose-limiting toxicity) in pediatric patients with refractory or relapsed solid tumors, including brain tumors.
• Determine the maximum tolerated dose of talabostat, when used in combination with temozolomide or carboplatin in pediatric patients, if dose-limiting toxicity attributed to talabostat is observed.
• Define the toxicity profile of talabostat when used in combination with temozolomide or carboplatin.
• Describe the pharmacokinetic profile of talabostat in pediatric patients.

Secondary

• Study levels, at baseline and after drug administration, of serum cytokines (interleukin [IL]-2, IL-6, IL-10, filgrastim [G-CSF], tumor necrosis factor-α, IL-1β, IL-8, IP10, and thrombospondin) that may be important in the immune-mediated antitumor effect of talabostat.
• Evaluate the antitumor effect of talabostat in combination with temozolomide or carboplatin on pediatric solid tumors by direct assessment of tumor response.
• Study the effect of talabostat on neutrophil function.
• Evaluate the expression of fibroblast activation protein (FAP) in pediatric tumors using immunohistochemistry to detect FAP in paraffin-embedded tissue sections from existing tumor specimens, when available.

OUTLINE: This is a dose-escalation study of talabostat. Patients are stratified according to tumor histology and prior therapy.

Based on stratification, patients receive either oral temozolomide on days 1-5 or carboplatin IV over 30 minutes on days 1-2. Patients also receive oral talabostat on days 7-20. Treatment repeats every 28 days in the absence of disease progression or unacceptable toxicity. (Closed to accrual as of 5/25/2009)

Cohorts of 2-6 patients receive escalating doses of talabostat until the maximum tolerated dose (MTD) is determined. The MTD is defined as the dose preceding that at which 2 of 6 or < 4 of 12 patients experience dose-limiting toxicity during the first course of therapy.

PROJECTED ACCRUAL: A total of 26 patients will be accrued for this study.

DISEASE CHARACTERISTICS:

• Histologically confirmed solid tumors, including, but not limited to, any of the following:

  • Rhabdomyosarcoma and other soft tissue sarcomas

  • Ewing's sarcoma family of tumors

  • Osteosarcoma

  • Neuroblastoma

  • Wilms' tumor

  • Hepatic tumors

  • Germ cell tumors

  • Primary brain tumors

    • In patients with brainstem or optic gliomas, requirement for histological confirmation can be waived if biopsy was not performed

      • Patients with brainstem gliomas that did not respond to therapy but that are without radiographic evidence of disease progression must have clinical evidence of progression

    • Patients with brain tumors must be on stable or tapering dose of corticosteroids for 7 days prior to study entry

• Measurable or evaluable disease

• Relapsed or failed to respond to frontline curative therapy, including any of the following:

  • Surgery
  • Radiotherapy
  • Chemotherapy
  • Combination of modalities

• No other potentially curative treatment options available

PATIENT CHARACTERISTICS:

• ECOG performance status 0-2

• Absolute neutrophil count ≥ 1,500/mm^3

• Hemoglobin ≥ 8 mg/dL

• Platelet count ≥ 100,000/mm^3 (platelet transfusion independent)

• Bilirubin ≤ 1.5 times upper limit of normal (ULN)

• SGPT ≤ 2.5 times ULN

• Creatinine clearance ≥ 60 mL/min OR age-adjusted creatinine* as follows:

  • No more than 0.8 mg/dL (for patients ≤ 5 years of age)
  • No more than 1.0 mg/dL (for patients 6 to 10 years of age)
  • No more than 1.2 mg/dL (for patients 11 to 15 years of age)
  • No more than 1.5 mg/dL (for patients > 15 years of age) NOTE: *For patients receiving carboplatin a nuclear glomerular filtration rate study, 24-hour urine collection, and serum creatinine for estimation of creatinine clearance is required if under 15 years of age OR serum creatinine and weight for estimation of creatinine clearance is required if 15-18 years of age

• Patients with history of seizures eligible if seizures controlled by anticonvulsants

• No clinically significant, unrelated systemic illness, including either of the following:

  • Serious infections
  • Hepatic, renal, or other organ dysfunction that would preclude study treatment

• Not pregnant or nursing

• Negative pregnancy test

• Fertile patients must use effective contraception

• No generalized pitting peripheral edema

• No sensitivity to valine-proline boronic acid

PRIOR CONCURRENT THERAPY:

• See Disease Characteristics
• Recovered to ≤ grade 1 from the acute toxic effects of all prior chemotherapy, immunotherapy, or radiotherapy prior to study entry
• Any number of prior chemotherapy regimens allowed
• Prior temozolomide or carboplatin as frontline therapy or in the adjuvant setting allowed provided patient did not experience severe toxicities related to the drug and tumor progressed during this therapy
• At least 3 weeks since last dose of all myelosuppressive chemotherapy
• At least 7 days since last dose of anticancer biologic agents (e.g., retinoids)
• At least 30 days since prior investigational agents
• At least 4 weeks since prior radiotherapy to > 25% of marrow-containing bones (pelvis, spine, or skull) (2 weeks for palliative [limited-port] radiotherapy)
• At least 2 months since prior autologous stem cell transplantation and recovered
• At least 1 week since prior filgrastim (G-CSF), sargramostim (GM-CSF), or epoetin alfa
• At least 2 weeks since prior pegfilgrastim
• No history of allogeneic stem cell transplantation
• No other concurrent anticancer chemotherapy, radiation therapy, or immunotherapy
• No other concurrent investigational agents