RATIONALE: Drugs used in chemotherapy, such as docetaxel, work in different ways to stop the growth of tumor cells, either by killing the cells or by stopping them from dividing. Vandetanib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth and by blocking blood flow to the tumor. It is not yet known whether docetaxel is more effective when given alone or together with vandetanib.

PURPOSE: This randomized phase II trial is studying docetaxel given together with or without vandetanib to see how well it works in treating patients with persistent or recurrent ovarian epithelial cancer, fallopian tube cancer, or primary peritoneal cancer.

OBJECTIVES:

• To evaluate the clinical efficacy of docetaxel and vandetanib relative to docetaxel alone in patients with platinum-resistant, recurrent, refractory, or progressive/persistent ovarian epithelial, primary peritoneal, or fallopian tube cancer, as measured by progression-free survival.
• To evaluate the response rate (complete and partial) and duration of overall survival of these patients.
• To evaluate the response (complete and partial) and time to treatment failure after treatment with single agent vandetanib following progression on single agent docetaxel.
• To evaluate the frequency and severity of adverse events as assessed by Common Toxicity Criteria for Adverse Effects (CTCAE) v4.0.
• To evaluate the toxicity of single agent vandetanib following docetaxel as assessed by CTCAE v4.0.

OUTLINE: Patients are stratified according to prior treatment with antiangiogenesis agents (yes vs no). Patients are randomized to 1 of 2 treatment arms.

• Arm I: Patients receive docetaxel IV over 1 hour on day 1. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity. Patients who progress also receive oral vandetanib once daily on days 1-21. Courses repeat every 21 days in the absence of a second disease progression or unacceptable toxicity.
• Arm II: Patients receive docetaxel IV over 1 hour on day 1 and oral vandetanib once daily on days 1-21. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed every 3 months for 2 years and then every 6 months for 3 years.

DISEASE CHARACTERISTICS:

• Histologically confirmed ovarian epithelial, fallopian tube, or primary peritoneal carcinoma

  • Recurrent, refractory, or progressive/persistent disease

• Measurable or non-measurable disease documented by CT scan of the abdomen and pelvis

• Must have received 1 prior platinum-based chemotherapy regimen for management of primary disease containing carboplatin, cisplatin, or other organoplatinum compound

  • Initial treatment may have included any of the following:

    • High-dose therapy
    • Consolidation therapy
    • Non-cytotoxic agent therapy
    • Extended therapy administered after surgical or non-surgical assessment

  • Additional cytotoxic regimen for recurrent, refractory, or progressive/persistent disease, including re-treatment with primary treatment regimen

• No more than 3 prior regimens for recurrent, refractory, persistent, or progressive disease.

PATIENT CHARACTERISTICS:

• Zubrod performance status 0-2

• Absolute neutrophil count (ANC) ≥ 1,500/mcl

• Platelet count ≥ 100,000/mcl

• Serum creatinine normal OR calculated creatinine clearance ≥ 30 mL/min

• Urine protein:creatinine ratio < 1

• Bilirubin ≤ 1.5 times upper limit of normal (ULN)

• aspartate aminotransferase - alanine aminotransferase (AST or ALT) ≤ 2.5 times ULN (≤ 5 times ULN if liver metastases are present)

• Alkaline phosphatase ≤ 2.5 times ULN (≤ 5 times ULN if liver metastases are present)

• Not pregnant or nursing

• Negative pregnancy test

• Fertile patients must use effective contraception during and for ≥ 6 months after completion of vandetanib therapy

• No neuropathy ≥ grade 2 CTCAE v4.0

• No active infection requiring systemic or intravenous antibiotics

• No significant traumatic injury within the past 28 days

• No significant cardiovascular disease, including any of the following:

  • Uncontrolled hypertension (i.e., systolic blood pressure [BP] > 140 mm Hg or diastolic BP > 90 mm Hg) within the past 28 days

  • Myocardial infarction superior vena cava syndrome, or New York Heart Association (NYHA) class II-IV heart disease within the past 3 months

  • Presence of left bundle branch block

  • Congenital long QT syndrome or first degree relative with unexplained sudden death < 40 years of age

  • QT interval with Bazett's correction that is unmeasurable or ≥ 480 msec by screening ECG

  • History of symptomatic arrhythmia (i.e., multifocal premature ventricular contractions, bigeminy, trigeminy, ventricular tachycardia, or uncontrolled atrial fibrillation) requiring treatment (≥ CTCAE grade 3) or asymptomatic sustained ventricular tachycardia

    • Atrial fibrillation controlled on medication allowed

PRIOR CONCURRENT THERAPY:

• See Disease Characteristics

• Recovered from all prior therapy (except alopecia) to NCI CTCAE v3.0 grade ≤ 1

• No prior vandetanib

  • Treatment with other anti-vascular endothelial growth factor (VEGF) targeted therapy allowed

• No prior docetaxel or any non-cytotoxic therapy (excluding hormonal therapy) for recurrent disease, regardless of whether it was part of primary treatment

  • Prior docetaxel as part of front-line cytotoxic regimen (including maintenance therapy) allowed as long as no disease progression on or within 6 months after receiving docetaxel

• At least 7 days since prior hormonal therapy for the malignant tumor

  • Concurrent hormone replacement therapy for menopausal symptoms allowed

• At least 28 days since other prior therapy for the malignant tumor, including immunologic agents

• More than 7 days since prior minor surgical procedures, fine needle aspirates, or core biopsies

• More than 14 days since prior and no concurrent potent inducers of cytochrome P450 3A4 (CYP3A4) function

• More than 14 days since prior and no concurrent medications having a risk of causing Torsades de Pointes or risk of QTc prolongation

  • Patients receiving a drug that has a risk of QTc prolongation must not have QTc ≥ 460 msec

• More than 28 days since prior investigational agents for any purpose

• More than 28 days since prior and no concurrent major surgical procedure or open biopsy

• More than 5 years since prior chemotherapy for abdominal or pelvic tumor, except treatment of ovarian, fallopian tube, or primary peritoneal cancer

  • Prior adjuvant chemotherapy for localized breast cancer allowed, provided it was completed more than 3 years prior to study, and the patient remains free of recurrent or metastatic disease

• More than 5 years since prior radiotherapy to any portion of the abdominal cavity or pelvis, except for the treatment of ovarian, fallopian tube, or primary peritoneal cancer

  • Prior radiotherapy for localized cancer of the breast, head and neck, or skin allowed, provided it was completed more than 3 years prior to study, and the patient remains free of recurrent or metastatic disease

  • No prior radiation to more than 25% of marrow-bearing areas

    • More than 28 days since prior radiotherapy

• No other concurrent investigational or commercial agents