This randomized phase II trial studies combination chemotherapy when given together with vismodegib to see how well it works compared with combination chemotherapy without vismodegib in treating patients with advanced stomach cancer or gastroesophageal junction cancer. Drugs used in chemotherapy, such as oxaliplatin, leucovorin calcium, and fluorouracil, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving more than one drug (combination chemotherapy) may kill more tumor cells. Vismodegib may stop the growth of stomach or gastroesophageal junction cancer by blocking the growth of new blood vessels necessary for tumor growth. It is not yet known whether combination chemotherapy is more effective when given with or without vismodegib in treating stomach cancer and gastroesophageal junction cancer.

PRIMARY OBJECTIVES:

I. To determine if the addition of GDC-0449 (vismodegib) to FOLFOX (fluorouracil, leucovorin calcium, oxaliplatin) chemotherapy improves median progression free survival (PFS) in the first line treatment of patients with advanced gastric and gastroesophageal junction (GEJ) adenocarcinoma.

SECONDARY OBJECTIVES:

I. To determine if the addition of GDC-0449 to FOLFOX chemotherapy affects overall survival.

II. To determine if the addition of GDC-0449 to FOLFOX chemotherapy affects response rate.

III. To determine if the addition of GDC-0449 to FOLFOX chemotherapy affects toxicity rates in the first line treatment of patients with advanced gastric and GEJ adenocarcinoma.

TERTIARY OBJECTIVES:

I. To determine the level of baseline hedgehog pathway activation and correlate with clinical outcome and response to treatment with GDC-0449.

II. In those patients who consent to repeat biopsy at week 4-5, hedgehog pathway expression will again be assessed (every attempt will be made to obtain repeat biopsy from the same site as the initial biopsy) and compared to baseline values and clinical outcome.

III. To determine a primary gastric cancer gene expression profile that may predict response to GDC-0449.

IV. To determine if serum shed collagen epitopes correlate with clinical outcome and may be used to assess efficacy of GDC-0449 treatment.

V. To determine if circulating endothelial progenitor cells (EPC)'s correlate with treatment response and may be used to assess efficacy of GDC-0449 treatment.

VI. To determine if hedgehog pathway expression is downregulated in EPC's following treatment with GDC-0449.

VII. To determine if serum expression of vascular endothelial growth factor (VEGF), transforming growth factor (TGF)-beta, and insulin-like growth factor binding protein (IGFBP) 3 correlate with clinical outcome and may be used to assess efficacy of GDC-0449 treatment.

VIII. To determine if human epidermal growth factor receptor 2 (Her2) expression is predictive in assessing the efficacy of GDC-0449 treatment. Of note, Her2 status will be collected retrospectively for those patients who were tested as part of standard of care established in October 2010.

OUTLINE: Patients are randomized to 1 of 2 treatment arms.

ARM I: Patients receive FOLFOX chemotherapy comprising oxaliplatin intravenously (IV) over 2 hours, leucovorin calcium IV over 2 hours, and fluorouracil IV over 46-48 hours on day 1. Patients also receive placebo orally (PO) once daily (QD) on days 1-14.

ARM II: Patients receive FOLFOX chemotherapy as in Arm I. Patients also receive vismodegib PO on days 1-14. In both arms, treatment repeats every 2 weeks in the absence of unacceptable toxicity or disease progression.

After completion of study treatment, patients are followed up every 3 months.

Inclusion Criteria:

• Patients must have histologically or cytologically confirmed gastric or gastroesophageal junction (GEJ) adenocarcinoma not amenable to surgical resection

• Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as >= 20 mm with conventional techniques or as >= 10 mm with spiral computed tomography (CT) scan

• No prior chemotherapy for advanced disease; patients may have receive adjuvant chemotherapy or chemoradiation if > 6 months has elapsed since completion of treatment

• Life expectancy of greater than 3 months

• Eastern Cooperative Oncology Group (ECOG) performance status < 2 (Karnofsky > 70%)

• Absolute neutrophil count >= 1,500/mcL

• Platelets >= 100,000/mcL

• Total bilirubin =< 1.5 times upper limit of normal (ULN)

• Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT]) =< 2.5 X institutional upper limit of normal (=< 5.0 X institutional upper limit of normal with presence of liver metastases)

• Creatinine =< 1.5 X institutional upper limit of normal OR creatinine clearance >= 60 mL/min/1.73 m^2 for patients with creatinine levels above institutional normal

• Baseline imaging studies performed =< 28 days of study registration; the treating investigator will determine the appropriate imaging studies, which may include CT scan, magnetic resonance imaging (MRI), and/or fludeoxyglucose F 18 (FDG)-positron emission tomography (PET)/CT

• Must be willing to provide blood and tissue samples for research purposes; patient has the right to later withdraw consent for research studies and/or tissue specimens

• Patients must agree to placement of a central venous catheter for chemotherapy administration

• Patients must be able to swallow whole capsules

• Patients taking medications with narrow therapeutic indices that are metabolized by cytochrome P450 (CYP450), including warfarin sodium (Coumadin), must be on a stable, therapeutic dose and have close monitoring of their levels

• Women of child-bearing potential and men must use two forms of contraception (i.e., barrier contraception and one other method of contraception) at least 4 weeks prior to study entry, for the duration of study participation, and for at least 12 months post-treatment; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately

• Pregnancy testing: women of childbearing potential are required to have a negative serum pregnancy test (with a sensitivity of at least 25 mIU/mL) within 10-14 days and within 24 hours prior to the first dose of GDC-0449/placebo (serum or urine); a pregnancy test (serum or urine) will be administered every 4 weeks if their menstrual cycles are regular or every 2 weeks if their cycles are irregular while on study within the 24-hour period prior to the administration of GDC-0449/placebo; a positive urine test must be confirmed by a serum pregnancy test; prior to dispensing GDC-0449/placebo, the investigator must confirm and document the patient's use of two contraceptive methods, dates of negative pregnancy test, and confirm the patient's understanding of the teratogenic potential of GDC-0449/placebo

• Female subjects of childbearing potential are defined as follows:

  • Patients with regular menses
  • Patients, after menarche with amenorrhea, irregular cycles, or using a contraceptive method that precludes withdrawal bleeding
  • Women who have had tubal ligation

• Female subjects may be considered to NOT be of childbearing potential for the following reasons:

  • The patient has undergone hysterectomy and/or bilateral oophorectomy.
  • The patient is post-menopausal defined by amenorrhea for at least 1 year in a woman > 45 years old

• Ability to understand and the willingness to sign a written informed consent document

Exclusion Criteria:

• Patients who have had chemotherapy or radiotherapy within 6 months prior to entering the study
• Patients may not be receiving any other investigational agents
• Patients with known brain metastases should be excluded from this clinical trial
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to GDC-0449, 5-fluorouracil or oxaliplatin
• GDC-0449 inhibits cytochrome P450, family 2, subfamily C, polypeptide 8 (CYP2C8), cytochrome P450, family 2, subfamily C, polypeptide 9 (CYP2C9), and cytochrome P450, family 2, subfamily C, polypeptide 19 (CYP2C19) drug metabolism enzymes in vitro at concentrations that may be clinically relevant; therefore, caution should be exercised when dosing GDC-0449 concurrently with medications that are substrates of CYP2C8, CYP2C9, and CYP2C19 and have narrow therapeutic windows
• Patients with malabsorption syndrome or other condition that would interfere with intestinal absorption
• Patients unable to swallow whole capsules
• Patients with clinically active liver disease, including viral or other hepatitis or cirrhosis are ineligible
• Patients with uncontrolled hypocalcemia, hypomagnesemia, hyponatremia or hypokalemia defined as less than the lower limit of normal for the institution, despite adequate electrolyte supplementation are excluded from this study
• Pre-existing > grade 1 peripheral sensory neuropathy
• Previous or concurrent malignancy; exceptions: treated basal cell or squamous cell skin cancer, in situ cervical cancer, or lobular carcinoma in situ in one breast; or other cancer which the patient has been disease-free ≥5 years
• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with GDC-0449
• Human immunodeficiency virus (HIV)-positive patients on combination antiretroviral therapy are ineligible