This randomized phase I trial studies the side effects and best dose of RO4929097 in treating patients with advanced solid tumors. RO4929097 may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth.

PRIMARY OBJECTIVES:

I. To evaluate the extent that RO4929097 induces its own metabolism using a 20mg dose on Regimen I and a 50mg dose on Regimen II by comparing Cycle 1 Day 1 and Day 10 plasma pharmacokinetic parameters.

SECONDARY OBJECTIVES:

I. To evaluate the effect of the strong inhibitor of CYP3A4, ketoconazole, on RO4929097 plasma pharmacokinetics.

II. To evaluate the effect of the strong inducer of CYP3A4, 2D6 and 2C9, rifampin on RO4929097 plasma pharmacokinetics.

III. To evaluate the effect of RO4929097 on the plasma pharmacokinetics of CYP450 substrates; midazolam (CYP3A4), omeprazole (CYP2C19), tolbutamide (CYP2C9) and dextromethorphan (CYP2D6) after single dose and chronic administration.

IV. To assess the influence of polymorphisms in CYP3A4, 3A5, 2C9, ABCB1, and 2D6 on RO4929097 plasma pharmacokinetics V. To assess any evidence of clinical activity (CR, PR, SD) in patients with advanced solid tumors.

OUTLINE: Patients are randomized to 1 of 2 treatment regimens.

Regimen I (low-dose of RO4929097): Patients receive low-dose RO4929097 orally (PO) once daily on days 1-3, 8-10, and 15-17 and midazolam hydrochloride IV over 1 minute, omeprazole PO, tolbutamide PO, and dextromethorphan hydrobromide PO on days 1 and 10. After completion of course 1, patients are randomized to 1 of 2 treatment arms.

Arm A: Patients receive low-dose RO4929097 PO once daily on days 1-3, 8-10, and 15-17 and ketoconazole PO once daily on days 1-10 for course 2 only. For course 3 and beyond, patients receive low-dose RO4929097 on days 1-3, 8-10, and 15-17. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

Arm B: Patients receive low-dose RO4929097 PO once daily on days 1-3, 8-10, and 15-17 and rifampin PO once daily on days 1-10 for course 2 only. For course 3 and beyond, patients receive low-dose RO4929097 on days 1-3, 8-10, and 15-17. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

Regimen II (high-dose of RO4929097): Patients receive high-dose RO4929097 PO once daily on days 1-3, 8-10, and 15-17 and midazolam hydrochloride IV, oral omeprazole, oral tolbutamide, and oral dextromethorphan hydrobromide on days 1 and 10. After completion of course 1, patients are randomized to 1 of 2 treatment arms.

Arm A: Patients receive high-dose RO4929097 once daily on days 1-3, 8-10, and 15-17 and oral ketoconazole once daily on days 1-10 for course 2 only.

Arm B: Patients receive high-dose RO4929097 once daily on days 1-3, 8-10, and 15-17 and oral rifampin once daily on days 1-10 for course 2 only.

In all arms, treatment with RO4929097 repeats every 21 days for >= 3 courses in the absence of disease progression or unacceptable toxicity.

Patients undergo blood sample collection at baseline and periodically during study for pharmacokinetics studies and evaluation of SNPs in CYP2D6, CYP2C9, ABCB1, and CYP3A4/5.

After completion of study therapy, patients are followed up for 30 days.

Inclusion Criteria:

• Patients must have histologically confirmed malignancy that is metastatic or unresectable and for which there is no standard therapy
• Patients must not have received radiation to > 25% of bone marrow
• ECOG performance status =< 2 (Karnofsky >= 60%)
• Life expectancy of greater than 12 weeks
• Hemoglobin >= 9 g/dL
• Leukocytes >= 3,000/mcL
• Absolute neutrophil count >= 1,500/mcL
• Platelets >= 100,000/mcL
• Total bilirubin within normal institutional limits
• AST(SGOT)/ALT(SGPT) =< 2.5 X institutional upper limit of normal
• Creatinine within normal institutional limits OR creatinine clearance >= 60 mL/min/1.73 m^2 for patients with creatinine levels above institutional normal
• Preclinical studies indicate that RO4929097 is a substrate of CYP3A4 and inducer of CYP3A4 enzyme activity; caution should be exercised when dosing RO4929097 concurrently with CYP3A4 substrates, inducers, and/or inhibitors; furthermore, patients who are taking concurrent medications that are strong inducers/inhibitors or substrates of CYP3A4 should be switched to alternative medications to minimize any potential risk; if such patients cannot be switched to alternative medications, they will be ineligible to participate in this study
• Caution should be exercised when dosing ketoconazole, rifampin, omeprazole, midazolam, tolbutamide, and dextromethorphan concurrently with CYP3A4 substrates, inducers, and/or inhibitors; furthermore, patients who are taking concurrent medications that are strong inducers/inhibitors or substrates of CYP3A4 should be switched to alternative medications to minimize any potential risk; if such patients cannot be switched to alternative medications, they will be ineligible to participate in this study
• The effects of RO4929097on the developing human fetus at the recommended therapeutic dose are unknown; for this reason and because Notch signal pathway inhibitors are known to be teratogenic, women of childbearing potential and men must use two forms of contraception (i.e., barrier contraception and one other method of contraception) at least 4 weeks prior to study entry, for the duration of study participation, and for at least 12 months post-treatment; should a woman become pregnant or suspect she is pregnant while she or her partner are participating in this study and for 12 months after study participation, the patient should inform the treating physician immediately
• Women randomized to Regimen I (Arm B) and Regimen II (Arm B), and receiving rifampin will need to use an additional, non-hormonal birth control during cycle 2; rifampin induces enzymes responsible for hormone metabolism, making hormonal birth control ineffective; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately
• Ability to understand and the willingness to sign a written informed consent document
• Patients must have measurable disease, or radiographically evaluable metastatic disease (e.g. osseous metastases) with evidence of disease progression (e.g. new lesions or rising tumor markers)
• Treated, stable brain metastases are allowed; patients must be four weeks from radiation with stable brain imaging and off any medications used to treat brain metastases, excepting those anti-epileptics not metabolized by cytochrome P450

Exclusion Criteria:

• Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered (=< grade 1) from clinically significant adverse events due to agents administered more than 4 weeks earlier; prior palliative radiotherapy is allowed if greater than 2 weeks have elapsed and patient has also recovered to baseline or grade < 1 from any treatment adverse effects
• Patients may not be receiving any other investigational agents
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to RO4929097, or other agents used in study
• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, and a history of torsades de pointes or other significant cardiac arrhythmia other than chronic, stable atrial fibrillation, or psychiatric illness/social situations that would limit compliance with study requirements
• Pregnant women are excluded from this study because RO4929097 is a gamma-secretase inhibitor with the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with RO4929097, breastfeeding should be discontinued if the mother is treated with RO4929097; these potential risks may also apply to other agents used in this study
• HIV-positive patients receiving combination antiretroviral therapy are ineligible because of the potential for pharmacokinetic interactions with RO4929097; in addition, these patients are at increased risk of lethal infections when treated with marrow-suppressive therapy; appropriate studies will be undertaken in patients receiving combination antiretroviral therapy when indicated
• Patients with uncontrolled hypocalcemia, hypomagnesemia, hyponatremia, hypophosphatemia or hypokalemia defined as less than the lower limit of normal for the institution, despite adequate electrolyte supplementation are excluded from this study; Note: it is acceptable to use corrected calcium when interpreting calcium levels
• Patients must not be taking omeprazole or dextromethorphan, or be willing to take only the study dose and formulation on the PK days; patients who require or are likely to require therapeutic doses of these drugs are excluded
• Patients must not be taking rifampin, ketoconazole, tolbutamide, or midazolam; patients who require or are likely to require therapeutic doses of these drugs are excluded
• Patients must not be taking monoamine oxidase inhibitors, such as Clorgyline, Iproniazid, Isocarboxazid, Moclobemide, Nialamide, Pargyline, Phenelzine, Procarbazine, Rasagiline, Selegiline, Teloxantrone, or Tranylcypromine because of drug interactions with dextromethorphan
• Patients must not have acute narrow-angle glaucoma or untreated open-angle glaucoma, as midazolam is contraindicated
• Baseline QTcF > 450 msec
• Patients who are serologically positive for Hepatitis A, B or C, or have a history of liver disease, other forms of hepatitis or cirrhosis are ineligible
• Patients with malabsorption syndrome or other condition that would interfere with intestinal absorption; patients must be able to swallow tablets
• Because opioid containing pain medications may be metabolized by the CYP3A4 pathway, patients should be monitored carefully to avoid toxicity and doses adjusted if necessary; subjects who would not tolerate adjustments in their opioid pain medications are excluded
• Subjects with a history of hypoglycemia, diabetes mellitus, or abnormal glucagon regulation are excluded; subjects with a current diagnosis of diabetes mellitus who are receiving treatment (i.e. insulin or other medication) are not eligible; a diagnosis of diabetes mellitus that is managed by diet is allowable
• Subjects with mild to severe skin exanthems are excluded
• A requirement for antiarrhythmics or other medications known to prolong QTc