The primary purpose of Parts A and B of this study is to evaluate the safety and toxicity of prexasertib (an inhibitor of checkpoint kinase 1[chk 1]) in participants with advanced or metastatic cancer (Part A), or squamous cell cancer of the head and neck or squamous cell cancer of any tumor type (Part B). Part C of the study will evaluate prexasertib in three different groups of participants; those with squamous cell cancer of the head and neck that has recurred or spread to other parts of the body, those with squamous non-small cell lung cancer that has recurred or spread, and those with squamous cell cancer of the anus that is not curable by existing therapy.

Part C added per protocol amendment (February, 2013).

Inclusion Criteria:

• Must be appropriate candidate for experimental therapy, as determined by investigator, after available standard therapies have failed
• Have adequate organ function
• Prior Therapies: Systemic treatments: must have discontinued previous systemic treatments for cancer and recovered from the acute effects of therapy. Participants must have discontinued mitomycin-C or nitrosourea therapy at least 42 days and have discontinued any cytotoxic therapies at least 28 days prior to study enrollment. Radiation therapy and surgery: must be completed at least 4 weeks before study enrollment
• Part A: Must have diagnosis of cancer that is advanced or metastatic
• Part B: Must have histologically confirmed squamous cell cancer of the head and neck or must have squamous cell cancer of any tumor type
• Part C: Must have histological diagnosis of squamous cell cancer of the head and neck, histological or cytological diagnosis of squamous non-small-cell lung cancer, or histological diagnosis of Stage IIIB (N2 or N3) or Stage IV squamous cell cancer of the anus that is not curable by local therapy
• Must be available during the duration of the study and willing to follow the study procedures
• If participant is of reproductive potential, must agree to use medically approved contraceptive precautions during the study and for three months following the last dose of study drug
• If the participant is a female of childbearing potential, must have had a negative serum or urine pregnancy test within 7 days of the first dose of study drug and must not be breast feeding

Exclusion Criteria:

• Must not have taken an unapproved drug as treatment for any indication within the last 28 days prior to starting study treatment
• Must not have an active symptomatic fungal, bacterial or viral infection, including human immunodeficiency virus (HIV) or Hepatitis A, B, or C
• Must not have a serious heart condition, such as congestive heart failure, unstable angina pectoris, or heart attack within the last three months
• Must not have systolic blood pressure <90 millimeters of mercury (mmHg) or recurrent symptomatic orthostatic hypotension
• Must not have a family history of long QTc syndrome or be taking drugs known to cause QTc prolongation or Torsades de Pointes
• Must not have a serotonin-secreting carcinoid tumor or a prior history of drug-induced serotonin syndrome
• Must not have acute leukemia