The main purpose of this study is to evaluate the safety, tolerability, and efficacy of the study drug known as galunisertib in combination with nivolumab in participants with advanced refractory solid tumors and in recurrent or refractory non-small cell lung cancer (NSCLC) or hepatocellular carcinoma (HCC).

Inclusion Criteria:

• For Phase 1b, must have advanced refractory solid tumors in any line of therapy.

• For Phase 2, must have one of the following tumor types: recurrent or refractory NSCLC (any histology), or HCC with elevated alpha-fetoprotein (AFP) ≥200 nanogram/milliliter (ng/mL).

• For Phase 2 only, have had disease progression or be refractory or intolerant to 1 prior line of therapy (first line therapy) for recurrent or refractory for NSCLC or HCC and have refused currently approved second-line therapy. First line therapy is defined as therapy used to treat advanced disease. This may include multiple chemotherapeutic, targeted or immunotherapeutic agents with or without radiation therapy and/or surgery. Each subsequent line of therapy is preceded by disease progression. A switch of an agent within the same drug class (eg, cisplatinum to carboplatinum) within a regimen in order to manage toxicity does not define the start of a new line of therapy.

• For NSCLC:

  • Prior lines of therapy must include a platinum-based therapy. Investigational agents used in combination with standard therapies are allowed. Participants who received platinum-based neoadjuvant or adjuvant therapy and subsequently received platinum-based therapy as first-line therapy are eligible.
  • Participants who have completed neo-adjuvant or adjuvant therapy with a platinum doublet and have experienced disease recurrence within 6 months of completing the platinum doublet are eligible.
  • Tumors with driver mutations (epidermal growth factor receptor mutation positive or anaplastic lymphoma kinase fusion oncogene positive) treated with a tyrosine kinase inhibitor or crizotinib are eligible. For participants who have progressed on a tyrosine kinase inhibitor or crizotinib or are intolerant to this targeted therapy, that participant must receive platinum-based therapy prior to enrollment in this study. Documentation of such mutations must be available and entered into the electronic case report form (eCRF).
  • Maintenance or switch maintenance therapy after first-line chemotherapy will be considered part of the first-line regimen and is acceptable.

Participants who completed and progressed on a platinum-containing regimen as adjuvant, neoadjuvant, or part of a course of chemoradiation therapy given from locally advanced disease and developed recurrent (local or metastatic) disease within the 6 months before screening would be counted as having received 1 prior platinum-containing regimen and therefore would not require re-treatment with a platinum-containing regimen for Stage IIIB, IV, or recurrent disease and are eligible. However, participants must have received at least 2 cycles of a platinum doublet based chemotherapy before discontinuation for toxicity. If participants received only one cycle of a platinum doublet and discontinue due to clear progression, that regimen should be counted as a prior line of therapy.

• For HCC:

  • One prior line of therapy which must include sorafenib or participant must have progressed or been intolerant to sorafenib for participants not eligible for transarterial chemoembolization. Participants who had sorafenib for locally advanced disease or are intolerant to sorafenib are eligible. Participants may have had clinical progression only following sorafenib or local therapy.
  • Must have Child-Pugh A only. Participants may have any viral status (hepatitis B, hepatitis C, or none).
  • Have a viral load <100 international units/milliliter (IU/mL).
  • For hepatitis B participants, must be on a nucleoside analog reverse transcriptase inhibitor (lamivudine, telbivudine, adefovir, tenofovir, or entecavir).

• Have adequate organ function.

• Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) scale.

• Use an approved contraceptive method.

Exclusion Criteria:

• For Phase 2 only, more than 1 prior line of therapy for their tumor type.

• Have moderate or severe cardiovascular disease:

  • Have the presence of cardiac disease, including a myocardial infarction within 6 months prior to study entry, unstable angina pectoris, New York Heart Association Class III/IV congestive heart failure, or uncontrolled hypertension.
  • Have documented major electrocardiogram (ECG) abnormalities which are clinically significant at the investigator's discretion (for example, symptomatic or sustained atrial or ventricular arrhythmias, second- or third-degree atrioventricular block, bundle-branch blocks, ventricular hypertrophy, or recent myocardial infarction).
  • Have major abnormalities documented by ECHO with Doppler:
  • Moderate or severe heart valve function defect including moderate or severe valve stenosis or regurgitation.
  • Left ventricular (LV) ejection fraction <50%, evaluation based on the institutional lower limit of normal.
  • Have septal aneurysm or other heart aneurysm.
  • Any aneurysm of the major vessels.

• Active infection with hepatitis B virus (HBV) (positive hepatitis B surface antigen); HCV is allowed only in HCC participants. HCC participants at risk for HBV reactivation (as defined by anti-hepatitis B core antibody positive) are only eligible in the HCC cohort.

• Have evidence of interstitial lung disease that is symptomatic or may interfere with the detection or management of suspected drug-related pulmonary toxicity or active, noninfectious pneumonitis.

Anonymized individual patient level data will be provided in a secure access environment upon approval of a research proposal and a signed data sharing agreement.

Data are available 6 months after the primary publication and approval of the indication studied in the US and EU, whichever is later. Data will be indefinitely available for requesting.

A research proposal must be approved by an independent review panel and researchers must sign a data sharing agreement.