This phase I/II trial studies the side effects and best dose of nivolumab when giving together with stereotactic radiosurgery or whole brain radiotherapy with or without ipilimumab and to see how well they work in treating patients with non-small cell lung cancer that has spread to the brain. Monoclonal antibodies, such as nivolumab and ipilimumab, may interfere with the ability of tumor cells to grow and spread. Stereotactic radiosurgery is a specialized radiation therapy that delivers a single, high dose of radiation directly to the tumor and may cause less damage to normal tissue. Radiation therapy, such as whole-brain radiotherapy, uses high energy x-rays to kill tumor cells and shrink tumors. Giving nivolumab together with stereotactic radiosurgery or whole brain radiotherapy with or without ipilimumab may work better in treating patients with non-small cell lung cancer that has spread to the brain.

PRIMARY OBJECTIVES:

I. To determine the recommended phase 2 dose (RP2D) of nivolumab with intracranial radiation and ipilimumab in combination with nivolumab and intracranial radiation in non-small cell lung cancer (NSCLC) with brain metastasis. (Phase I) II. To determine the RP2D of nivolumab in combination with stereotactic radiosurgery (SRS). (Phase I) III. To determine the RP2D of nivolumab in combination with whole brain radiation therapy (WBRT). (Phase I) IV. To determine the RP2D of ipilimumab in combination with nivolumab and SRS. (Phase I) V. To determine the RP2D of ipilimumab in combination with nivolumab and WBRT. (Phase I) VI. To estimate the 4-month intracranial progression free survival overall and within each treatment group; nivolumab and SRS; nivolumab and WBRT; nivolumab + ipilimumab and SRS; nivolumab + ipilimumab and WBRT. (Phase II)

SECONDARY OBJECTIVES:

I. To assess the potential neurocognitive changes in all treatment groups using the Hopkins Verbal Learning Revised (HVLT-R) total recall test. (Phase II) II. To estimate the rate of extracranial progression overall and within each treatment group. (Phase II) III. To estimate overall survival overall and within each treatment group. (Phase II) IV. To estimate the objective response rate of extracranial disease among all groups and within each treatment group. (Phase II) V. To estimate the duration of treatment response extracranially in patients who achieve an objective response. (Phase II) VI. To estimate steroid requirements in patients overall and within each treatment group. (Phase II)

OUTLINE: This is a phase I, dose-escalation study of nivolumab followed by a phase II study. Patients are assigned to 1 of 4 groups.

GROUP A: Patients receive nivolumab intravenously (IV) over 90 minutes every 2 weeks in the absence of disease progression or unacceptable toxicity. Patients undergo SRS once the day after nivolumab administration.

GROUP B: Patients then receive nivolumab as in Group A. Patients undergo WBRT once daily for 10 days.

GROUP C: Patients receive nivolumab as in Group A and ipilimumab IV over 90 minutes every 6 weeks in the absence of disease progression or unacceptable toxicity. Patients undergo SRS once the day after nivolumab administration.

GROUP D: Patients receive nivolumab as in Group A and ipilimumab as in Group C. Patients undergo WBRT once daily for 10 days.

After completion of study treatment, patients are followed up at 30 days and then every 12 weeks for up to 1 year.

Inclusion Criteria:

• Pathologically confirmed non-small lung cancer
• Stage IV metastatic disease with intracranial disease visible with magnetic resonance image (MRI)
• At least one brain lesion size >= 0.3 cm in the longest axis amenable to radiation therapy (either via SRS or WBRT)
• Be willing and able to provide written informed consent/assent for the trial
• Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) performance scale
• Absolute neutrophil count (ANC) >= 1,000 /mcL (performed 28 days prior to study registration up to the first dose of study drug)
• Platelets >= 100,000 /mcL (performed 28 days prior to study registration up to the first dose of study drug)
• Hemoglobin >= 9 g/dL or >= 5.6 mmol/L (performed 28 days prior to study registration up to the first dose of study drug)
• Serum creatinine or measured or calculated creatinine clearance (glomerular filtration rate [GFR] can also be used in place of creatinine or creatinine clearance [CrCl]) =< 1.5 X upper limit of normal (ULN) or >= 40 mL/min CrCl using the Cockroft-Gault formula (performed 28 days prior to study registration up to the first dose of study drug)
• Serum total bilirubin =< 1.5 X ULN (except for subjects with Gilbert syndrome, who may have total bilirubin < 3.0 mg/dl) or direct bilirubin =< ULN for subjects with total bilirubin levels > 1.5 x ULN (performed 28 days prior to study registration up to the first dose of study drug)
• Aspartate aminotransferase (AST [serum glutamic-oxaloacetic transaminase (SGOT)]) and alanine aminotransferase (ALT [serum glutamate pyruvate transaminase (SGPT)]) =< 3 X ULN or =< 5 X ULN for subjects with the liver metastases (performed 28 days prior to study registration up to the first dose of study drug)
• International normalized ratio (INR) or prothrombin time (PT) =< 1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants (performed 28 days prior to study registration up to the first dose of study drug)
• Activated partial thromboplastin time (aPTT) =< 1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants (performed 28 days prior to study registration up to the first dose of study drug)
• Female subject of childbearing potential should have a negative urine or serum pregnancy within 24 hours of study enrollment up to administration of the dose of study drug; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required
• Female subjects of childbearing potential should be willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 31 weeks after the last dose of study medication; subjects of childbearing potential are those who have not been surgically sterilized or have not been free from menses for > 1 year
• Male subjects should agree to use an adequate method of contraception starting with the first dose of study therapy through 31 weeks after the last dose of study therapy
• We will allow prior radiation to other sites, with no washout period, prior to study entry as long as the high dose regions of the prior and proposed radiation fields do not overlap; in patients where the prior high dose area would overlap with the high dose area of the intended radiation, a 4 month washout period will be required; the safety of such treatment will be at discretion of the treating radiation oncologist
• Prior central nervous system (CNS) radiation is allowed as long as cumulative radiation doses do not exceed tolerance of critical structures as judged by the treating radiation oncologist

Exclusion Criteria:

• Is currently participating in or has participated in a study of an investigational agent or using an investigational device within 4 weeks of the first dose of treatment or 5 half-lives, whichever is shorter
• Has a diagnosis of severe active scleroderma, lupus, other rheumatologic or autoimmune disease within the past 3 months; patients with a documented history of clinically severe autoimmune disease or a syndrome requiring systemic steroids or immunosuppressive agents will not be allowed on this study; subjects with vitiligo or resolved childhood asthma/atopy are an exception to this rule; subjects that require intermittent use of bronchodilators or local steroid injections are not excluded from the study; subjects with hypothyroidism stable on hormone replacement are not excluded from this study
• Has had a prior monoclonal antibody within 4 weeks or 5 half-lives, whichever is shorter, prior to study day 1 or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier
• For the nivolumab only arm patients who received anti PD1 or anti PD-L1 therapies will be excluded, for ipilimumab and nivolumab arms patients who received anti PD1 or anti PD L1 therapies will be eligible
• Has had prior chemotherapy or targeted small molecule therapy within 3 weeks prior to administration of the study drug or who has not recovered (i.e., =< grade 1 or at baseline) from adverse events due to a previously administered agent; *Note: Subjects with permanent =< grade 2 toxicities (e.g. neuropathy) or toxicities corrected through routine medical management (e.g. thyroid replacement for hypothyroidism), are an exception to this criterion and may qualify for the study; *Note: If subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy; *Note: Subjects with =< grade 2 amylase or lipase elevations abnormalities that have no corresponding clinical manifestations (e.g. manifestation of pancreatitis), are an exception to this criterion and may qualify for the study
• Has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin, indolent lymphomas, or in situ cervical cancer that has undergone potentially curative therapy
• Has known carcinomatous meningitis (also known as leptomeningeal disease)
• Has an active infection requiring intravenous systemic therapy or hospital admission
• Has a history or current evidence of any condition, therapy, or laboratory abnormality, including psychiatric or substance abuse disorder, that might confound the results of the trial, interfere with the subject's participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator
• Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the screening visit through 31 weeks after the last dose of trial treatment
• Has a known history of human immunodeficiency virus (HIV) (HIV 1/2 antibodies)
• Patients should be excluded if they are positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus ribonucleic acid (hepatitis C virus [HCV] antibody) indicating acute or chronic infection
• Has received a live vaccine 30 days prior to the first dose of trial treatment
• Has experienced grade 4 toxicity on treatment with prior radiation
• Has experienced grade 3-4 intracranial toxicity (hypophysitis or CNS toxicity) with either prior intracranial radiation, anti programmed cell death-1 (PD-1), or cytotoxic t-lymphocyte-associated protein 4 (CTLA-4) inhibitor therapy
• Is taking > 4mg/day of dexamethasone or its equivalent at the start of immunotherapy or has required > 4mg/day of dexamethasone or its equivalent for 3 consecutive days within 1 week of starting treatment
• Patients whose tumor exhibit activating EGFR mutation, ALK or ROS translocation and have a standard of care molecular targeted therapy available for these mutations, will be excluded from this study; patients who progressed or could not tolerate these standard of care molecular targeted agents are eligible for this study; adenocarcinoma patients may be consented prior to the EGFR and ALK status being known, but EGFR and ALK status must be determined prior to initiating therapy
• Allergies and adverse drug reaction to the following: history of allergy to study drug components; history of severe hypersensitivity reaction to any monoclonal antibody
• Previous CNS surgery within 2 weeks of treatment, with the exception of biopsy
• Unable or unwilling to tolerate an intracranial MRI