This phase I/II trial tests the safety, side effects, and best dose of abemaciclib and whether it works before 177Lu-PSMA-617 in treating patients with castration resistant prostate cancer that has spread to other places in the body (metastatic). Abemaciclib is in a class of medications called kinase inhibitors. It is highly selective inhibitors of cyclin-dependent kinase 4 and 6, which are proteins involved in cell differentiation and growth. It works by blocking the action of an abnormal protein that signals cancer cells to multiply. Radioligand therapy uses a small molecule (in this case 177Lu-PSMA-617), which carries a radioactive component to destroys tumor cells. When 177Lu-PSMA-617 is injected into the body, it attaches to the prostate-specific membrane antigen (PSMA) receptor found on tumor cells. After 177Lu-PSMA-617 attaches to the PSMA receptor, its radiation component destroys the tumor cell. Giving abemaciclib before 177Lu-PSMA-617 may help 177Lu-PSMA-617 kill more tumor cells.

PRIMARY OBJECTIVES:

I. To determine the recommended phase II dose (RP2D) for abemaciclib given as lead-in treatment prior to lutetium Lu 177 vipivotide tetraxetan (177Lu-PSMA-617) for each treatment cycle, as well as dose limiting toxicities (DLTs) of this combination regimen. (Part A)

II. To determine the change in prostate-specific membrane antigen (PSMA) uptake on gallium Ga 68 gozetotide (68Ga-PSMA-11) positron emission tomography (PET) scan following fourteen days of priming with abemaciclib treatment, relative to the pre-treatment baseline scan. (Part B (Expanded Cohort)).

SECONDARY OBJECTIVES:

I. To determine the (proportion of patients who experience >= 50% decline from baseline in serum prostate specific antigen (PSA) (PSA50) response of this combination treatment. (Part B [Expanded Cohort]) II. To describe the safety of this combination treatment regimen using National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) version (v.)5.0. (Part B [Expanded Cohort]) III. To determine the radiographic progression free survival (rPFS) according to Prostate Cancer Working Group 3 (PCWG3) guidelines among patients treated with this combination regimen. (Part B [Expanded Cohort]) IV. To determine the objective response rate (ORR) (complete response [CR] + partial response [PR]) as measured by Response Evaluation Criteria in Solid Tumors (RECIST) v1.1 response in soft tissue, lymph node and visceral lesions. (Part B [Expanded Cohort]) V. To determine the disease control rate (DCR) (CR + PR + stable disease (SD)) as measured by RECIST v1.1 response in soft tissue, lymph node and visceral lesions. (Part B [Expanded Cohort]) VI. To determine the overall survival (OS) of patients treated with this combination regimen. (Part B [Expanded Cohort]) VII. To determine the median duration of response (DOR) in patients treated with this combination regimen who achieve CR or PR as measured by RECIST v1.1. (Part B [Expanded Cohort])

EXPLORATORY OBJECTIVES:

I. To assess changes in tumor microenvironment using ribonucleic acid (RNA) and whole exome sequencing by comparing biopsies obtained pre and post-combination treatment using the established institutional biopsy protocol (PSMA biopsy study).

II. To assess changes in PSMA expression in biopsies at the time of progression relative to pre-treatment biopsies using immunohistochemistry (IHC).

III. To describe PSMA upregulation on imaging following 7 days of treatment with abemaciclib, and in particular compare to PSMA upregulation on imaging seen following 14 days of abemaciclib treatment.

IV. To describe PSMA expression and upregulation on imaging following combined treatment with abemaciclib and 177Lu-PSMA-617 at the time of subsequent therapy cycles (pre and post abemaciclib treatment with cycle 3) in some patients treated with RP2D in Part B (expansion cohort).

V. To describe patterns of progression following completion of treatment, including in patients with available 68Ga-PSMA-11 PET scans.

VI. To compare response to treatment and clinical outcomes with different dose levels of abemaciclib used in this study.

OUTLINE: This is a dose-escalation study of abemaciclib.

Patients receive abemaciclib orally (PO) twice daily (BID) on days 1-14 and lutetium Lu 177 vipivotide tetraxetan intravenously (IV) over 30 minutes on day 15. Treatment repeats every 6 weeks for up to 4 cycles in the absence of disease progression or unacceptable toxicity.

After completion of study intervention, patients are followed up at 30 days, and then will subsequently be followed for overall survival in the long-term follow-up period and contacted approximately every 3 months until consent is withdrawn, death occurs, patient is lost to follow-up or the study is concluded.

Inclusion Criteria:

1. Participants must have histologically or cytologically confirmed prostate cancer. Either fresh biopsy or archival tissue can be used for confirmation.

2. Age >= 18 years.

3. Patients must have metastatic castration resistant prostate cancer (mCRPC) with progression based on Prostate Cancer Working Group 3 (PCWG3) criteria.

4. Patients must have adenocarcinoma histology.

5. Prior treatment with at least one novel hormonal agents (NHA) such as abiraterone acetate, enzalutamide, apalutamide, darolutamide etc.

6. Patients must have prior orchiectomy and/or ongoing androgen-deprivation therapy and a castrate level of serum testosterone (< 50 ng/dL or < 1.7 nmol/L)

7. Patients must have a 68Ga-PSMA-11 PET scan with at least one PSMA-positive lesions (maximum standardized uptake value [SUVmax] greater than SUVmax of liver) as determined by nuclear medicine review prior to start of lead-in treatment with abemaciclib

8. Patients must have Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0-2

9. Patients must have life expectancy of > 6 months

10. Patients must have adequate organ function as outlined below and bone marrow reserve

    • White blood cell (WBC) > 2.5
    • Absolute neutrophil count (ANC) > 1.5
    • Hemoglobin (Hgb) >= 8.0 [Note- Participants may receive erythrocyte transfusions to achieve this hemoglobin level at the discretion of the investigator. Initial treatment must not begin earlier than the day after the erythrocyte transfusion]
    • Platelets (Plt) >= 100 x 10^9/Liter (100,000/Microliter)
    • Total bilirubin =< 1.5 x the institutional upper limit of normal (ULN). For patients with known Gilbert's Syndrome =< 2 ULN and direct bilirubin within normal limits is permitted
    • Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase (SGOT)) =< 3 X institutional upper limit of normal (=< 5.0 ULN for patients with liver metastases)
    • Alanine aminotransferase (ALT)(serum glutamic pyruvic transaminase (SGPT)) =< 3 X institutional upper limit of normal (=< 5.0 ULN for patients with liver metastases)
    • Creatinine =< 1.5 x within institutional upper limit of normal OR creatinine clearance glomerular filtration rate (GFR) >= 30 mL/min/1.73 m, calculated using the Cockcroft-Gault equation.

11. Patient must be able to swallow oral medications

12. Patients must have the ability to understand a written informed consent document, and the willingness to sign it

13. Human immunodeficiency virus (HIV)-infected individuals on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial

14. For participants with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated

15. Individuals with a history of hepatitis C virus (HCV) infection must have been treated and cured. For individuals with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load

16. Individuals with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial

17. Patients with reproductive potential must agree to use effective contraception and to not donate sperm during the study and for at least 2 months following the last dose of study treatment. Effective method of contraception means male condom with spermicide, female condom with spermicide, diaphragm with spermicide, cervical sponge, or cervical cap with spermicide

Exclusion Criteria:

1. Patients with small cell or neuroendocrine carcinoma histology.
2. Patients with a super scan seen in the baseline bone scan. Super scan refers to a bone scan with diffusely increased skeletal radioisotope uptake relative to soft tissue
3. Patients with prior treatment with CDK4/6 inhibitors
4. Patients with prior treatment with PSMA-targeted radioligand therapy. Patients with previous treatment with PSMA targeting therapies (Such as Chimeric antigen receptor T cells (CAR-T) or Bi-specific T-cell engagers (BiTEs) are eligible.
5. Patients treated with Radium-223 within 6 weeks prior to study entry.
6. Any systemic anti-cancer therapy within 3 weeks of study entry
7. Patients who have experienced significant radiation-related adverse events (AEs) from prior radiation treatment (>= grade 3) or have experienced persistent radiation-related AEs that have not resolved by the time of study randomization
8. Patients with a history of central nervous system (CNS) metastases are ineligible unless they have received prior therapy (surgery, radiation therapy (RT), gamma knife) are asymptomatic, and not receiving corticosteroids for this indication. Head imaging is not required
9. Patients with symptoms of cord compression or impending cord compression
10. Patients with concurrent serious medical conditions as determined by primary investigator
11. Patients with other significant malignancies that are expected to alter life expectancy or interfere with disease assessment. Patients with adequately treated skin cancer, non-muscle-invasive bladder cancer and patients with prior history of malignancy who have been disease free for more than 2 years are eligible. Patients with history of in-situ/early stage melanoma will not be excluded
12. Patients who have not recovered from adverse events due to prior anti-cancer therapy to =< grade 1 or baseline (other than alopecia or peripheral neuropathy)
13. Patients with serious and/or uncontrolled preexisting medical condition(s) that, in the judgment of the investigator, would preclude participation in this study (for example, interstitial lung disease, severe dyspnea at rest or requiring oxygen therapy, history of major surgical resection involving the stomach or small bowel, or preexisting Crohn's disease or ulcerative colitis or a preexisting chronic condition resulting in baseline grade 2 or higher diarrhea)
14. The patient has active systemic bacterial infection (requiring intravenous (IV) antibiotics at time of initiating study treatment), fungal infection, or detectable viral infection (such as known human immunodeficiency virus positivity or with known active hepatitis B or C (for example, hepatitis B surface antigen positive). Screening is not required for enrollment
15. The patient has a personal history of any of the following conditions: syncope of cardiovascular etiology, ventricular arrhythmia of pathological origin (including, but not limited to, ventricular tachycardia and ventricular fibrillation), or sudden cardiac arrest.
16. Patients currently receiving any other investigational therapeutic agents.