Pembrolizumab, Endocrine Therapy, and Palbociclib in Treating Postmenopausal Patients With Newly Diagnosed Metastatic Stage IV Estrogen Receptor Positive Breast Cancer

Inclusion Criteria

• •Willing and able to provide written informed consent/assent for the trial
• •Men or women \>= 18 years of age on day of signing informed consent
• •Willing and able to comply with all aspects of the treatment protocol
• •Postmenopausal patients defined by at least one of the following criteria:
• •* Prior bilateral oophorectomy OR amenorrheic for \>= 12 months (if =\< 55 years of age and prior to chemotherapy, or on medical ovarian ablative therapy OR
• •* Previous hysterectomy with one or both ovaries left in place (previous hysterectomy in which documentation of bilateral oophorectomy is unavailable AND follicle stimulating hormone \[FSH\] values consistent with the institutional normal values for the post-menopausal state; FSH levels must be obtained within 28 days prior to registration)
• •Presence of measurable disease meeting the following criteria: at least 1 lesion of \> 10 mm in long axis diameter for non-lymph nodes or \> 15 mm in short axis diameter for lymph nodes that is serially measurable according to RECIST version 1.1 using computerized tomography, magnetic resonance imaging, or panoramic and close-up color photography
• •Stage IV metastatic ER+HER2- breast cancer histologically proven per current American Society of Clinical Oncology (ASCO)/College of American Pathologists (CAP) guidelines; allow up to 30 days prior use of CDK4/6 inhibitors and up to 60 days of letrozole or other aromatase inhibitors for treatment of metastatic ER+ breast cancer
• •Life expectancy of \>= 3 months
• •Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0-2
• •Willing to provide tissue from a newly obtained core or excisional biopsy of a tumor lesion; newly-obtained is defined as a specimen obtained up-to 6 weeks (42 days) prior to initiation of treatment on day 1 and day -28 for cohort 3; subjects for whom newly-obtained samples cannot be provided (e.g. inaccessible or subject safety concern) may submit an archived specimen only upon agreement from the study principal investigator (PI)
• •For Cohort 3, willing to undergo tumor biopsies at baseline (within 6 weeks of study onset), cycle 2 day 1 (C2D1) (+/-1 week) and at time-of-progression or end-of-treatment when feasible
• •Absolute neutrophil count (ANC) \>= 1,000 /mcL (performed within 10 days of treatment initiation)
• •Platelets \>= 100,000 /mcL (performed within 10 days of treatment initiation)
• •Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L without transfusion or erythropoietin (EPO) dependency (within 7 days of assessment) (performed within 10 days of treatment initiation)
• •Serum creatinine =\< 1.5 X upper limit of normal (ULN) OR measured or calculated creatinine clearance (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) \>= 60 mL/min for subject with creatinine levels \> 1.5 x institutional ULN (performed within 10 days of treatment initiation)
• •* Creatinine clearance should be calculated per institutional standard
• •Serum total bilirubin =\< 1.5 X ULN OR direct bilirubin =\< ULN for subjects with total bilirubin levels \> 1.5 ULN (performed within 10 days of treatment initiation)
• •Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 X ULN OR =\< 5 X ULN for subjects with liver metastases (performed within 10 days of treatment initiation)
• •Albumin \>= 2.5 mg/dL (performed within 10 days of treatment initiation)
• •International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants
• •Activated partial thromboplastin Time (aPTT) =\< 1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants
• •Female subjects of childbearing potential should have a negative urine or serum pregnancy within 72 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required
• •Female subjects of childbearing potential should be willing to use two methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication; subjects of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year
• •* Cohort 1 (accrual to 6 patients) is for patients who had ongoing stable disease (SD) on letrozole + palbociclib, enrolled on prior version of eligibility criteria to receive pembrolizumab after obtaining stable disease on letrozole + palbociclib; these patients must have been on treatment with letrozole and palbociclib for 6 months with SD per RECIST 1.1; received up to 3 lies of previous therapy including endocrine and/or chemotherapy in advanced setting prior to initiation of letrozole and palbociclib; no grade 3 toxicities except alopecia