Testing the Addition of Immunotherapy With Hu5F9-G4 (Magrolimab) to the Usual PARP Inhibitor, Olaparib for Treatment of Metastatic or Recurrent Breast or Castrate-Resistant Prostate Cancer With BRCA Mutations

PRIMARY OBJECTIVES: I. To evaluate the safety, tolerability, toxicity profile (dose-limiting toxicities \[DLTs\], maximum tolerated doses \[MTDs\]), and phase 2 recommended dose (P2RD) of Hu5F9-G4 (magrolimab) combined with olaparib. II. To evaluate the alteration in the immune microenvironment after combination therapy including alteration of specific immune signatures and pathways related to innate immune response and STING pathway. (Dose expansion) SECONDARY OBJECTIVES: I. To evaluate the status of homologous recombination repair (HR)/DNA damage response (DDR) and other genomic mutations. II. To characterize the pharmacokinetic (PK) profile of olaparib and magrolimab in combination. EXPLORATORY OBJECTIVES: I. To explore the activity of the combination regimen in entire cohort and in the dose expansion cohorts in term of objective response rate (ORR), and progression-free survival (PFS). II. To evaluate the peripheral immune profile. III. To evaluate alterations in the HR/DDR pathway by circulating tumor (ct)DNA. IV. To correlate drug exposure with response and/or toxicity. OUTLINE: This is a dose-escalation study of magrolimab in combination with fixed-dose Olaparib followed by a dose-expansion study. Patients receive magrolimab intravenously (IV) on days 1, 8, 15 and 22 of cycle 1 and days 1 and 15 of subsequent cycles. Patients also receive olaparib orally (PO) twice a day (BID) during each cycle. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients also undergo collection of blood samples and computed tomography (CT) and/or magnetic resonance imaging (MRI) throughout the trial. Patients in the dose-expansion portion of the study also undergo tumor biopsies during screening and on study. Patients are followed for 30 days after removal from study treatment, and then followed every 12 months for 36 months or until death whichever comes first.