Phase I/II Study of Immunotherapy Combination BN-Brachyury Vaccine, M7824, N-803 and Epacadostat (QuEST1)

Background: * Programmed cell death protein 1 (PD-1)/programmed death ligand 1 (PD-L1) signaling appears to be a major inhibitor of activated T cell anti-tumor immune responses. The rapid, deep and durable responses seen in various malignancies with PD-1/PD-L1 targeted agents demonstrate that blockade of this axis is key to facilitating immune responses within the tumor microenvironment (TME). * Prostate cancer is poorly recognized by T cells. Lack of an immune response is one explanation for the lower response rates (\<15%) observed with anti-PD-1/PD-L1 therapies for prostate cancer. * Increasing response rates will likely require therapeutic nullification of multiple immune deficits by combining immunotherapies that generate tumor-specific T cells (vaccine), dampen the inhibitory milieu of the TME, and enhance T and natural killer (NK) cell activity within the TME. * A quick efficacy seeking trial, utilizing sequential arms offers a means to identify signals of activity for combinations of immunotherapy, added sequentially, in castration resistant prostate cancer (CRPC) participants. * Bavarian Nordic (BN)-Brachyury is a novel recombinant vector-based therapeutic cancer vaccine designed to induce an enhanced immune response against brachyury, which is overexpressed in many solid tumor types, including prostate adenocarcinoma. BN-Brachyury collectively refers to the priming doses (modified vaccinia Ankara (MVA)-BN-Brachyury) and the boost doses (fowlpox virus (FPV)-Brachyury) of the vaccine platform. * bintrafusp alfa (M7824) is a bifunctional fusion protein consisting of an anti-programmed death ligand 1 (PD-L1) antibody and the extracellular domain of transforming growth factor beta (TGF- beta) receptor type 2, a TGF-beta trap. M7824 can also mediate antibody-dependent cellular cytotoxicity in vitro. * Anktiva (N-803) is an interleukin 15 (IL-15)/IL-15R alpha superagonist complex that can enhance NK cell mediated antibody-dependent cell-mediated cytotoxicity (ADCC) and T-cell cytotoxicity. * Synergistic anti-tumor effects have been observed in vitro when combining M7824 and N-803, and in vivo when combining these agents with tumor vaccine in animal models. * Indoleamine 2,3-dioxygenase 1 (IDO1) is overexpressed in many solid tumors and can contribute to immune escape by tumor cells. INCB024360 (Epacadostat) is an IDO1 inhibitor under investigation in combination with different immunotherapies in treatment of various malignancies. * In treating of CRPC, we hypothesize that these agents and their effects will be complementary. Tumor-specific T cells generated by vaccine may become more functional in a TME following treatment with M7824 and Epacadostat. N-803 can further enhance the activity of antigen-specific T cells as well as NK cells. Objective: -To determine if there is clinical benefit to any of a set of 3 possible treatments for participants with CRPC: * BN-Brachyury + M7824 * BN-Brachyury + M7824 + N-803 * BN-Brachyury + M7824 + N-803 + Epacadostat Eligibility: * Adults with histologically proven CRPC, or metastatic solid tumor of any type for which there is no standard treatment or standard treatment has failed. * Adequate organ function as defined by liver, kidney, and hematologic laboratory testing. * Participants with acquired immune defects, active systemic autoimmune disease, history of organ transplant, history of chronic infections, or history of active inflammatory bowel disease are excluded. Design: Open label Phase I/II trial with following randomization during the expansion. Phase I: Cohort 1, Arm 1.1 -Up to 18 participants with any solid tumor will be enrolled in dose escalation Cohort 1 for treatment in Arm 1.1 (flat dose of M7824 + different dose levels of N-803). Phase IIA: expansion with sequential enrollment into Cohort 2A, Arms 2.1A, 2.2A and 2.3 A * Concurrently with the enrollment to Arm 1.1, 13 participants with CRPC will start enrollment in Cohort 2A for treatment in Arm 2.1A (M7824 + BN-Brachyury). * When safe dosing of N-803 is identified during Phase I, 13 participants have enrolled in arm 2.1A and the first 6 participants, treated in Arm 2.1A, have met safety requirements, 13 participants with CRPC will start enrollment in Cohort 2A for treatment in Arm 2.2A (M7824 + BN-Brachyury + N-803). * When 13 participants have enrolled in Arm 2.2A and the first 6 participants, treated in Arm 2.2A, have met safety requirements, 13 participants with CRPC will start enrollment in Cohort 2A for treatment in Arm 2.3A (M7824 + BN-Brachyury + N-803 + Epacadostat). Phase IIB: expansion with randomized enrollment into Cohorts 2D and 2R, Arms 2.1B, 2.2B. and 2.3B * Each Arm in Cohorts 2D and 2R: 2.1B, 2.2B and 2.3B will be open for additional enrollment (25 evaluable participants total) when the initial 13 participants have accrued, safety requirements are meet and a positive signal (defined as Objective Response by Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 or sustained prostate-specific antigen (PSA) decrease \>= 30% sustained for \> 21 days) in \>= 2 participants is shown. * If only one arm is open for additional enrollment, participants will be directly assigned to this arm. If 2 arms are open for additional enrollment, participants will be randomized between these 2 open arms. If 3 arms are open for additional enrollment, participants will be randomized among these 3 open arms. If there are \>= 6 of 25 participants with a positive signal of activity in any expansion arm, that arm will be considered of interest for future studies.