Veliparib and Pegylated Liposomal Doxorubicin Hydrochloride in Treating Patients With Recurrent Ovarian Cancer, Fallopian Tube Cancer, or Primary Peritoneal Cancer or Metastatic Breast Cancer

Inclusion Criteria

• •Histologically confirmed diagnosis of recurrent or residual epithelial ovarian cancer, primary peritoneal carcinoma or fallopian tube carcinoma, OR histologically confirmed metastatic breast cancer, that is estrogen receptor (ER)-negative, progesterone receptor (PR)-negative, and human epidermal growth factor receptor 2 (HER2)/neu negative (as determined by local pathology laboratory)
• •Prior chemotherapy:
• •* Breast cancer: patients may have received 0-2 prior chemotherapy regimens for metastatic disease; breast cancer patients may not have received prior PLD, and will not be eligible for the expanded cohort A
• •* Interval between prior chemotherapy and registration for breast and ovarian cancer; there should be at least a 3 week interval between the last chemotherapy regimen and registration, and the patient should have recovered from acute toxicity related to prior therapy (6 weeks if the last regiment included BCNU or mitomycin C)
• •* Dose escalating cohorts only: patients will be categorized in the following strata based upon prior PLD exposure: Stratum A -patients with ovarian cancer who have had prior PLD exposure and received at least 3 cycles of PLD without prohibitive (i.e. no grade 3 or 4 skin toxicity) and have not had progressive disease; Stratum B: patients with ovarian or breast cancer who have had no prior PLD exposure
• •Eastern Cooperative Oncology Group (ECOG) performance score 0-2
• •All potential subjects should be evaluated for whether breast cancer (BRCA)1-2 testing is medically appropriate; individuals who have a 10% or higher risk of having a BRCA1-2 mutation (Myriad tables at www.myriad.com) are encouraged (but not required) to have mutation testing and results known; information regarding mutation status (positive \[including specific mutation\], negative, or unknown) and projected risk of having a mutation (as determined by Myriad tables) will be collected at the time of diagnosis
• •Non-measurable and/or measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) criteria, or abnormal cancer antigen (CA)-125 to levels (in patients with ovarian cancer) at least 1.5 x normal documented by two independent measurements at least 4 weeks apart
• •Ability to give voluntary informed consent and to comply with treatment and required tests
• •Ability to tolerate oral medications
• •Female subjects age \>= 18 years (males with breast cancer are eligible)
• •Absolute neutrophil count \>= 1500/mL
• •Platelets \>= 100,000/mL
• •Creatinine =\< 1.5 mg/dL
• •Total bilirubin =\< 1.5 x institutional upper limit of normal
• •Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 3 x institutional upper limit of normal (=\< 5 x institutional upper limit of normal if evidence of liver metastasis)
• •Left ventricular ejection fraction at or above institutional lower limit of normal (obtained within 8 weeks of registration by multigated acquisition \[MUGA\] scan or echocardiogram; the same test performed at baseline should be repeated after every 3 cycles of therapy)
• •Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately