This is a Phase 1, dose escalation trial evaluating the tolerability, pharmacokinetics, and pharmacodynamics of ABT-767 in subjects with advanced BRCA1 or BRCA2-mutated solid tumors and high grade serous ovarian, fallopian tube, or primary peritoneal cancer. ABT-767 is a potent oral inhibitor of the enzymes poly (ADP-ribose) polymerase 1 and 2 (PARP-1 and PARP-2). Malignancies with deficiencies in homologous repair, such as BRCA-1 and BRCA-2 deficient tumors, are more dependent on PARP for deoxyribonucleic acid (DNA) repair than normal cells and, thus, are thought to be more sensitive to PARP inhibition. The study design is a single-arm dose escalation study to determine dose-limiting toxicities, maximum tolerated dose and the recommended Phase 2 dose (RPTD) of orally administered ABT-767 in subjects with BRCA mutations and malignancies. In order to further evaluate the safety and tolerability of ABT-767 at the RPTD, 20 additional subjects will be enrolled in an expanded safety cohort consisting of BRCA1- or BRCA2-mutated Breast cancer and Ovarian cancer.