This phase I/II trial studies the side effects and best dose of olaparib when give together with durvalumab and tremelimumab and to see how well they work in treating patients with ovarian, fallopian tube, or primary peritoneal cancer with BRCA1 or BRCA2 genetic mutation that has come back or has not responded to treatment. Drugs, such as olaparib, may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth and kill tumors cells with BRCA1 or BRCA2 mutation. Monoclonal antibodies, such as durvalumab and tremelimumab, may help stimulate the immune system in different ways to attack and stop tumor cells from growing. Giving olaparib with durvalumab and tremelimumab may work better in treating patients with ovarian, fallopian tube, or primary peritoneal cancer.