This phase I/II trial studies the best dose and side effects of glutaminase inhibitor CB-839 and how well it works with panitumumab and irinotecan hydrochloride in treating patients with RAS wildtype colorectal cancer that has spread to other places in the body and does not respond to treatment. Glutaminase inhibitor CB-839 may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Monoclonal antibodies, such as panitumumab, may interfere with the ability of tumor cells to grow and spread. Drugs used in chemotherapy, such as irinotecan hydrochloride, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving glutaminase inhibitor CB-839 with panitumumab and irinotecan hydrochloride may work better in treating patients with colorectal cancer.