This randomized phase I trial studies how well anti-SEMA4D monoclonal antibody VX15/2503 with or without ipilimumab or nivolumab work in treating patients with stage I-III pancreatic cancer that can be removed by surgery or stage IV colorectal cancer that has spread to the liver and can be removed by surgery. Monoclonal antibodies, such as anti-SEMA4D monoclonal antibody VX15/2503, ipilimumab, and nivolumab, may interfere with the ability of tumor cells to grow and spread.