To test the safety and effectiveness of the combination of PROSTVAC and nivolumab. To test this for people with castration resistant prostate cancer and then for other people with localized prostate cancer who are candidates for surgical removal of the prostate.