This phase II trial studies how well pembrolizumab works in treating patients with prostate cancer that has spread to other places in the body and keeps growing even when the amount of testosterone in the body is reduced to very low levels despite previous treatment with enzalutamide. Monoclonal antibodies, such as pembrolizumab, may block tumor growth in different ways by targeting certain cells.