This phase I trial studies the side effects and best dose of ipilimumab when given together with whole brain radiation therapy or stereotactic radiosurgery in treating patients with melanoma with brain metastases. Monoclonal antibodies, such as ipilimumab, can block tumor growth in different ways. Some block the ability of the tumor to grow and spread. Others find Tumor cells and help kill them or carry tumor-killing substances to them. Radiation therapy, such uses high-energy x-rays and other types of radiation to kill tumor cells. Giving radiation therapy in different ways may kill more tumor cells. Giving ipilimumab together with whole-brain radiation therapy or stereotactic radiosurgery may kill more tumor cells.