Thank you! That solved that issue but now I am getting the following issue 
segment = mask*curr_color # should have shape 1, 3, 100, 100
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Did one of mask or curr_color get implicitly allocated on the CPU? I thought everything gets allocated on the GPU by default.