I work on the similar project. The problem is, pytorch keep gives me error in
    prediction[:,:,:2] += x_y_offset

RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor

edit : already solve it by adding this line in top of def predict_transform
    prediction = prediction.to(torch.device("cuda"))


but right now, i found the same exact problem with you …