The code for visualization should generally work:
output = torch.randn(1, 2, 224, 224)
pred = torch.argmax(output.squeeze(), 0).numpy()
seg_output = decode_segmap(pred)

plt.imshow(seg_output)

This code snippet gives a random segmentation output.
Did you observe your training and validation loss and accuracy?
Your model might be overfitting to a single class, but that should already be visible during training in your validation loss.