The number of dimensions look alright. However the spatial size is different.
While your data is [504 x 504], your target is [584 x 565], which will throw this error.
Also, be careful about using TF.to_tensor on your mask, as this might normalize the class indices into float values in the range [0, 1].