Hey,
I already commented on this issue on GitHub but got some new ideas today which might be worth to check out:

Check if the loss is correct (correct sign +1, -1)
Use Deconvolution instead of MaxPool

For 2. you could take a look at https://github.com/meetshah1995/pytorch-semseg/blob/master/ptsemseg/models/utils.py#L94-L108 179
Good luck!