I am trying to write a custom CNN layer that applies softmax to each convolution operation. So each pixel in the output image is gonna be valued between [0, 1] and it is the sum of the convolved pixel. An example of TensorFlow implementation can be seen here 1.
Ideally, this should be trained with binary cross-entropy loss. I tried below but it does not train.
class TransitionModel(nn.Module):
    def __init__(self):
        super(TransitionModel, self).__init__()

        self.weight = nn.Parameter(torch.Tensor(1, 1, 3, 3))
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        self.softmax = nn.Softmax(dim=1)

    def forward(self, s, test=False):
        return F.conv2d(s, self.softmax(self.weight), padding=1)

model = TransitionModel()
pred_s_p = model(s)
pred_s_p = pred_s_p.squeeze()
loss = F.binary_cross_entropy(pred_s_p, s_p)

s.shape = [1, 1, 32, 32]
pred_s_p.shape = [32, 32]
It throws the following error:
RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered