Hum I see.
For example, if my kernel sizes are kernel_size1 = 5, kernel_size2 = 4, kernel_size3 = 4, kernel_size4 = 5, kernel_size5 = 4, the output is correct because the difference between kernels is not big.
Top: predicted, Bottom: expected
P (20) 00000000111110000000 (5)
E (20) 00000000111110000000 (5)

P (20) 00000000000001111000 (4)
E (20) 00000000000001111000 (4)

P (20) 01111000000000000000 (4)
E (20) 01111000000000000000 (4)

P (20) 11111000000000000000 (5)
E (20) 11111000000000000000 (5)

P (20) 00000000000000000011 (2)
E (20) 00000000000000000011 (2)

If I replace some kernels with larger values, some results are correct but not all
kernel_size1 = 5, kernel_size2 = 4, kernel_size3 = 7, kernel_size4 = 6, kernel_size5 = 4
// Here
P (20) 00000111111111100000 (10)
E (20) 00000000111111111100 (10)

P (20) 00000000000001111000 (4)
E (20) 00000000000001111000 (4)

// Here
P (20) 11111110000000000000 (7)
E (20) 01111111000000000000 (7)

// Here
P (20) 11111000000000000000 (5)
E (20) 11111100000000000000 (6)

P (20) 00000000000000000011 (2)
E (20) 00000000000000000011 (2)

So there are four possibilities for the results being wrong:

The select after the convolution [:, :, :inputs.size(-1)]

Padding of the kernel
Padding of the conv
Conv padding should be dynamic

I’ve been working on 2, 3 a lot but couldn’t find a solution which works.