@ptrblck I am just trying with a small input length to see what actually is going on.  I am seeing this peculiar behaviour from the binary conv layer.
For example:
The kernels are: [[[[-0.4636,  0.6064]], [[-1.1505, -0.9865]]], [[[-0.1688, -0.3571]], [[-1.7502, -0.6587]]]]
Inputs to the layer: [[[[0., 0., 1., 1.]], [[0., 0., 1., 1.]]]]
The output should be: [[[[0., -0.3801, -1.9952]], [[0., -1.0158, -2.9348]]]]
However, it produces the output: [[[[0., 0., 0.]], [[0., 0., 0.]]]]
I have no clue what is going on.