In [256, 64, 28, 28] the 256 is the batch size. So each input tensor to the first layer in self.layer_attend1 has the shape [64, 28, 28]. Of this, the 64 is the number of channels in the input. The first argument to nn.Conv2d should therefore be 64.