I want to take convolutions over the channels instead of pixels. The input shape is “Batch size * Num_Channels * Num_Timesteps * H * W”. There are three subspaces in this task (action recognition); spatial, temporal, and semantic (channels). What should be the number of input and output channels in this regard? In the below code I multiplied other subspaces dimensions (T * H * W) as the number of channels. What should be the shape of the tensor that is going to be passed to “channel_conv1d”?
class Channelwise_1D(Module):
def __init__(self, input_shape, kernel_size):
    super(Channelwise_1D, self).__init__()


    self.kernel_size = kernel_size

    nun_channels = input_shape[1]
    num_timesteps = input_shape[2]
    spatial_window_size_1 = input_shape[3]
    spatial_window_size_2 = input_shape[4]
    n_virtual_channels = num_timesteps * spatial_window_size_1 * spatial_window_size_2

    self.channel_conv1d = Conv1d(n_virtual_channels, n_virtual_channels, kernel_size)

def forward(self, input):
    
    input_shape = input.size()
    BN, C, T, H, W = input_shape
    tensor = input.permute(0, 3, 4, 2, 1)
    tensor = tensor.contiguous()
    tensor = tensor.view(-1, C)

    tensor = self.channel_conv1d(tensor)
    
    return tensor