The input is actually like a video in which 4000 is the number of frames, spatial window size is 200 for both horizontal and vertical and 20 is the number of channels. I am using two Tesla V100 PCIe 16GB GPUs.
The model architeture is a stack of two of these layers:
1DTemporalConv-1DChannelconv-2DSpatialConv-2DSpatialMaxPooling
I also tried to test the network without training and the same problem happens during testing.