Thanks a lot @A_A for replying. So  basically what you are saying is that any kind of convolutions are sufficient?.
I was actually going for some kind of a multiScale CNN architechture where I would etract features from the temporal as well the fourier domain and combine them later. Is that kind of redundant ?