@ptrblck, Thank you for your time.
If you add Dropout layer after BatchNorm1d i.e. (Conv1d --> ReLU --> BatchNorm1d --> Dropout), it works like charm.
I have also checked PReLU as activation function (Conv1d --> PReLU --> BatchNorm1d), which again works perfectly, only ReLU is causing problem. I have successfully trained model with this setup.
I hope this might help you to debug further.