If I understand correct, the rows of a csv file is a temporal sequence representing the audio’s information at each time (interval). If that is the case, I recommend you look into recurrent neural networks (RNN). RNNs are specifically designed to handle ordered sequential data.
The PyTorch NLP tutorial covers a lot of knowledge on RNN: http://pytorch.org/tutorials/beginner/deep_learning_nlp_tutorial.html 18. While it uses language tasks as examples, many techniques apply to audio sequences.