Well everything depends on if you are interested on frequency info or temporal info.
In short, if you look at CNN design through time, at the beggining people used to flatten features and send then to a linear layer.
Nowadays the common way is to use a pooling to get a 1D array and send that to a linear layer.
Note that you are not interested in keeping the spatial features. So imagine your output is 256x8x8.
If you have the speech at the beginning of the track, the elements with useful info would be the [:,:,:]4 ones in the left.
With max pooling the linear layer faces a simpler task.