How many frames did you use in the former approach?
The memory usage depends also on your model architecture.
E.g. if you are using an RNN, trading the sequence length for the batch size might not result in the same memory usage.