Hi all,
I came across this post and I’m facing a similar issue with PyTorch where I don’t know what would be the best approach:
  
      
      Machine Learning Mastery – 14 May 17
  
  
    

How to use Different Batch Sizes when Training and Predicting with LSTMs 1

Keras uses fast symbolic mathematical libraries as a backend, such as TensorFlow and Theano. A downside of using these libraries […]

  Est. reading time: 13 minutes
    

  
  
    
    
  
  


I am working on a network that is supposed to run on real-time in CPU during inference, taking a noisy audio chunk of 512 samples as input to produce a clean version of the same audio chunk of 512 samples as output. The network I am using involves LSTM layers that according to the documentation require a known batch size during training of dimensions (seq_len, batch_size, input_size) which in my case would be (1, 1, 512):
https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html 1
I would ideally like to train the network on batches bigger than 1 (e.g. batch_size=32) but use the model during inference with batch_size=1 since it will be running on real time taking single input frames of 512 samples. Also, the model has to be stateful and as far as I understand, this can be implemented by detaching the output hidden and cell states in order to feed them back in as the inicial hidden and cell states of the next batch, is that correct? Along with this one, my other questions would be:
1. Is it possible in PyTorch to train LSTM on batch_size=32 and then use the same saved model to do inference in single steps of batch_size=1? (i.e.  train on (1, 32, 512) and inference in (1, 1, 512) assuming batch_first=False) if so, how? and, may the predictions be affected by choosing a different batch_size during inference?
2. Would it be okay to just do train and inference on batch_size=1 and use the model like that or would it be expected to be less time efficient than (1) during training? Overall, my main concert is to obtain the least CPU intensive model during inference.
Sorry if some questions may have an easy answer, but I couldn’t find much information about similar scenarios. Any guidelines are highly appreciated.
Esteban