wizardk:

For padding, you can sort the samples by length and do a local random. That will let the total padding size keep smaller. And maybe you will get faster convergence as well.


Yeah, I had read a few ideas about doing that. Although I would assume some randomness would have to be added to prevent the same batch from being presented every epoch.
I’m curious if you have any suggestions about how to do the padding when going through a CNN, instead of a RNN, so that the padded samples aren’t calculated.