jekbradbury:

CUDNN actually supports this internally but PyTorch does not yet have a wrapper (expect one fairly soon).


Looking forward to it very much.



 jekbradbury:

BTW, are there benchmarks on TF Fold? I can’t imagine their repeated concatenations+splits are all that much faster than they’d be in PyTorch.


Also wondering whether this strategy is faster than padding + masking solution.
For now, I will use the old-fashion padding (maybe with masking) approach.