mhtrinh:

Is this a crazy idea ?


I don’t think any idea is too crazy to run some experiments. 
Note that the returned indices from this pooling layer are “detached”, i.e. they do not have a grad_fn and thus Autograd won’t backpropagate through them, so you should keep this information in mind when experimenting with it.