Will it still accumulate gradients into self.module.weight or only produce gradInput?

Only the inputs given. But you can give model.parameters() as well 

Also, is there a way to bypass clone() / input copy?

I’m not 100% sure why you need to clone here, I think detaching and requiring gradient again would work.