cc autograd expert @albanD for autograd and Hessian vector product questions 

I’m also wondering how much difference is there between DDP and DataParallel regarding training speedup.

This depends on things like model sizes, number of GPUs, GPU interconnects, etc. DDP can avoid replicating models in every iteration and avoid GIL contention. These overhead might weigh differently in different applications.