Hi,
torch::NoGradGuard does not contain any mutex, only a thread local variable.
There are no mutexes that I know of in any of the regular layers.
You might want to control the number of threads used by open mp (corresponding to this 14 python api). If you do large enough ops, they will parallelize internally and might be trying to use too many ressources.