I am building torch from source and seeing slow performance when using torch.nn.functional.linear.
OS: Ubuntu 20.04
commit hash: 4d6314a16e78027832186f5442df888dbabbc159 (9/21/21)
export USE_CUDA="0"
python3.9 setup.py bdist_wheel

import datetime
import torch
import torch.nn.functional as F

t1 = torch.ones((32, 1024, 512))
t2 = torch.ones((50257, 512))
e = datetime.datetime.now()
print(f"Before F.linear: hour: {e.hour}, minute: {e.minute}, second: {e.second}", flush=True)
logits_parallel = F.linear(t1, t2)
e = datetime.datetime.now()    
print(f"After F.linear: hour: {e.hour}, minute: {e.minute}, second: {e.second}", flush=True)

torch built from source takes 15m 15s while the latest torch from PyPI takes 17 s. Any ideas on what might be the issue?

Running htop while running the above script shows that only 1 core is getting exercised
I am not building with MKL