This is using the same pytorch version, right?
Did you try windows GPU? How does that compare against ubuntu GPU? What about windows CPU vs ubuntu CPU?
Also, things like cudnn and mkldnn matter, as they provide a lot of the conv algorithms used.