Tensorflow Adam –   “Momentum decay (beta1) is also applied to the entire momentum
accumulator. This means that the sparse behavior is equivalent to the dense
behavior (in contrast to some momentum implementations which ignore momentum
unless a variable slice was actually used).”
Yeah this will cause performance differences.
But it’s not that pytorch is using a weird Adam optimizer in contrary it’s looks like pytorch just has the plain vanilla version. You guys see the same thing?