Thanks for the information.
I’ve found this in the TF docs:

For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since AdamOptimizer uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the “epsilon” referred to here is “epsilon hat” in the paper.

I didn’t have a chance to look into the formulas closely and I’m not sure, if the TF docs are applicable to the PyTorch optimizer.
On a general question: were you able to use the default Adam optimizer and get “stable” results?