Hi everyone,
I want to solstice some useful advice on how to debug a strange model behavior in my training. Recently, I have taken up a ML project, and in an effort to speed up training time, I converted the original training script’s Data parallel with torch’s distributed data-parallel with Nvidia Apex. My new model will run, however, I noted that no matter how long I trained it, the results are never any better than when it started, unlike the original implementation which was able to get good results.
The only thing peculiar about this model is that it used 2 separate Adam optimizers, one to optimize the layer’s parameters, and the other is used to optimize a user-defined parameters with torch grad on.
I am not quite sure why this is happening. Any advice is appreciated.