Hey Tae and @el_samou_samou … This is my bad. SN with DP currently is broken. They don’t work on training or eval mode. A fix is in the works. But a workaround is:

use this DP:

class DataParallel(nn.parallel.DataParallel):
    def replicate(self, module, device_ids):
        replicas = super(DataParallel, self).replicate(module, device_ids)
        replicas[0] = module
        return replicas


use this SN: https://gist.github.com/SsnL/8e638bcfd49e71d6b1930db0df87d970 5
Note that only line 56 changed.