I don’t think @mattmacy is getting linear speed-up, as his epoch time does not change, not time-per-minibatch. It’s more like no speed-up at all (same time per epoch indicates twice-as-big time for twice-as-big minibatch).
There are many scenarios where speed-up from data parallel would not be that great - e.g. OpenNMT example hardly benefits from data parallel.