Are you using DDP?
If so, the slower card might sync the faster one.
Or are you profiling the cards separately? If so, your code might have other bottlenecks (e.g. data loading). Have you profiled it?