Just to be sure that you did a fair bemchmark: did you set all seeds, set cudnn deterministic mode and disabled its benchmark mode? Different seeds can mean differently shuffled batches, meaning which might be more efficient in some cases vs. others (depending on how the padding is done). Something like this:
def set_seed(seed: int):
    """Set all seeds to make results reproducible (deterministic mode).
       When seed is None, disables deterministic mode.
    :param seed: an integer to your choosing
    """
    if seed is not None:
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        np.random.seed(seed)
        random.seed(seed)
        os.environ['PYTHONHASHSEED'] = str(seed)