I have been dabbling in PyTorch for a while and noticed a weird behaviour, likely because of seeding/instantiating issue.
    # Model, cost function and optimizer instancing
    model = models.CustomNeuralNet().to(device)
    if is_forward_pass:
        # Forward Sanity Check
        _forward_X, _forward_y = models.forward_pass(
            loader=train_loader, model=model
        )

    my_trainer: trainer.Trainer = trainer.Trainer(
        params=TRAIN_PARAMS,
        model=model,
        device=device,
        wandb_run=wandb_run,
    )

    curr_fold_best_checkpoint = my_trainer.fit(
        train_loader, valid_loader, fold
    )

I won’t be able to paste the whole code pipeline as it is pretty long, but it would be good to know why this is the case.
For example, if I set is_forward_pass to False, then my trainer will just start training, and say my loss is 0.1 for epoch 1. As a cautious move, I will always check if this particular loss changes when I modified the function (I value reproducibility a lot, though it is a black box to me). So when I switched on is_forward_pass to True, then my loss changed. Once I turned it off, the loss is the same value so I am sure it is this flag. My hunch is somehow the DataLoader or the Model is being called again? Has anyone faced this issue?