Personnally I would do this via a single script file or method that has as input fold index. In this case, there will be no problems with restarting model, optimizer, lr_scheduler and other state-full object (e.g. amp if nvidia/apex is used).
So, a basic code will be something like that:
from ignite.engine import create_supervised_trainer, create_supervised_evaluator, Events
from ignite.metrics import Accuracy


def train_on_single_fold(*args, fold_index=0, **kwargs):

    train_loader, val_loader = get_dataflow(fold_index=fold_index, **kwargs)
    model, optimizer, criterion, lr_scheduler = initialize_model(**kwargs)
    metrics = {
         "accuracy": Accuracy(),
    }

    trainer = create_supervised_trainer(model, optimizer, criterion)

    @trainer.on(Events.EPOCH_COMPLETED)
    def update_lr_scheduler(_):
        lr_scheduler.step()

    evaluator = create_supervised_evaluator(model, metrics=metrics)

    @trainer.on(Events.ITERATION_COMPLETED(every=validate_every))
    def validate(trainer):
        evaluator.run(val_loader)
        metrics = evaluator.state.metrics
        print("After {} iterations, binary accuracy = {:.2f}"
              .format(trainer.state.iteration, metrics['accuracy']))

    trainer.run(train_loader, max_epochs=kwargs.get(max_epochs, 100))

Hope this helps. Please, feel free to ask more about ignite usage if needed (and do not forget to set the category “ignite” otherwise our team won’t get the notification).
PS: @ptrblck thanks a lot for mentioning !