My training + cross-validation loop usually looks something like:
for epoch in range(max_epoch):
    print('Epoch:',epoch)

    # TRAINING
    mean_err = 0
    net.train()
    for train_idx, train_data in enumerate(train_dataloader):
        # wrap train_data in Variables; move to GPU
        # zero gradients
        # do forward/backward propagation stuff
        # update mean_err
    print('Mean Training Error:', mean_err)
    
    # CROSS-VALIDATION
    mean_err = 0
    net.eval()
    for cv_idx, cv_data in enumerate(cv_dataloader):
        # wrap cv_data in Variables; move to GPU
        # do forward propagation stuff
        # update mean_err
    print('Mean Cross-Validation Error:', mean_err)

What I often find is that there is a particular batch size in which the training loop runs just fine, but then the GPU immediately runs out of memory and crashes in the cross-validation loop.  If I step down the batch size, the system becomes stable, and will run indefinitely through many epochs.  What’s going on?  The inference-only cross-validation should be less memory intensive than the training, so obviously not all the memory used during the training loop is being freed.  How do I fix this so that I can run larger batches?