One other thing that I was looking at is the way I tried to implement mini-batch processing.  TBH, I don’t see how it would be causing the resume to work incorrectly.
Here is my line for the DataLoader:
    dataloader = DataLoader(text_data,
                            batch_size=parms.MINI_BATCH,
                            shuffle=True,
                            num_workers=3)

Later, in the training I use slices to pass in a MINI_BATCH of records at a time.
...
    for epoch in EPOCH_RANGE:

        current_start = 0 #This keeps track of the current starting row in the mini
        total_loss = torch.Tensor([0])

        keep_going = True
        while keep_going:
            if current_start + parms.MINI_BATCH < len(text_data.target_ids):
                minibatchids = slice(current_start, current_start + parms.MINI_BATCH -1)
            else:
                minibatchids = slice(current_start, len(text_data.target_ids))
                keep_going = False

            model.zero_grad()
            log_probs = model(torch.Tensor(text_data.context_ids[minibatchids]).long()) # This passes in a slice of instances from the data.
            loss = loss_function(log_probs,
                                 torch.autograd.Variable(
                                     torch.squeeze(
                                        torch.Tensor(text_data.target_ids[minibatchids]).long()
                                        )
                                     )
                                 ) * parms.MINI_BATCH
            loss.backward()
            optimizer.step()
...


This SEEMs to work and I get a ten times improvement in throughput, but maybe this is part of the problem?  It shouldn’t be as its not the weights.  Maybe there is a better way to implement this in a more PyTorch fashion.
Since I didn’t document it earlier, I am running:

Linux 16.04 LTS
Python 3.6
PyTorch 0.40 (CPU bound as no eligible GPU)

Any help would be greatly appreciated