Thank you for paying attention to my question.
Basically, our code is based on the following code:

  
      github.com
  
  
    huggingface/transformers/blob/master/examples/run_squad.py 3
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet)."""


import argparse
import glob


  This file has been truncated. show original

  
  
    
    
  
  


Then, in our code, we call the main function of run_squad.py every iteration as follows:
from run_squad import main
...
for _ in range(100):
    main()

(this is not exact code since we slightly modified it that we can call this function iteratively, however, I think you can catch what we try to do by this code snippet)
Since this code loads the training dataset at every iteration like this (line 791),
        train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)

it costs 15~30 seconds at every iteration for loading dataset.
Instead, we try to load the dataset before iteration instead of loading the dataset at every iteration as follows:
from run_squad import main, load_and_cache_examples
...
train_dataset = load_and_cache_examples(...)
for _ in range(100):
    main(train_dataset)

However, it ends up with a stack of “Non-releasable Memory in GPU” and finally the out of memory error occurs because of stacked non-releasable GPU memory.
This situation did not occur if we load the dataset inside every main function call.
I’m sorry for not providing the detailed code. If you need more information on inspecting this problem, please let me know.
Thanks.