Hi, all!
I am new to Pytorch and I meet a strange problem while training a my model with GPU. The GPU memory use increase gradually which training and will finally be stable. At the beginning, it will consume about 4G GPU memory, and will increase to around 7G. This happens in the first epoch and the memory use will be stable.
Meanwhile, the training speed will unacceptably slow down after every epoch.  In four epochs, it costs 1000, 2000, 3500, 5200 ms per batch, respectively. Here is the code snippet:

def train_epoch(model, data_train, optimizer, criterion_cls, args):
model.train()
total_loss = 0
cnt = 0
label_list = list()
pred_list = list()
start_time = time.time()
for batch, batch_data in enumerate(data_train):
    data, targets= batch_data
    padding = torch.zeros(data.size(0), args.window_size-1).long()
    data = torch.cat([padding, data],dim=1)
    cnt += targets.size(0)
    label = get_label(targets)
    if args.cuda:
        data = data.cuda()
        label = label.cuda()

    output = model(data)

    loss = criterion_cls(output, label)
    total_loss += loss.item()

    optimizer.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), args.clip)
    optimizer.step()

    prediction = torch.max(output, 1)[1]
    label_list.extend(label.tolist())
    pred_list.extend(prediction.tolist())

    if (batch+1) % args.log_interval == 0:

        p = precision_score(label_list, pred_list, average='weighted')
        r = recall_score(label_list, pred_list, average='weighted')
        f = f1_score(label_list, pred_list, average='weighted')
        acc = accuracy_score(label_list, pred_list)

        elapsed = time.time() - start_time
        train_loss, acc, p, r, f = total_loss/cnt, acc, p, r, f

        cnt = 0
        total_loss = 0
        label_list.clear()
        pred_list.clear()
        start_time = time.time()


The evaluation code is similar and with torch.no_grad() is also used. I guess the problem is not related to the specific model, because I encounter this issue with different models (CNN, RNN, etc).
Could someone help me out? Thanks a lot!!