Hi @ptrblck,
I’m suspecting that my model has a bottleneck in loading data, is below the correct way to time the operations?
for epoch in range(n_epoch):
    epoch_tic = time.time()
    for x_batch, y_batch in training_loader:
        batch_load_tic = time.time()
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        batch_load_toc = time.time()

        nnet = nnet.train()
        yhat = nnet(x_batch)
        loss = loss_fn(yhat, y_batch)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        update_toc = time.time()
    epoch_toc = time.time()

Because I’ve been training models on a GPU but lately the jobs became mysteriously slow for some reason. I wanted to see which part of the training process is taking longer than expected.
What I observe is
epoch_toc - epoch_tic = 45 min
batch_load_toc - batch_load_tic = 0.005s
update_toc - batch_load_toc = 0.0423s

One epoch in my case contains 250 batches, but (0.005+0.0423)*250 is only about 12 minutes. So I’m wondering where are the remaining 45-12=33 minutes and is there a way to time it?
Thanks!