Am using Trainer.predict but have noticed that it’s actually taking twice as long as displayed by the progress bar.
print('Predicting on test samples...')
t0 = time.time()
predictions, label_ids, _ = trainer.predict(tokenized_datasets['test'])
print(f'completed in {(time.time() - t0) / 60:.2f} mins.')
print('Argmaxing...')
t0 = time.time()
predictions = predictions.argmax(axis=2)
print(f'completed in {(time.time() - t0) / 60:.2f} mins.')


trainer_predict_timing1040×226 11.9 KB

During the period shown by the progress bar, the GPU is used, but it’s not used after the progress bar has completed.  Different sizes of tokenized_datasets['test'] has been tried, and it appears that the trend is that it takes twice as long regardless of the size.
Is this normal, or expected?