Hello,
I have been trying to make a CPU accelerated inference for our model. The only thing I change from GPU to CPU is torch.set_num_threads(threads). I am running the model on AWS m5.24xlarge with 96 CPUs. I set the num_threads to 86 and num_workers to 8. It starts pretty well, giving an ETA of ~16hours but after 40% completion, most of the pytorch processes start jumping between “S-R” (stall and running) states. It got stuck at 47%, and I had to kill it.
Should I do something else for a CPU-accelerated version?
This is on Ubuntu 18.04 (the deeplearning AMI available from AWS)
Python3: 3.6.7
PyTorch: 1.1
The model is here: https://github.com/kishwarshafin/helen/blob/master/modules/python/models/TransducerModel.py
And the way I am doing is here:

  
      github.com
  
  
    kishwarshafin/helen/blob/master/modules/python/models/predict.py
import sys
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from modules.python.models.dataloader_predict import SequenceDataset
from modules.python.TextColor import TextColor
from tqdm import tqdm
import numpy as np
from modules.python.models.ModelHander import ModelHandler
from modules.python.Options import ImageSizeOptions, TrainOptions
from modules.python.DataStore import DataStore
"""
This script implements the predict method that is used by the call consensus method.

The algorithm is described here:

  1) INPUTS:
    - directory path to the image files generated by MarginPolish
    - model path directing to a trained model
    - batch size for minibatch prediction


  This file has been truncated. show original

  
  
    
    
  
  


Any help would be highly appreciated.