In my custom dataset, the data is loaded from Pandas. There is augmentation by creating a new item based on a simple rule:
def augmentation_by_color(x, colors):
    x = np.array(x)
    return np.array([colors[xi] for xi in x.flatten()]).reshape(x.shape)
colors = [0,1,2,3,4,5,6,7,8,9]
colors = np.random.permutation(colors)
new_data = (dataitem_from_pandas, colors)
# new_data to tensor and return for training on CUDA

Everything works on CPU and then converted to tensor that will be processed on CUDA. How to speed up this process? Should I do all operations on tensors (convert Pandas item to tensor and the do permutations there)? If so can you suggest a code?
If loading data takes most of time then should I increase the num_workers?