Yeah you’re right @iffiX, I should mention here what I actually want to do.
I am trying to figure out a way to parallelize over multiple GPUs on non-neural net computations in PyTorch. More specifically, I have developed an estimator (non-neural net) in Scikit-learn and the speed performance is slow when a certain hyperparameter is increased. To solve this, I am re-writing the estimator in PyTorch so that I can make use of GPU processing and hopefully multiple GPUs as well.
I have posted a question here in PyTorch Forums (1-2 days ago), Stackoverflow, PyTorch Github and multiple channels in Reddit and I haven’t gotten a reply yet or no one seems to know the full solution. In fact, what I got is the opposite where a lot of folks over at Reddit wanted to know how this is done too. So far, I feel like you’re the only one who seems to know 
Anyhow, I did initially tried using torch.multiprocessing.pool. The MRE code is as below:
import numpy as np
import torch
from torch.multiprocessing import Pool, set_start_method

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X)

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power

if __name__ == '__main__':
  set_start_method('spawn', force=True)
  with Pool(processes = 2) as p:   # Parallelizing over 2 GPUs
    results = p.map(X_power_func, range(4))

However when I run this code, it hangs or keeps running forever without any errors.
When I removed set_start_method('spawn', force=True), the code ran properly and gave me the results, but this only works for when I run the code once. When I ran the code again in subsequent runs, I get the error RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method.
Someone in Reddit suggested that I should use torch.multiprocessing.spawn which lead me here back to PyTorch Forum with this post.