aclifton314:

If I understand correctly, because I have 4 GPUs world_size = 4. However, I’m not entirely sure what rank should be given that I have 4 GPUs. Also, I have my data stored in a pickled pandas dataframe with two columns being the embeddings that would be used in the cosine similarity calculation. It looks something like cos_sim(embed_a, embed_b). I’m not entirely sure how to read in the data properly for DistributedDataParallel and the forward method of CosSimNetwork. Any advice is much appreciated.


For DistributedDataParallel, you need to run one process per GPU and use ranks 0 - 3 for GPUs 0 - 3. I think your example code above should already do that automatically since mp.spawn would pass in the appropriate rank. For reading the data, each process can read a chunk of the data and process it. For example, maybe create 4 chunks of the ./embed_pairs_df.pkl file or read the same file on all processes and each process only processes a part of the input. For example process 0 computes rows 0, 4, 8 and so on etc.