Hi,
I was able to resolve this by using send instead of isend in slave code.
def slaveRun(rank):
    global numParams
    print "runSlave: Hello from rank"+str(rank) 

    mytensor = torch.zeros(1,numParams)
    torch.add(mytensor,rank, out=mytensor)

    print "slave"+str(rank)+" :initial mytensor = "+str(mytensor)

    dist.send(tensor=mytensor, dst=0) #send the tensor to 0 (master)
    print "slave"+str(rank)+" sending tensor"

However this gives me some confusion on when and how isend/irecv can be used as opposed to send/recv.
Can isend from the worker be paired with a irecv from Master? didn’t work
Can isend from the worker be paired with a recv from Master? didn’t work
Can send from the worker be paired with a irecv from Master? it works
Can send from the worker be paired with a irecv from Master? it works
Would be great if someone could give me a little bit insight into the working of isend/irecv operators.
My final target is to implement an sync SGD where I ignore the slowest b workers. I want to analyze the effect of this straggler mitigation on the time/epoch, #epochs till convergence & final accuracy