Solved by mMagmer in post #4 
              
              
                suppose cm is your confusion matrix, then 
cm.fill(0); # you can zero it every 10 batch for better estimation.

for i , j in zip(local_labels,predicted_labels):
            cm[i,j] += 1

recall = cm.diagonal() / (cm.sum(axis=1) + 1e-15)
AvgRecall = recall.mean()