I was lucky and found that normalising was the problem. So I did vector/np.linalg.norm(vector) and now all layers gives different cosine similarity between words.
And then I get the last layer by using:
torch_vector = hidden_states_1[-1][0][index]