It should give you an output of shape [1, 20, 20], so that’s strange.
Could you check it again please?
The cosine similarity will be calculated between both tensors in the specified dimension. All other dimensions apparently deal as an additional storage and won’t be used in the calculation.
You can also reshape your input tensors to [batch_size, 2] and will get the same result:
res1 = F.cosine_similarity(input1, input2, 1)

res2 = F.cosine_similarity(
    input1.permute(0, 2, 3, 1).view(-1, 2), 
    input2.permute(0, 2, 3, 1).view(-1, 2),
    1).view(1, 20, 20)

print((res1 == res2).all())

You can find the implementation here 192.