einsum can express any reductions, perhaps what you want looks:
torch.einsum("bsce,be->bsc", ovectors, ivectors)

Matmul for this reduction can be done with shapes (2,2,3,5) @ (2,1,5,1) = (2,2,3,1) (this does bmm with two first dimensions joined&broadcasted)
torch.matmul(ovectors, ivectors.reshape(batch_sz,1,embedding_sz,1)).squeeze(-1)