I tried both concatenating and using the last layer’s hidden, frozen and not frozen, and in all cases (in my task back then), the concatenation worked best. So I use it as my go-to method now, although I cannot be certain that it works equally well for other tasks or models.