Okay I think I have figured it out. You can recursively apply _init_weights to all submodules using apply. So, for example, if you wanted to re-initialize the last layer, the following should work:
from transformers import AutoModel

model = AutoModel.from_pretrained("bert-base-uncased")
# Print the weights before and after the call to _init_weights to confirm they have be re-initialized
# print(model.encoder.layer[-1].attention.output.dense.weight)
model.encoder.layer[-1].apply(model._init_weights)
# print(model.encoder.layer[-1].attention.output.dense.weight)