Yes, you are correct. I have just one doubt:
How could I find the layers of BERT in a sequential manner so that I can just index it and get all the layers after the embedding layer by doing bert.layers[1:].
But, there is no layer attribute like in your TF code.
Any ideas on this?
Because I could not find it, I tried to change the layer itself instead.

I can get the hidden states of the trained model. But how would I get the parts of the model itself so that I can train/finetune it as per my sub module above?
Thanks