No, as I said before those circles are not individual parts in the model. This is a very dummy visualization of Bert VS others simply to illustrate the directionality of the model. BERT is bidirectional and so it can predict a token in a masked position by taking into account left and right context. GPT is autoregressive and can only do left to right prediction/generation. That’s what the illustration shows. If you want to understand the architecture of BERT you should reread the Vaswani et al paper and look at the illustration of the Transformer. BERT is basically the encoder-only part of that.