We feed the input sentence or text into a transformer network like BERT. BERT produces contextualized word embeddings for all input tokens in our text. As we want a fixed-sized output representation (vector u), we need a pooling layer. Different pooling options are available, the most basic one is mean-pooling: We simply average all contextualized word embeddings BERT is giving us. This gives us a fixed 768 dimensional output vector independent how long our input text was.
The depicted architecture, consisting on a BERT layer and a pooling layer is one final SentenceTransformer model.
Creating Networks from Scratch
In the quick start & usage examples, we used pre-trained SentenceTransformer models that already come with a BERT layer and a pooling layer.
But we can create the networks architectures from scratch by defining the individual layers. For example, the following code would create the depicted network architecture:
