image695×342 5.2 KB
Relevant 3
Thank you so much for taking the time to share this.
Was just reading a related blog, you mean what is depicted in this diagram?
I find it strange that the model requires SOS and EOS tokens for whatever is input into the encoder/decoder, especially since those tokens are converted into their embeddings anyways. I figured that since the model is ignoring the target masking component for the current embeddings so why should it treat the presence of SOS & EOS any different. How does it know to finally start using the target masks? It not like there is some explicit mechanism inside the transformer model that says “Hey we’re finally being fed SOS and EOS tokens, so lets get our act together and start using the target mask!”