You can initialize the weights of the decoder with the weights of any encoder-only model too, like BERT. This is because a decoder is also just a stack of blocks (self-attention + feedforward neural networks), similar to an encoder. The only difference is that a decoder also adds cross-attention layers. Hence, if you initialize the weights of a decoder with the weights of an encoder-only model, the weights of the cross-attention layers will be randomly initialized, and need to be fine-tuned on a downstream task (like summarization, machine translation, image captioning).
Also, to answer your question: models like BERT/RoBERTa/GPT-x all operate on subword tokens, rather than words. This means that a word like “cookie” might be tokenized into “coo” and “kie”. These models learn an embedding vector for each individual subword token.