Hi !
You can indeed put the tokenized text on GPU and give it to the model.
Also you can make it significantly faster by using a batched map:
def embed(examples):
    tokenized_examples = ctx_tokenizer(
        examples["passage"],
        return_tensors="pt",
        padding="longest",
        truncation=True,
        max_length=512
    ).to(device=ctx_encoder.device)
    embeddings = ctx_encoder(**tokenized_examples)[0]
    return {"embeddings": embeddings}

dataset_embedded_passages = dataset_passages.map(embed, batched=True, batch_size=16)

Let me know if it helps