input decoder embedding for model.generate()
Hi, I'm using T5 model and is trying to generate a sentence using model.generate(). 
However, rather than using `decoder_input_ids` which gives the input_ids for the decoder part of T5, I would like to give a specific embedding I have to the decoder. Is there any way I can give specific decoder input embedding to model.generate()?

I noticed that in T5 model, I can give the embedding by `decoder_inputs_embeds`. Does this work in model.generate() too?

thanks :)
cc @patrickvonplaten <|||||>Hi there, 
I've also encountered the problem myself, while I've attempted to use GPT causal model instead of Google T5 as my base model. 

I was meant to use beam search for next sentence generation, while adding some self-defined vectors as prefix (which are not inside the embeddings).

While tests on self-implemented algorithms is successful in predicting the sentence, the method is tremendously slow compared to encapsulated method `model.generate()`. But a change to the method generate() shows several error, for `input_ids` is a required parameter, while I was trying to use the concatenated embedding vector as input.

The source doc shows that **model_kwargs will check for additional parameter `inputs_embeds`: 
`if input_ids is None and "inputs_embeds" not in model_kwargs:`
but even if I inserted the parameter, the method will still check for `input_ids`, and throw out an error for there's no "input_ids" in input parameters.<|||||>@aaafc Hi, Thanks for sharing the issue. 

I looked through the codes in `generation_utils.py` and also noticed the same problem. Though we pass `decoder_inputs_embeds` in the `model.generate()`, `model_inputs` in https://github.com/huggingface/transformers/blob/2024faf1713272464336ad97f371787b24cb3c53/src/transformers/generation_utils.py#L1785 doesn't contain the argument. 

I could partially solve the issue by separating the case where `decoder_inputs_embeds` value is passed or not
```
if decoder_inputs_embeds is None:
            outputs = self(
                **model_inputs,
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )
else:
           del model_inputs['decoder_input_ids'] 
            outputs = self( 
                **model_inputs,
                decoder_inputs_embeds=decoder_inputs_embeds
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )
```

However, the issue still exists in that the last part of the process where the embeddings change to ids and are passed on to the next iteration by adding into input_ids. (https://github.com/huggingface/transformers/blob/2024faf1713272464336ad97f371787b24cb3c53/src/transformers/generation_utils.py#L1851) This part should also be changed to a separate case where it is passed by ***ids*** or ***embeds***. I'm still working on how to change the whole iteration process by adding and passing the embedding rather than ids. 
It would be helpful if you leave any ideas or suggestions on this :) <|||||>Actually, I think I solved the issue by changing part under https://github.com/huggingface/transformers/blob/2024faf1713272464336ad97f371787b24cb3c53/src/transformers/generation_utils.py#L1773 as

```
iter_by_emb=True 
while True:
     ...
    if decoder_inputs_embeds is None:
            outputs = self(
                **model_inputs,
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )
    else:
           del model_inputs['decoder_input_ids'] 
            outputs = self( 
                **model_inputs,
                decoder_inputs_embeds=decoder_inputs_embeds
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )
    ....
    
    # argmax
    next_tokens = torch.argmax(next_tokens_scores, dim=-1)
    if iter_by_emb: 
      next_embs = self.get_input_embeddings()(next_tokens).unsqueeze(0)
   
    ....

    input_ids = torch.cat([input_idds, next_tokens[:, None]], dim=-1)
    if iter_by_emb: 
      decoder_inputs_embeds = next_embs 
```

I checked by running code a modified code from huggingface doc of `T5ForConditionalGeneration` (https://huggingface.co/transformers/model_doc/t5.html#inference)

```
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small')

# training
input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
labels = tokenizer('<extra_id_0> cute dog <extra_id_1> the <extra_id_2>', return_tensors='pt').input_ids
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss
logits = outputs.logits

# inference
input_ids = tokenizer("summarize: studies have shown that owning a dog is good for you", return_tensors="pt").input_ids  # Batch size 1
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# studies have shown that owning a dog is good for you.
```

to

```
from transformers import T5Tokenizer, T5ForCondititonalGeneration
tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small')
decoder_input_ids = tokenizer('summarize: I am happy', return_tensors='pt').input_ids
decoder_inputs_embeds = model.get_input_embeddings()(decoder_input_ids)
input_ids = tokenizer('summarize: studies have shown that owing a dog is good for you', return_tensors='pt').input_ids 

output_ids = model.generate(input_ids, decoder_input_ids=decoder_input_ids, max_length=30)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

output_embeds = model.generate(input_ids, decoder_inputs_embeds=decoder_inputs_embeds, max_length=30)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

returns

```
summarize: I am happy i have been a dog since I was a dog owner.
i have been a dog since I was a dog owner.
```

Though the generated sentence seems weird. I could get the same result except for the front part where I converted to embedding in the first place, "summarize: I am happy". <|||||>@amy-hyunji Hi, thanks for the reply.
I'd like to share some other issues, though its still far from solving the problem, but hopefully they could do some help.

Apart from the former mentioned parts where it tracks whether `input_ids` exists or not, there are also parts where it uses `input_ids` to convert other parameters or uses it for processing beam scores

> https://github.com/huggingface/transformers/blob/9eda0d156dcf34203e3f7ea5ad220c960fd3227b/src/transformers/generation_utils.py#L1839-L1840

or logits 

> https://github.com/huggingface/transformers/blob/9eda0d156dcf34203e3f7ea5ad220c960fd3227b/src/transformers/generation_utils.py#L1806

Thus I think simply ignore `input_ids` and turn to `inputs_embeds` at the model section may not work due to `input_ids` will be used for other sections. 

Since the line

> https://github.com/huggingface/transformers/blob/9eda0d156dcf34203e3f7ea5ad220c960fd3227b/src/transformers/generation_utils.py#L1785

Where it converts the model input just before the model section, I'm trying to overwrite the section implemented by Pretrained model

> https://github.com/huggingface/transformers/blob/9eda0d156dcf34203e3f7ea5ad220c960fd3227b/src/transformers/models/gpt2/modeling_gpt2.py#L982-L1008

and re-write it to convert the `input_ids` to the embeddings inside the method. 

```
class PrefixCausalGenerationModel(GPT2LMHeadModel, PrefixGenerationMixin):
    pass

class PrefixGenerationMixin(GenerationMixin):
    def prepare_inputs_for_generation(
        self, 
        input_ids: torch.LongTensor, 
        past = None, 
        **kwargs
    ):
    # re-write to convert input_ids here
```

I think this could retain `input_ids` for other processes, while it could mapped to some self-defined vectors here, without directly changing the model embeddings of base-model (In this case, GPT2). This is still in progress, though.




<|||||>Wow, Thanks.
It appears your solution is more clear. I'll try it myself lately.<|||||>Thanks for pointing out more issues. I do agree that when it goes to beam search, things are going to be more complicated so I started off from greedy_search. 

For the part about 
https://github.com/huggingface/transformers/blob/9eda0d156dcf34203e3f7ea5ad220c960fd3227b/src/transformers/generation_utils.py#L1806
based on my understanding, I believe there won't be any problem since by giving `decoder_inputs_embeds` to the model, we are simply expecting the model to generate the next token based on the embeddings and don't expect this to modify any part about the logits_processor which will work based on the `input_ids` that is generated by the model *based* on the given embeddings. However, this won't be an appropriate solution if you want the input embedding (decoder_inputs_embeds) to be converted into the most likely ids and return the value by `model.generate()` function. 

Also, I agree that overwriting `prepare_inputs_for_generration` is one of the solutions! :) <|||||>Hi there,

Well, I've tried to apply the former solution on GPT-2, while it appears that it doesn't work for Casual LMs, since Casual LMs requires `input_ids` directly as the input, thus we cannot send in `inputs_embeds` and ignore `inputs_ids`, even after we modified `model()` section.

As an alternative, I tried to implement the solution which changes `prepare_inputs_for_generation` method: 

```
class GPT2LMHeadModel(GPT2PreTrainedModel):

    ...    # other methods

    def prepare_inputs_for_generation(self, input_ids, past=None, **kwargs):

        ...    # lines which modifies input_ids

        use_embeds = kwargs.get("use_embeds", False)
        if (use_embeds):
            print("uses embeds.") # to test whether embeddings are used or not
            embeddings = kwargs.get("embeddings", self.get_input_embeddings())
            inputs_embeds = embeddings(input_ids)

        ...    # other operations

        model_inputs = {
            "past_key_values": past,
            "use_cache": kwargs.get("use_cache"),
            "position_ids": position_ids,
            "attention_mask": attention_mask,
            "token_type_ids": token_type_ids,
        }
        if (use_embeds):
            model_inputs["inputs_embeds"] = inputs_embeds
        else:
            model_inputs["input_ids"] = input_ids
        return model_inputs
```

And uses `generate()` method as:

```
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2')

a = model.generate(tokenizer.encode("Hello world", return_tensors='pt').to(model.device), max_length=30)
b = model.generate(tokenizer.encode("Hello world", return_tensors='pt').to(model.device), max_length=30, use_embeds=True)
tokenizer.decode(a[0])
tokenizer.decode(b[0])
```

and get

```
...    # Many lines of "uses embeds"

'Hello world, I\'m not sure what to say.\n\n"I\'m sorry, but I\'m not sure what to say.\n\n"'
'Hello world, I\'m not sure what to say.\n\n"I\'m sorry, but I\'m not sure what to say.\n\n"'
```
as the result. 

The method `generate()` here doesn't uses inputs_embeds as an input parameter, instead it requires an embedding matrix to map the input_ids to embeds. Thus, if we're willing to adding our own embedding vectors, we could create our own embedding matrix first.

```
    def __init__(self, **kwargs):
        ...
        self.embeddings = self.model.get_input_embeddings()
        self.prefixed_embeddings = nn.Embedding(self.tokenizer.vocab_size, self.embeddings.embedding_dim) 
        # the scale which is defined here doesn't really matter in following calculations 

    ...

    # do this before each time calling generate() method
    # if were intended to use self-defined input vectors for pseudo tokens
    def update_prefixed_embeddings(self):
        prefix_matrix = ... # the source for the prefix matrix
        self.set_prefix_embeddings(prefix_matrix)

    def set_prefixed_embeddings(self, pseudo_tokens_matrix):
        self.prefixed_embeddings.weight.data = torch.cat((self.embeddings.weight.data.detach().to(self.device), pseudo_tokens_matrix), dim=0).to(self.device)
        assert len(self.prefixed_embeddings.weight.data) == (self.origin_vocab_size + len(self.pseudo_tokens_list))
```
Then, uses 

```
trainer.update_prefixed_embeddings()
a = trainer.model.generate(trainer.tokenizer.encode("[PROMPT1]Hello[PROMPT2] world[PROMPT3]", return_tensors='pt').to(model.device), max_length=30, use_embeds=True, embeddings=trainer.prefixed_embeddings)
trainer.tokenizer.decode(a)
```

and get

```
...    # Also many lines of "uses embeds"

'[PROMPT1]Hello[PROMPT2] world[PROMPT3] world, the world of the world of the world of the world of the world of the world of the world of the world'
```

Well, it's ... even weirder after appending prefixes; but at least there's some changes. It's a good start.

By the way, I've also attempted to use
```
class PrefixGenerationMixin(GenerationMixin):
    def prepare_inputs_for_generation(

class PrefixCausalGenerationModel(AutoModelForCausalLM, PrefixGenerationMixin):
    pass
```
to avoid directly changing source code, but it doesn't work, since the model will not goes to the overwritten method but call the original one at `transformers.models.gpt2.modeling_gpt2.prepare_inputs_for_generation`. I'm attempting to find a way on improving this, well, later, though.


<|||||>Thanks for sharing! I have one question about the implementation. Is it necessary to create a new embedding matrix first if we want to use embeddings that are not in the model embeddings? I have the embeddings in another file so I thought that if we keep the embeddings in the right shape by concatenating them directly in the previous embeddings, updating the model embeddings isn't necessary.

Also, have you tried getting the score with the modified codes? I tried with the code I shared previously and the results were so low. I was wondering if you had similar issues with your codes.<|||||>Thanks for the reply.

As for the question, although my previous implementation created a new embedding matrix, it appears that we don't really need to create one. I did this cause I'm not sure whether uses the old embedding matrix with some additional rows during generation will influence the original embeddings or not, and I intended to prevent them from modifications. But since 

> https://github.com/huggingface/transformers/blob/ca2ef7dfcdf2f4f59f7935c058e3cf042596cb03/src/transformers/generation_utils.py#L639-L640

already prevent any gradients to be recorded, or one could simply freeze the parameters, there's no need to create a new embedding matrix.

As for the scores, my code is still on training process. The code uses web_nlg dataset (provided by HuggingFace datasets package) and GPT-2 as backbone model, and current logging does not show pretty good losses as well. Could be a lot of reasons, though, probably bad prompting templates, few prompt tokens, unfit initialization for the prefix matrix, or maybe just not performance well during training process. (I just hope the reason is not that prompting is not suitable for generation tasks :( )

I'm attempting to uses more flexible initializations or templates, but that is after this round is finished. My GPU is pretty old, so it takes a few days to train.<|||||>I'll also be working on this so if I find any other issues that would help, I'll leave some notes here. Thanks for sharing :) <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>This should have been solved by https://github.com/huggingface/transformers/pull/14443 I think :-)