How to use hugging on onw embedding
i already get a embedding，or i have a matrix for input. Then I want to use transformer to train on. For example use Longformer，i set input_ids to None，and set the input on parameter embed_input. Could it be OK？ anything else need to take care？ I also input the mask.
Hello @Yangxiaojun1230 ,

From your message I assume you are trying to train a Longformer model using embeddings as the input of the model.

You can either use input_ids or inputs_embeds in the forward function of the models. I think your idea will work, please let us know if it does.

You can look at more information about your model in the documentation: https://huggingface.co/transformers/model_doc/longformer.html
<|||||>@na

> Hello @Yangxiaojun1230 ,
> 
> From your message I assume you are trying to train a Longformer model using embeddings as the input of the model.
> 
> You can either use input_ids or inputs_embeds in the forward function of the models. I think your idea will work, please let us know if it does.
> 
> You can look at more information about your model in the documentation: https://huggingface.co/transformers/model_doc/longformer.html

Thank you for your answer, actually I try to train the model and meet an error.
My code:
att_mask[:,[100,300,500,800,1200,]]=2
logit=LongformerForSequenceClassification(inputs_embeds=embding,attention_mask=att_mask)

but error occurs:
pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [205,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCReduceAll.cuh line=327 error=710 : device-side assert triggered

"/home/yangjeff/anaconda3/lib/python3.6/site-packages/transformers/modeling_longformer.py", line 272, in forward
    max_num_extra_indices_per_batch = num_extra_indices_per_batch.max()
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCReduceAll.cuh:327

In the modeling_longformer.py:
after:extra_attention_mask = attention_mask > 0
I could not access to extra_attention_mask，
But in the example:
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
        attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
                                            # classification: the <s> token
                                            # QA: question tokens
                                            # LM: potentially on the beginning of sentences and paragraphs
        sequence_output, pooled_output = model(input_ids, attention_mask=attention_mask)
The only different in my code is I input inputs_embeds instead input_ids<|||||>I fixed finally