Bert perform way worse than simple LSTM+Glove
Hi, I am doing a very straightforward entity classification task, but Bert is not giving a good result. I am wondering if there is something wrong with my code. 

My task is give a sentence and an entity boundary in that sentence, I predict entity type.

Here is my code to prepare input data. The basic idea is I have a sentence batch `batch_data`, then  I use tokenizer to encode that sentence batch to `s_encoded`. Then to pad the encoded ids so that each encoded id list can have same length, I pad them with 0's according the longest sentence in that batch (`max_len`).  Because I need to extract feature for the entity, so I use `entity_idx` to keep track of the entity boundary after it's tokenized.

Later, I just use `s_tensor` and `attn_mask_list` as input for BERT, and use `entity_idx` list to extract BERT feature for the entity.

Is there anything wrong with this part? Thank you!

```
s_bat = [x.tokens for x in batch_data]
s_encoded = tokenizer.batch_encode_plus([' '.join(x) for x in s_bat], add_special_tokens=True)
max_len = max([len(x) for x in s_encoded['input_ids']])

start_ids = [x.mention_start for x in batch_data]
ids_list = []
entity_idx = []
attn_mask_list = []
for i in range(len(s_bat)):
    ids = s_encoded['input_ids'][i]
    ids.extend([0 for _ in range(max_len-len(ids))])
    attn = s_encoded['attention_mask'][i]
    attn.extend([0 for _ in range(max_len-len(attn))])
    ids_list.append(ids)
    attn_mask_list.append(attn)

    m_start_idx = len(self.tokenizer.encode(' '.join(s_bat[i][:start_ids[i]]))) if start_ids[i]>0 else 0
    entity_idx.append([1+m_start_idx, 1+m_start_idx+len(m_encoded['input_ids'][i])]) #consider start special token
s_tensor = torch.LongTensor(ids_list)
attn_mask_list = torch.Tensor(attn_mask_list)

```

Well, how are you actually using it? Are you actually fine-tuning the model? What's your train loop?<|||||>I actually just solved the issue. It seems that the code I posted was
correct, but it has to do with where I placed the scheduler.step().

On Mon, Jan 20, 2020 at 6:57 AM Bram Vanroy <notifications@github.com>
wrote:

> Well, how are you actually using it? Are you actually fine-tuning the
> model? What's your train loop?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2567?email_source=notifications&email_token=AFB56KPQC3V5YOYAOQN5YRLQ6WGRXA5CNFSM4KIOCMLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJMMIKY#issuecomment-576242731>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFB56KMFFHCCUDF7SFZNR63Q6WGRXANCNFSM4KIOCMLA>
> .
>
<|||||>Glad to hear that you solved the issue. Please close the question. 