More SEPs
I want to segment input sentences in more segments, like [CLS]S1[SEP]S2[SEP]S3[SEP]. Therefore, when I convert example to features, I do the following.
`segment_ids = [0] * len(tokens_s1)`
`segment_ids += [1] * len(tokens_s2)`
`segment_ids += [2] * len(tokens_s2)`

but I got the following error when I run the `self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)`:

>   File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 1065, in forward
    sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 712, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 264, in forward
    embeddings = self.dropout(embeddings)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 53, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/functional.py", line 595, in dropout
    return _functions.dropout.Dropout.apply(input, p, training, inplace)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/_functions/dropout.py", line 40, in forward
    ctx.noise.bernoulli_(1 - ctx.p).div_(1 - ctx.p)
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCTensorRandom.cu:34

after changing `segment_ids += [2] * len(tokens_s2)` to `segment_ids += [1] * len(tokens_s2)`, everything seems work but that is not what I want. Any suggestions? Thanks!


Hi, only two segment labels are pre-trained in BERT.
You could fine-tune a new vocabulary token but we don't have a script to do that currently so you would have to modify the vocabulary and model.
GPT and GPT-2 have option to do that where you can take inspiration from.
I'm happy to welcome a PR on this if somebody feels like giving it a try.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
