[UPDATED] I got a workaround.
It seems like initializing BartTokenizer from vocab.json and merge.txt cause the problem.
Even when I initialize BartTokenizer with vocab.json and merge.txt form Roberta’s pre-trained ones, same problem happened.
Here’s my codes.
# import BartTokenizer
from transformers import BartTokenizer

tokenizer = BartTokenizer(
    vocab_file="roberta/vocab.json",
    merges_file="roberta/merges.txt",
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
)

vocab.json and merge.txt was downloaded from below.
https://huggingface.co/roberta-base/resolve/main/vocab.json
https://huggingface.co/roberta-base/resolve/main/merges.txt

input1 = "This is a pen."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[ 713,   16,   10, 7670,    4]])

input1 = "<s>This is a pen.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[41552,    29, 15698,   713,    16,    10,  7670, 49803,    29, 15698]]) < ???


input1 = "<s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[41552,    29, 15698]]) < ???

I got similar problem even with from_pretrained.
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base', add_prefix_space=True)

input1 = "This is a pen."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[ 152,   16,   10, 7670,    4]])

input1 = "<s> This is a pen.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[1437,    0,  152,   16,   10, 7670,    4,    2]]) < ???

encoded = tokenizer("<s>", add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[1437,    0]]) <<< ???

But when I use AutoTokenizer, it works fine.
from transformers import AutoTokenizer

# tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "facebook/bart-base",
)

input1 = "This is a pen."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[ 713,   16,   10, 7670,    4]])

input1 = "<s>This is a pen.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[   0,  713,   16,   10, 7670,    4,    2]])

encoded = tokenizer("<s>", add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[0]])  < looks good

And I found a workaround.
Saving pre-trained tokenizer model first and replacing vocab.json and merge.txt with the files created by ByteLevelBPETokenizer works.
# save tokenizer model.
tokenizer.save_pretrained("./saved_model")

# replace vocab.json and merge.txt

# load tokenizer model 
tokenizer = AutoTokenizer.from_pretrained('./saved_model/')

input1 = "Mi estas Julien."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[  958,   316, 14540,   276,    18]])

input1 = "<s>Mi estas Julien.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[    0,   958,   316, 14540,   276,    18,     2]])

input1 = "<s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[0]]) < looks good!