Martin:

pt = tokenizers.pt.tokenize(pt)


In the tokenize_pairs, ‘pt’ is a Tensor. A tokenizer usually takes a string object and return a list.  Why does this tokenizers.pt.tokenize() takes a tensor as input? How to make such a tokenizer to work with this tutorial? I imported the transformers tokenizer but it didn’t work, because it complains about the Tensor as tokenize input:
from transformers import BertTokenizer
tokenizer_en = BertTokenizer.from_pretrained("bert-base-cased")
tokenizer_zh = BertTokenizer.from_pretrained("bert-base-chinese")

def tokenize_pairs(zh, en):
    zh = tokenizer_zh.tokenize(zh)
    # Convert from ragged to dense, padding with zeros.
    zh = zh.to_tensor()
    en = tokenizer_en.tokenize(en)
    # Convert from ragged to dense, padding with zeros.
    en = en.to_tensor()
    return zh, en

BUFFER_SIZE = 20000
BATCH_SIZE = 64
def make_batches(ds):
  return (
      ds
      .cache()
      .shuffle(BUFFER_SIZE)
      .batch(BATCH_SIZE)
      .map(tokenize_pairs, num_parallel_calls=tf.data.experimental.AUTOTUNE)
      .prefetch(tf.data.experimental.AUTOTUNE))

train_batches = make_batches(train_examples)
val_batches = make_batches(val_examples)

So the question is how to adapt the transformer pretrained tokenizers to work with tutorial?