maybe @markdaoust can help here

I hope so, I did write these tutorials.

In the tokenize_pairs, ‘pt’ is a Tensor. A tokenizer usually takes a string object and return a list. Why does this tokenizers.pt.tokenize() takes a tensor as input?

Here I built the tokenizer in TensorFlow. This way when you export the model, at the end, the resulting saved model just takes a tensor of strings as inputs. The caller doesn’t need to think about the tokenizer.
The process to create those tokenizers is shown in the subwords_tokenizer tutorial. But that approach is not appropriate for Chinese. You’d need to use sentencepiece.
The simplest approach to getting this to work for Chinese would be to grab a pretrained Chinese segmentation model from tfhub, like:

TensorFlow Hub 1
TensorFlow Hub 1


How to make such a tokenizer to work with this tutorial? I imported the transformers tokenizer but it didn’t work, because it complains about the Tensor as tokenize input:


from transformers import BertTokenizer


Right, huggingface’s tokenizers operate in python. To use a python tokenizer in TensorFlow you’ll need to call it with tf.py_function or tf.numpy_function
def py_wrap_tokenize_pairs(zh, en):
  return tf.numpy_function(tokenize_pairs, [zh, en])
...

ds.map(py_wrap_tokenize_pairs,  num_parallel_calls=tf.data.experimental.AUTOTUNE)

That way the function runs in a regular python, it receives numpy arrays as input.