In the original tutorial, the type of pt and en in tokenize_pairs is:
<class 'tensorflow.python.framework.ops.Tensor'>
My code above, the type of zh and en in tokenize_pairs is:
<class 'numpy.ndarray'>
I take the map-then-batch approach:
def make_batches(ds):
      return (
          ds
          .cache()
          .shuffle(BUFFER_SIZE)
          .map(py_wrap_tokenize_pairs, num_parallel_calls=tf.data.experimental.AUTOTUNE)
          .batch(BATCH_SIZE)
          .prefetch(tf.data.experimental.AUTOTUNE))

But now the input type becomes ‘bytes’ from ‘numpy.ndarray’, and this will produces the error:
File "/Users/cong/.venv/tf2/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 285, in split_on_token
split_text = text.split(tok)

TypeError: a bytes-like object is required, not 'str'

If I add a str(zh), this produces a str object, not a Tensor. Because the original tokenize_pairs expect a Tensor of string as input, how to make the ‘zh’ become a tensor before it is fed into:
zh = tokenizer_zh.tokenize(zh)