I found this might be the incorrect data type I assigned to the input layer of the base model(dtype=int32 → dtype = float32, as shown below). After the modification, it works in both TF2.6 and TF2.7. (Still don’t know the reason why the results in the TF2.6 and TF2.7 are different if the input data type are wrongly assigned)
# Build the model
def build_embedding_bilstm_crf_model(
    vocab_size: int, embed_dims: int, lstm_unit: int
) -> tf.keras.Model:
    x = tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name="x")
    y = tf.keras.layers.Embedding(vocab_size, embed_dims, mask_zero=True)(x)
    y = tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(lstm_unit, return_sequences=True)
    )(y)

    return tf.keras.Model(
        inputs=x, outputs=y
    )


base_model = build_embedding_bilstm_crf_model(VOCAB_SIZE, 32, 64)