Hello @XiaoquanKong, I have worked out to include the metrics whilst using Method one: Using the CRF layer in a custom training loop:
optimizer = tf.keras.optimizers.Adam(0.02)
train_loss = tf.keras.metrics.Mean(name="train_loss")
train_acc_metric = tf.keras.metrics.BinaryAccuracy()
val_acc_metric = tf.keras.metrics.BinaryAccuracy()

@tf.function(experimental_relax_shapes=True)
def train_step(x, y):
    with tf.GradientTape() as tape:
        decoded_sequence, potentials, sequence_length, kernel = model(x)
        crf_loss = crf_loss_func(potentials, sequence_length, kernel, y)
        loss = crf_loss + tf.reduce_sum(model.losses)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    
    train_acc_metric.update_state(y, decoded_sequence)
    train_loss(loss)

EPOCHS = 10
for epoch in range(EPOCHS):
    # Reset the metrics at the start of the next epoch
    train_loss.reset_states()
    train_acc_metric.reset_states()

    for x, y in train_dataset:
        train_step(x, y)

    print(f"Epoch {epoch + 1}, " f"Loss: {train_loss.result()}")
    print(f"Epoch {epoch + 1}, " f"Accuracy: {train_acc_metric.result()}")

    for x, y in validation_dataset:
        decoded_sequence, potentials, sequence_length, kernel = model(x, training=False)
        val_acc_metric.update_state(y, decoded_sequence)
    val_acc = val_acc_metric.result()
    val_acc_metric.reset_states()
    print("Validation acc: %.4f" % (float(val_acc),))

This works well although there is some slowness, which is probably to be expected given the complexities of CRFs, and the training and validation accuracies both tend to drop after a few epochs of training - I am not sure if the latter phenomenon is due to the CRF’s inclusion. However, I am still unsure of how and where to add these metrics when using the CRF layer via model subclassing, which is the method I would prefer to use as with verbose=1, I can more easily monitor model performance.