Hello Olivier,
TF-DF has two independent means of model evaluation: 1) The internal metrics / loss used to train the model configured with the task or loss hyper-parameters, and 2) the Keras metrics supplied with compile. There is currently no connection between the two sets of metrics.
tfdf_clf.make_inspector().training_logs() returns the former while tfdf_clf.evaluate() returns the latter.
As you noted, only the internal metrics will be given for each number of trees in the model. The Keras metrics will only be given for the entire model.
Currently, the only solution to your problem is to manually remove trees from the model using the model builder and to evaluate it (more precisely, to create a new model for each number of trees you want to evaluate).
I understand that this is not a user-friendly solution. If you want, you can create a feature request on the project’s github page 2 to help us prioritize it.
Cheers,
Mathieu