As I understand it, base scenario is with the training set - you check which features are discovered as important during the training. There may be some advanced uses on a held-out dataset (e.g. comparision plots or exploring generalization problems), but that’s more serious scrutinizing than basic X “columnwise” importance checking.