Many thanks @Narsil !
I find most models do not have such information in their README. For example, this huggingface model 1 doesn’t have a README at all. It’s also difficult to get any information from its vocab file. By “dive into the model” do you mean just try a bunch of examples? That’d be tricky I think. Even for the models which instructed to use SEPARATION tokens, you can even get somewhat reasonable results w/o using them as it seems. For examples, using this model 1, if you remove </s></s> the inference API still returns similar results…