SequenceSummary: config.summary_activation = 'relu' would be ignored
This isn't a bug, but merely an unintuitive argument name.

`summary_activation` sounds like it can be general, e.g. "relu" or "gelu" or something, but if its not "tanh" its ignored.

Since I assume it's annoying to go through all the configs and rename a field to use_tanh=True, I propose that we raise if summary_activation is a string thats not tanh, instead of silently just using no activation.

Another approach could integrate an `ACT2FN` dictionary (see  https://github.com/huggingface/transformers/issues/1347)
to actually support the other activation functions.

Happy to do either approach if others think it would be useful.

I think both approaches are reasonable. @LysandreJik @thomwolf?<|||||>I agree with both approaches as well. The second one would probably be the most useful.<|||||>Yes, like Lysandre