text: Came here to ask about dropout too! I saw something saying it had become unfashionable in LLMs but presumably could help with avoiding such localisation (although super weight phenomena might be useful for interpretability?)