Yes, that’s what I’d expect. But if I perform a single forward in train mode it messes with my model when I evaluate under eval mode. That’s what is weird, if you take a look at my first post, the do_validation function sets my model back to eval mode.