Thanks for replying to this “vague” question. Yes, it feels like there could be any kind of causes depending on use case.
If the model contains dropout and layenorm, do we expect to see zero loss for a single example during training? In this case, we should set model.eval() to expect zero loss, right?
what is a “standard” criterion and what’s not? Thanks.