@RichardWang
Thanks for responding. I understand that is_replaced = generated == inputs  would be the true label, but my issue is when an input is a masked token, generated will never equal the input, however, if the generated token is the value of the masked token before masking, then that particular token should not be considered replaced, which seems to not be the case under the setup you described. I am confused as to whether this is dealt with.