SOLVED
Apparently, if even you are calculating the loss on your own, you have to pass in labels if you pass ing decoder_input_ids.
Ya’ll may want to add this to the docs or update the behavior to not require the labels argument.
Thanks - wg