Thanks for the feedback @tom!
I reduced the resolution of the input and that fixed the issue suggesting that it was a capacity problem. I had checked permuting the labels and that still had the same behavior.