Someone suggested adding this parameter in hopes of getting rid of this problem:
ctc_zero_infinity=True
Loss is gonna be gigantic and it does hold that every time I faced this issue, the first training loss was Inf so this is probably a good fix for the issue!