Different GPT-2 outputs with mixed precision vs single precision
When using GPT-2 with mixed precision, the generated text is different from that produced by running it normally. This is true for both conditional and unconditional generation, and for top_k=1 (deterministic) and top_k=40. Typically the mixed precision and single precision outputs agree for a number of tokens and then begin to disagree (sometimes early, sometimes late).

Using GPT-2 with mixed precision would be useful to take advantage of the tensor cores on V100 and T4 GPUs.

Testing by calling `model.half()` on GPT2LMHeadModel tends to start producing incorrect outputs early, while instead using Apex's AMP usually produces correct outputs for a little longer but still generally deviates. My tests were on the 117M model, with Apex installed.

It surprises me that the top_k=1 results often differ, sometimes very early in the sequence. They only take the largest logits, so this means the ranking of the logits is different.

I think the cause is compounding errors in the "past" tensor used by the attention function. Each time a new token is generated, its past has some error in it. When subsequent token generations then use those values (in higher attention layers), their own pasts have _more_ error. And so on, up through 16 layers for 117M or 24 for 345M. For cases where the top 2 logit values are almost the same, those 16 steps of error might be enough to change which one is larger and thereby change even the top_k=1 output. I haven't verified this idea yet.

I'm not sure if this necessarily means the outputs will be _qualitatively worse_, but that's a hard thing to measure.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@AdamDanielKing did you manage to fix the divergence somehow? <|||||>Was this finally an issue? It seems important and it was closed due to inactivity. @AdamDanielKing did you work it around?<|||||>@Damiox While sampling with mixed precision gives different results, they seem to still be of high quality. I've been using mixed precision on [talktotransformer.com](https://talktotransformer.com) for at least 6-7 months now and the quality has been excellent.<|||||>I am not using the "past" output at the moment, just doing topk=30. I ran into this issue as I am trying to improve my inferences times.
I have apex installed, and using the gpt2-medium model I get in average per sentence an inference time of 11ms for ~50 tokens within a batch size of 30 sentences on Tesla T4. Bigger batches aren't increasing the throughput. I just tried turning on fp16 into my model via ".half()" and it seems to be 3x faster. Is it possible? I am wondering whether that is fine or I need to do anything else (e.g. initializing apex, re-training my model with fp16). I feel I may be losing something. What do you think?<|||||>Talktotransformer.com just uses Apex to cast the model to fp16--no
retraining. I use the opt level that casts the weights themselves as well
(allows larger batch sizes). It seems to work well.

If you're getting 3x improvement from calling  .half() instead of
initializing with Apex, that is strange and I can't imagine why. I've found
that this methods diverges more quickly from fp32 than using Apex so I
haven't tested much with it.

On a separate note, I'd consider trying cheaper GPUs. It may be
counter-intuitive, but I've found for example on Google that cheap P4 GPUs
give greater throughput _for their cost_ than a T4 even though they don't
have tensor cores. I think this is for the following reason: in generating
each token after the first, only the values for a single position are being
computed at each iteration, which is very little computation for the amount
of memory being used. I think this results in a lot of time being spent on
GPU-CPU round trips rather than actual work. Batch size becomes more
important than GPU speed.

On Mon, Mar 9, 2020, 5:56 PM Damian Nardelli, <notifications@github.com>
wrote:

> I am not using the "past" output at the moment, just doing topk=30. I ran
> into this issue as I am trying to improve my inferences times.
> I have apex installed, and using the gpt2-medium model I get in average
> per sentence an inference time of 11ms for ~50 tokens within a batch size
> of 30 sentences on Tesla T4. Bigger batches aren't increasing the
> throughput. I just tried adding turning fp16 into my model via ".half()"
> and it seems to be 3x faster. Is it possible? I am wondering whether that
> is fine or I need to do anything else (e.g. initializing apex, re-training
> my model with fp16). I feel I may be losing something. What do you think?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/602?email_source=notifications&email_token=ABKUZHOUB74EQKNVRORWKB3RGVQX7A5CNFSM4HMG65OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOJG2FA#issuecomment-596798740>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABKUZHOS54LHVYM4OSGCR7LRGVQX7ANCNFSM4HMG65OA>
> .the
>
<|||||>I haven't tried initializing apex explicitly in my code. I believe I just thought that was being done by `GPT2LMHeadModel` behind the scenes, but it doesn't look like... I had installed it because I saw the following warning when running my inference app: `Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex`. Does this mean that the gpt2-medium is compiled with apex support? I'm not certain about that.

Should I try doing the amp.initialize() with the opt level in my inference app? Are you using O1? From https://nvidia.github.io/apex/amp.html#opt-levels-and-properties looks like I shouldn't be explicitly calling `half()`, so probably I should try initializing apex. What do you think?

Thanks in advance for all your help!<|||||>I had a question related to this: would the outputs in generation with GPT-2 change if the batch size changes?<|||||>Currently generation only allows `batch_size=1`