Emergent Features in 6.7B+ Param Transformers
Could any explain this section a little more? I didn’t quite follow this part. What does it mean to squish by 4 or by 2?> The only way to improve quantization is through more normalization constants. A normalization constant squishes the input distribution, for example, I5, into the target distribution, for example, I3. We can increase precision, by squishing each vector only as much as is needed. For example, if you have the two vectors:
> 
> [3, 1, 2, 3] 
> [0, 2, 2, 0]
> 
> Then you can squish the first by 4 and the second by 2. This will give you twice the precision to quantize the second vector because the inputs are now spread over a broader range of the I3 data type.