Bartowski PRO

Bf16 can't be offloaded to GPUs so imatrix becomes slow to make :')

I suppose I should add, that this is more valuable as a pseudo comparison to bf16

Since bf16 can represent the range (1, -1) with more precision than fp16, there is much debate as to whether it's safe to convert from bf16 to fp16, or if you should keep bf16, or even upcast to fp32, in order to preserve the original quality of the model for as long as possible before quantizing to 8 bits

This test shows that fp16 is capable of represent 99.97% of the weights in an FP32 model precisely, and therefore represents a negligible at best difference

Additionally, since the weights it can't represent are between 6e-5 and -6e-5, the weights it can't represent are so small that they most likely do not contribute to the finally output of the model and are relatively safe to prune

also maybe there should be a new feature to be explicitly notified about new repositories

That would be amazing, probably for average users but especially for me, where I sometimes stumble upon a model uploaded days ago that I somehow didn't notice from a creator I enjoy

We will have to see if something like that is possible without cluttering up the profile pages too much. But we'll try.

That sounds awesome, could even consider something like a toggle in the settings for "show this model on my page" or something, and possibly as a variable when using huggingface-cli or the HF python API

I think we'll be doing a social features sprint soon and this is exactly the kind of feedback we need! Thank you so much!

Beautiful, I love this :D If you need feedback on anything specific feel free to reach out, would love to be a guinea pig or just early eyes !

Oh another big pain point: notifications

I would love to be able to subscribe to be notified of new models posted by people or organizations, but it's near impossible as is

I would love better filtering

First I think sort by created is broken, but haven't checked on desktop recently

Second, I would love date filtering, like show me trending models that were only posted or updated in the past 7 days and such

I'm happy to hear this too, money in the bank is good, but upwards momentum makes it so much easier to justify investing in new technology and improving things!

much more difficult though if you're trying to iterate, definitely an interesting final validation

oh god dammit haha, i did not think of that possibility AT ALL 🤦

KL Divergence is almost identical - though even then upsetting that it's "almost" - but yup there's huge differences in the top p...

====== Perplexity statistics ======
Mean PPL(Q)                   :   6.339378 ±   0.038949
Mean PPL(base)                :   6.337070 ±   0.038896
Cor(ln(PPL(Q)), ln(PPL(base))):  99.99%
Mean ln(PPL(Q)/PPL(base))     :   0.000364 ±   0.000067
Mean PPL(Q)/PPL(base)         :   1.000364 ±   0.000067
Mean PPL(Q)-PPL(base)         :   0.002308 ±   0.000427

====== KL divergence statistics ======
Mean    KLD:   0.000005 ±   0.000001
Maximum KLD:   0.113848
99.9%   KLD:   0.000346
99.0%   KLD:   0.000055
99.0%   KLD:   0.000055
Median  KLD:   0.000001
10.0%   KLD:  -0.000014
 5.0%   KLD:  -0.000021
 1.0%   KLD:  -0.000035
Minimum KLD:  -0.000120

====== Token probability statistics ======
Mean    Δp:  0.002 ± 0.000 %
Maximum Δp: 19.102%
99.9%   Δp:  0.417%
99.0%   Δp:  0.155%
95.0%   Δp:  0.067%
90.0%   Δp:  0.040%
75.0%   Δp:  0.010%
Median  Δp:  0.000%
25.0%   Δp: -0.007%
10.0%   Δp: -0.034%
 5.0%   Δp: -0.062%
 1.0%   Δp: -0.154%
 0.1%   Δp: -0.439%
Minimum Δp: -5.820%
RMS Δp    :  0.078 ± 0.016 %
Same top p: 99.927 ± 0.007 %

Either way I appreciate the insight and now question all my life decisions, especially the ones that involved me uploading fp32 files and spending 3x the time calculating imatrix on bf16 instead of fp16

just for my own curiousity, I ran my fp16 conversion vs the fp32 KLD and got this:

====== Perplexity statistics ======
Mean PPL(Q)                   :   6.341096 ±   0.038970
Mean PPL(base)                :   6.337070 ±   0.038896
Cor(ln(PPL(Q)), ln(PPL(base))):  99.99%
Mean ln(PPL(Q)/PPL(base))     :   0.000635 ±   0.000085
Mean PPL(Q)/PPL(base)         :   1.000635 ±   0.000085
Mean PPL(Q)-PPL(base)         :   0.004026 ±   0.000543

====== KL divergence statistics ======
Mean    KLD:   0.000199 ±   0.000001
Maximum KLD:   0.066079
99.9%   KLD:   0.002797
99.0%   KLD:   0.001297
99.0%   KLD:   0.001297
Median  KLD:   0.000126
10.0%   KLD:   0.000001
 5.0%   KLD:  -0.000001
 1.0%   KLD:  -0.000012
Minimum KLD:  -0.000115

====== Token probability statistics ======
Mean    Δp:  0.005 ± 0.001 %
Maximum Δp:  6.664%
99.9%   Δp:  2.699%
99.0%   Δp:  1.539%
95.0%   Δp:  0.794%
90.0%   Δp:  0.483%
75.0%   Δp:  0.108%
Median  Δp:  0.000%
25.0%   Δp: -0.098%
10.0%   Δp: -0.466%
 5.0%   Δp: -0.779%
 1.0%   Δp: -1.518%
 0.1%   Δp: -2.630%
Minimum Δp: -9.853%
RMS Δp    :  0.493 ± 0.002 %
Same top p: 99.106 ± 0.024 %

so it looks like there IS a difference, but I guess after you quantize there's just so much more noise it's irrelevant (or, as you said, because "the scales for quantized data are always FP16")

The only thing then is to determine if it matters at all for the imatrix, but it seems unlikely considering my Q4_K_M results are at best statistical noise

Alternatively, I suppose it's possible that any values between 0 and 6e-5 are so small, that truncating them to 0 is the exact same as leaving them at full precision - they're just so tiny that their values don't change any perceived results (after quantization)

Yeah the BF16 -> FP32 being lossless makes sense to me, I'm just surprised that BF16 -> FP16 -> Q8 is identical to BF16 -> FP32 -> Q8, unless ALL values are within that range as you mentioned I would expect at minimum some noise

I could possibly find a way to check if all the weights are in that interval, and if they are, that would mean that fp16 is also lossless I suppose

But basically you're suggesting that at the end of the day, whether I convert to FP32, BF16, or FP16 (assuming a BF16 origin), the arithmetic in llama.cpp will make it so that it's irrelevant?

Here's a table showing the main results: