--- license: llama2 library_name: peft tags: - generated_from_trainer base_model: unsloth/llama-3-8b-bnb-4bit metrics: - accuracy model-index: - name: llama3-qwantz-coherent results: [] pipeline_tag: text-classification --- # llama3-qwantz-coherent This model is a fine-tuned version of [unsloth/llama-3-8b-bnb-4bit](https://huggingface.co/unsloth/llama-3-8b-bnb-4bit) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.3295 - Accuracy: 0.8758 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | |:-------------:|:-----:|:----:|:---------------:|:--------:| | 0.4482 | 1.0 | 1428 | 0.3295 | 0.8758 | ``` 2-4 fake words at the end (like training set): Can save 90% of coherent strings by discarding 82% of dp strings (cutoff is -67.26065874099731) Can save 95% of coherent strings by discarding 72% of dp strings (cutoff is -88.40824365615845) Can save 98% of coherent strings by discarding 62% of dp strings (cutoff is -95.06730437278748) Can save 99% of coherent strings by discarding 54% of dp strings (cutoff is -97.79982566833496) 1 fake word at the end: Can save 90% of coherent strings by discarding 48% of dp strings (cutoff is -77.83336043357849) Can save 95% of coherent strings by discarding 30% of dp strings (cutoff is -92.52431392669678) Can save 98% of coherent strings by discarding 26% of dp strings (cutoff is -95.45100927352905) Can save 99% of coherent strings by discarding 21% of dp strings (cutoff is -97.32990860939026) Examples (2-4 fake words): My only problem (s) have to do with ==> coherent: 99.12% My only problem (s) to cheer them personally ==> dp: 99.69% (in small text) crazy utahraptor ==> coherent: 88.54% (in small text) ". ==> coherent: 54.82% Well, I've made up my own joke to get him today. All I need to do is " ==> coherent: 77.98% Well, I've made up my own joke to get him today. All I need a father and gentlemen ==> dp: 99.79% I will be immortalized by kicking an evil ==> dp: 72.79% I will be immortalized by kicking other punches ==> dp: 99.49% Aw shoot, I was supposed to ==> coherent: 99.80% Aw shoot, I was APOCALYPSE PORN ==> dp: 94.10% Get it? Because CRIME DOESN'T PAY!! Listen, my story has ==> coherent: 66.25% Get it? Because CRIME DOESN'T PAY!! Listen, transcriptions of it ==> dp: 99.75% Utahraptor!! DON'T LISTEN TO ==> coherent: 99.96% Utahraptor! This is sort of ==> coherent: 95.96% Doesn't exist in my mouth, that is!! Because it's too big ==> coherent: 95.38% Doesn't exist in my mouth, that is!! Because if Superman. ==> dp: 99.66% Now, HERE'S how ==> coherent: 98.67% Now, guys would ==> dp: 95.30% But I am a rock star ==> coherent: 92.34% But I am a guy come ==> dp: 99.40% But I have a solution to make them interesting again: all you need is stories where not ==> coherent: 94.51% But I have a solution to make them interesting again: all you need is gonna! Diseases ==> dp: 99.94% At that point, there's a sequence of six nines in a row, and his joke was that he'd like to memorize pi up to that point, so that when reciting he could end with "9,9,9,9,9,9... and so on. " Others ==> coherent: 70.68% At that point, there's a sequence of six nines in a row, and his joke was that he'd like to memorize pi up to that point, so that when reciting he could end with "9,9,9,9,9,9... and so it's great he looks ==> dp: 99.54% This is definitely called " T -Rex's Hilarious e joke ", okay ==> coherent: 79.86% This is definitely called " T -Rex's Hilarious e joke AND IN THE ==> dp: 98.83% " Your mouth is full of cockroaches: ==> coherent: 93.64% " Your mouth is full of smooches. ==> coherent: 99.10% Excuse me, sexual congress? Everyone else on the planet is dead, and ==> coherent: 89.66% Excuse me, sexual congress? Everyone else on the planet without syntactic ambiguity! ==> dp: 97.02% Sony is going to write swears on my bathroom ==> coherent: 99.75% Sony is going to write their babies need to ==> dp: 99.68% Beginning with the most modest: why am I ==> coherent: 99.62% Beginning with the most modest: why T - ==> dp: 92.38% Is there any greater meaning -to anything ==> coherent: 95.96% Is there any greater meaning? When you ==> dp: 73.11% I've also got steaks AND ==> coherent: 92.46% I've also cold -deterministic ==> dp: 99.48% I had a friend (female) who dated her roommate (also female) ==> coherent: 98.78% I had a friend (female) who dated her roommate, je suis grand ==> dp: 97.52% Yes... TOO BAD INDEED ==> dp: 93.13% Yes... TOO MANY YEARS ==> coherent: 65.94% Examples (1 fake word): My only problem (s) have to do with you ==> dp: 83.54% My only problem (s) have to do with no ==> dp: 53.68% (in small text ==> coherent: 93.38% (in small changes ==> dp: 97.64% Well, I've made up my own joke to get him today. All I need to do is " ==> coherent: 77.98% Well, I've made up my own joke to get him today. All I need to do is already ==> dp: 97.05% I will be immortalized by kicking an evil kangaroo ==> coherent: 92.81% I will be immortalized by kicking an evil! ==> dp: 99.50% Aw shoot, I was supposed ==> coherent: 98.61% Aw shoot, I was how ==> dp: 99.55% Get it? Because CRIME DOESN'T PAY!! Listen, my story has both a hilarious twist ending and also ==> coherent: 99.64% Get it? Because CRIME DOESN'T PAY!! Listen, my story has both a hilarious twist ending and genders ==> dp: 88.29% Utahraptor!! DON'T LISTEN TO MY ==> coherent: 94.44% Utahraptor!! DON'T LISTEN TO THE ==> coherent: 97.86% Doesn't exist in my mouth, that is!! Because it's too ==> coherent: 96.91% Doesn't exist in my mouth, that is!! Because it's Well ==> dp: 97.87% Now, HERE'S how putting the things ==> dp: 97.33% Now, HERE'S how putting the wall ==> dp: 87.15% But I am a rock ==> coherent: 52.13% But I am a time ==> dp: 93.86% But I have a solution to make them interesting again: all you need is ==> coherent: 99.83% But I have a solution to make them interesting again: all you need to ==> coherent: 99.81% At that point, there's a sequence of six nines in a row, and his joke was that he'd like to memorize pi up to that point, so that when reciting he could end with "9,9,9 ==> coherent: 99.95% At that point, there's a sequence of six nines in a row, and his joke was that he'd like to memorize pi up to that point, so that when reciting he could end with "9,9, and ==> coherent: 95.76% This is definitely ==> coherent: 89.56% This is especially ==> coherent: 98.87% " Your mouth is full of cockroaches: many of them are dead, but those that ==> coherent: 99.96% " Your mouth is full of cockroaches: many of them are dead, but those taste ==> dp: 78.53% Excuse me, sexual congress? Everyone ==> dp: 94.45% Excuse me, sexual congress? " ==> dp: 91.74% Sony is going to write swears ==> coherent: 74.56% Sony is going to write that ==> dp: 98.15% Beginning with the ==> coherent: 84.62% Beginning with! ==> dp: 98.17% Is there any greater meaning -to ==> dp: 61.69% Is there any greater meaning -Rex ==> dp: 99.31% I've also got steaks AND pork chops ==> coherent: 99.84% I've also got steaks AND pork meat ==> coherent: 95.80% I had a friend (female ==> coherent: 85.77% I had a friend (on ==> coherent: 57.30% Yes... TOO ==> dp: 77.83% Yes... All ==> dp: 67.88% ``` ### Framework versions - PEFT 0.10.0 - Transformers 4.40.1 - Pytorch 2.2.1+cu121 - Datasets 2.19.0 - Tokenizers 0.19.1