gguf endpoints_compatible region:us
---
{}
---
Quantization made by Richard Erkhov.

[Github](https://github.com/RichardErkhov)

[Discord](https://discord.gg/pvy7H8DZMG)

[Request more models](https://github.com/RichardErkhov/quant_request)


gpt2-medium-biochem-bioasq-pubmedqa-demo - GGUF
- Model creator: https://huggingface.co/pikhan/
- Original model: https://huggingface.co/pikhan/gpt2-medium-biochem-bioasq-pubmedqa-demo/


| Name | Quant method | Size |
| ---- | ---- | ---- |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q2_K.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q2_K.gguf) | Q2_K | 0.17GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K_S.gguf) | Q3_K_S | 0.19GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K.gguf) | Q3_K | 0.21GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K_M.gguf) | Q3_K_M | 0.21GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q3_K_L.gguf) | Q3_K_L | 0.23GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.IQ4_XS.gguf) | IQ4_XS | 0.22GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_0.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_0.gguf) | Q4_0 | 0.23GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.IQ4_NL.gguf) | IQ4_NL | 0.23GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_K_S.gguf) | Q4_K_S | 0.23GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_K.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_K.gguf) | Q4_K | 0.25GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_K_M.gguf) | Q4_K_M | 0.25GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_1.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q4_1.gguf) | Q4_1 | 0.25GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_0.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_0.gguf) | Q5_0 | 0.27GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_K_S.gguf) | Q5_K_S | 0.27GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_K.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_K.gguf) | Q5_K | 0.29GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_K_M.gguf) | Q5_K_M | 0.29GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_1.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q5_1.gguf) | Q5_1 | 0.29GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q6_K.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q6_K.gguf) | Q6_K | 0.32GB |
| [gpt2-medium-biochem-bioasq-pubmedqa-demo.Q8_0.gguf](https://huggingface.co/RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf/blob/main/gpt2-medium-biochem-bioasq-pubmedqa-demo.Q8_0.gguf) | Q8_0 | 0.41GB |




Original model description:
---
tags:
- 4th gen xeon
---
datasets:
  - qiaojin/PubMedQA
  - kroshan/BioASQ
language:
  - en
library_name: transformers
pipeline_tag: table-question-answering
tags:
  - chemistry
  - biology
  - molecular
  - synthetic
  - language model
Description:
This model is an example of how a fine-tuned LLM even without the full depth, size, and complexity of larger and more expensive models can be useful in context-sensitive situations. In our use-case, we are applying this LLM as part of a broader electronic lab notebook software setup for molecular and computational biologists. This GPT-2 has been finetuned on datasets from BioASQ and PubMedQA and is now knowledgeable enough in biochemistry to assist scientists and integrates as not just a copilot-like tool but also as a lab partner to the overall Design-Built-Test-Learn workflow ever growing in prominence in synthetic biology.

Intel Optimization Inference Code Sample:
We made use of both the BF16 datatype and INT8 quantization to improve performance. BF16 halves the memory compared to FP32, allowing larger models and/or larger batches to fit into memory. Moreover, BF16 is supported by modern Intel CPUs and operations with it are optimized. Quantizing models to INT8 can reduce the model size, making better use of cache and speeding up load times. Additionally, we then optimized further with OpenVino to make it run better on Intel Hardware by converting it to an onxx model to then OpenVINO Intermediate Representation

from openvino.runtime import Core
import numpy as np

# Initialize the OpenVINO runtime Core
ie = Core()

# Load and compile the model for the CPU device
compiled_model = ie.compile_model(model='../ovc_output/converted_model.xml', device_name="CPU")

# Prepare input: a non tokenized example just for examples sake
input_ids = np.random.randint(0, 50256, (1, 10))

# Create a dictionary for the inputs expected by the model
inputs = {"input_ids": input_ids}

# Create an infer request and start synchronous inference
result = compiled_model.create_infer_request().infer(inputs=inputs)

# Access output tensor data directly from the result using the appropriate output key
output = result['outputs']

print("Inference results:", output)
In the finetuning file you will see our other optimizations.

We perform BF16 conversion as follows (we also implement a custom collator):

model = GPT2LMHeadModel.from_pretrained('gpt2-medium').to(torch.bfloat16)
We perform Int8 quantization as follows:

# Load the full-precision model
model.eval()  # Ensure the model is in evaluation mode
quantized_model = quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

