Model Card for Model ID
Qwen 1.5 7B Fine-tuned for Oil & Gas Drilling Operations
This is a fine-tuned version of the Qwen 1.5 7B Large Language Model, specifically adapted for question-answering tasks related to oil and gas drilling operations. It was fine-tuned using the QLoRA method.
Model Details
Model Description
This model is a fine-tuned version of the Qwen 1.5 7B Large Language Model, specifically adapted for question-answering (Q&A) tasks within the domain of oil and gas drilling operations. It has undergone a process of QLoRA (Quantized Low-Rank Adaptation) fine-tuning using a proprietary dataset of approximately 6,000 Q&A pairs relevant to this industry. The fine-tuning process aimed to imbue the model with specialized knowledge, enabling it to generate accurate, coherent, and contextually appropriate responses to queries concerning drilling techniques, equipment, well control, wellbore stability, and related operational procedures. This adaptation allows the model to perform effectively as a domain-specific assistant, leveraging the powerful generative capabilities of the Qwen 1.5 base model while focusing its expertise on the intricacies of the drilling sector.
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
- Developed by: hoda-nik
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: hoda-nik
- Model type: Large Language Model (LLM), Qwen 1.5 7B, fine-tuned with QLoRA
- Language(s) (NLP): English
- License: Apache 2.0
- Finetuned from model [optional]: Qwen/Qwen1.5-7B-Chat
Model Sources [optional]
- Repository: https://huggingface.co/hoda-nik/qwen1.5-oil-gas-drilling
- Paper [optional]: [More Information Needed]
- Demo [optional]: (Add your Hugging Face Space link here once it's live, e.g.,
https://huggingface.co/spaces/hoda-nik/qwen1.5-oil-gas-drilling-demo)
Uses
Direct Use
This model is intended for direct use in answering specific questions and providing information related to oil and gas drilling operations. This includes, but is not limited to, queries about drilling techniques, equipment, terminology, safety protocols, and operational procedures within the oil and gas sector.
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
The model is not intended for:
- Providing legal, medical, or financial advice.
- Generating harmful, unethical, or biased content.
- Applications outside the domain of oil and gas drilling operations where it has not been specifically fine-tuned.
- High-stakes decision-making without human oversight.
Bias, Risks, and Limitations
As with all large language models, this model may exhibit:
- Bias: It may reflect biases present in its pre-training data or even in the fine-tuning data, which could lead to skewed or unfair responses.
- Hallucinations: It might generate factually incorrect or nonsensical information, especially when presented with ambiguous queries or outside its domain of expertise.
- Limited Knowledge Cutoff: Its knowledge is based on its training data and may not include the very latest information or developments in the oil and gas industry post its pre-training cutoff.
- Domain Specificity: While fine-tuned for oil and gas, its performance outside this specific domain may be significantly degraded.
- Lack of Real-world Understanding: The model processes text patterns and does not possess genuine understanding, consciousness, or real-world experience.
Recommendations
Users (both direct and downstream) should be made aware of these risks, biases, and limitations. It is crucial to implement human oversight and verification, especially for critical applications. Encourage feedback on model responses to help identify and mitigate issues.
How to Get Started with the Model
Use the code below to get started with the model.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
# Load the merged model directly from the Hub after pushing it
# (This assumes you merged the adapter weights into the base model before pushing)
model_id = "hoda-nik/qwen1.5-oil-gas-drilling"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16, # or torch.float16 depending on your GPU and model precision
device_map="auto"
)
# Example for text generation
messages = [
{"role": "system", "content": "You are a helpful assistant specialized in oil and gas drilling operations."},
{"role": "user", "content": "What is directional drilling?"}
]
text_input = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
input_ids = tokenizer(text_input, return_tensors='pt').input_ids.to(model.device)
generated_ids = model.generate(input_ids, max_new_tokens=500, do_sample=True, temperature=0.7, top_p=0.9)
generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(generated_text)
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
Dataset: Approximately 6,000 Question & Answer (Q&A) pairs.
Domain: Oil and gas drilling operations.
Maximum Token Length: Prompts + completions were capped at 800 tokens.
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** Fine-tuned using QLoRA (Quantized Low-Rank Adaptation)
per_device_train_batch_size: 4 (leading to an effective batch size of 16)
gradient_accumulation_steps: 4
num_train_epochs: 4
learning_rate: 5e-5
lr_scheduler_type: "cosine"
warmup_ratio: 0.05
optim: "paged_adamw_8bit"
gradient_checkpointing: True
fp16: True
seed: 42
logging_steps: 20
save_steps: 100
eval_steps: 100
eval_strategy: "steps"
save_strategy: "steps"
load_best_model_at_end: True
metric_for_best_model: "loss"
greater_is_better: False <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
Custom Q&A test set of oil and gas drilling operations.
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
Training Loss
Evaluation Loss (eval_loss)
### Results
Final Training Loss: ~0.717
Final Evaluation Loss: Approximately ~0.7 (converged close to training loss, indicating good generalization).
Qualitative Assessment: The model demonstrates high relevance, accuracy, coherence, and domain specificity in its completions, significantly improved after addressing truncation issues.
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** NVIDIA A100 GPU
- **Hours used:** Approximately 1.6 hours (5758.23 seconds) for training runtime.
- **Cloud Provider:** Google Colab
- **Compute Region:** Norway
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[email protected]
- Downloads last month
- 14
Model tree for hoda-nik/qwen1.5-oil-gas-drilling
Base model
Qwen/Qwen1.5-7B-Chat