File size: 1,517 Bytes
0194203 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
---
language:
- hi
tags:
- hindi
- text-generation
- causal-lm
- lm
license: mit
datasets:
- custom_hindi_corpus
---
# Hindi-CausalLM
A Hindi language generation model with the following specifications:
## Model Architecture
- **Type**: Causal Language Model with Transformer architecture
- **Hidden size**: 768
- **Layers**: 12
- **Attention heads**: 16
- **Key-value heads**: 4 (using grouped-query attention)
- **Vocabulary size**: 16000
- **Parameters**: ~74.1M
- **Context window**: 512 tokens
- **Trained on**: Large corpus of Hindi text
## Training
The model was trained on a large corpus of Hindi text using a cosine learning rate schedule with warmup. Training utilized mixed-precision and distributed data parallel across multiple GPUs.
## Capabilities
This model can:
- Generate coherent Hindi text
- Continue text from a given prompt
- Create stories, explanations, and other content in Hindi
## Limitations
- Performance varies based on the similarity of the input to the training data
- May occasionally generate repetitive content for longer texts
- May produce grammatically incorrect Hindi in some contexts
- Has no knowledge of events beyond its training corpus
## Intended Use
This model is intended for Hindi language generation tasks, creative writing assistance, and as a foundation for fine-tuning on specific tasks.
## Ethical Considerations
Users should be aware that like all language models, this model may reproduce biases or generate problematic content in certain contexts.
|