Pretrain GPT2 from scratch in Romanian
Train a Romanian GPT2 on a 24 GB Romanian dataset

2. Language
Currently, there is no GPT2 model that was trained from scratch for Romanian on the hub.
For this project, the goal is to create a strong language generation model for Romanian.

3. Model
A randomly initialized GPT2 model

4. Datasets
We created a dataset from OSCAR, text from books and web crawled datasets.
The entire dataset has 24 GB.

5. Training scripts
A causal language modeling script for FLAX is available here:
  

      github.com
  

  
    huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py


    #!/usr/bin/env python
# coding=utf-8
# Copyright 2021 The HuggingFace Team All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Pre-training/Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.

Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=causal-lm



  This file has been truncated. show original

  

  
    
    
  

  



6. (Optional) Challenges
<(Optionally) FILL ME: 6. What are possible challenges?>

7. (Optional) Desired project outcome
There are no causal language models for Romanian language. This will be the first causal model for Romanian language

8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.

  
      

      colab.research.google.com
  

  
    

Google Colaboratory 6