readme: # GPT-Code-Clippy-125M-Code-Search-Py
  > **Please refer to our new [GitHub Wiki](https://github.com/ncoop57/gpt-code-clippy/wiki) which documents our efforts in detail in creating the open source version of GitHub  Copilot**
  
  ## Model Description
  
  GPT-CC-125M-Code-Search is a [GPT-Neo-125M model](https://huggingface.co/EleutherAI/gpt-neo-125M) finetuned using causal language modeling on only the python language in the [CodeSearchNet Challenge dataset](https://huggingface.co/datasets/code_search_net). This model is specialized to autocomplete methods in the python language.
  
  ## Training data
  
  [CodeSearchNet Challenge dataset](https://huggingface.co/datasets/code_search_net).
  
  ## Training procedure
  
  The training script used to train this model can be found [here](https://github.com/ncoop57/gpt-code-clippy/blob/camera-ready/training/run_clm_flax.py).
  
  ```bash
  ./run_clm_flax.py \
      --output_dir $HOME/gpt-neo-125M-code-search-py \
      --model_name_or_path="EleutherAI/gpt-neo-125M" \
      --dataset_name code_search_net \
      --dataset_config_name="python" \
      --do_train --do_eval \
      --block_size="512" \
      --per_device_train_batch_size="32" \
      --per_device_eval_batch_size="64" \
      --preprocessing_num_workers="8" \
      --learning_rate="1.2e-4" \
      --num_train_epochs 20 \
      --warmup_steps 3000 \
      --adam_beta1="0.9" \
      --adam_beta2="0.95" \
      --weight_decay="0.1" \
      --overwrite_output_dir \
      --logging_steps="25" \
      --eval_steps="500" \
      --push_to_hub="False" \
      --report_to="all" \
      --dtype="bfloat16" \
      --skip_memory_metrics="True" \
      --save_steps="500" \
      --save_total_limit 10 \
      --report_to="wandb" \
      --run_name="gpt-neo-125M-code-search-py"
  ```
  
  ## Intended Use and Limitations
  
  The model is finetuned methods from the python language and is intended to autocomplete python methods given some prompt (method signature and docstring).
  
  ### How to use
  
  You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:
  
  ```py
  
  from transformers import AutoModelForCausalLM, AutoTokenizer, FlaxAutoModelForCausalLM
  
  model = AutoModelForCausalLM.from_pretrained("flax-community/gpt-neo-125M-code-clippy-code-search-py")
  
  tokenizer = AutoTokenizer.from_pretrained("flax-community/gpt-neo-125M-code-clippy-code-search-py")
  
  prompt = """def greet(name):
    '''A function to greet user. Given a user name it should say hello'''
  """ 
  
  input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device)
  
  start = input_ids.size(1)
  
  out = model.generate(input_ids, do_sample=True, max_length=50, num_beams=2, 
  
                       early_stopping=True, eos_token_id=tokenizer.eos_token_id, )
  
  print(tokenizer.decode(out[0][start:]))
  
  ```
  
  ### Limitations and Biases
  
  The model is intended to be used for research purposes and comes with no guarantees of quality of generated code.
  
  GPT-CC is finetuned from GPT-Neo and might have inherited biases and limitations from it. See [GPT-Neo model card](https://huggingface.co/EleutherAI/gpt-neo-125M#limitations-and-biases) for details.
  
  ## Eval results
  
  Coming soon...