readme: ---
  license: apache-2.0
  tags:
  - distigpt2
  - hearthstone
  metrics:
  - bleu
  - dvitel/codebleu
  - exact_match
  - chrf
  datasets:
  - dvitel/hearthstone
  model-index:
  - name: h1
    results:
    - task:
        type: text-generation
        name: Python Code Synthesis
      dataset:
        type: dvitel/hearthstone
        name: HearthStone
        split: test
      metrics:
        - type: exact_match
          value: 0.21212121212121213
          name: Exact Match
        - type: bleu
          value: 0.9637468196180485
          name: BLEU        
        - type: dvitel/codebleu
          value: 0.8884667222252154
          name: CodeBLEU                
        - type: chrf
          value: 96.5942286007928
          name: chrF     
  ---
  
  # h1
  
  This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on [hearthstone](https://huggingface.co/datasets/dvitel/hearthstone) dataset.
  [GitHub repo](https://github.com/dvitel/nlp-sem-parsing/blob/master/h1.py).
  It achieves the following results on the evaluation set:
  - Loss: 0.0890
  - Exact Match: 0.1970
  - Bleu: 0.9737
  - Codebleu: 0.9172
  - Ngram Match Score: 0.8984
  - Weighted Ngram Match Score: 0.8985
  - Syntax Match Score: 0.9293
  - Dataflow Match Score: 0.9429
  - Chrf: 97.5313
  
  ## Model description
  
  DistilGPT2 applied onto HearthStone dataset with preprocessing of python code to dumped AST. Example:
  ```python
  #gold labels
  Module([ClassDef('Innervate', [Name('SpellCard', Load())], [], [FunctionDef('__init__', arguments([], [arg('self', None, None)], None, [], [], None, []), [Expr(Call(Attribute(Call(Name('super', Load()), [], []), '__init__', Load()), [Constant('Innervate', None), Constant(0, None), Attribute(Name('CHARACTER_CLASS', Load()), 'DRUID', Load()), Attribute(Name('CARD_RARITY', Load()), 'FREE', Load())], []))], [], None, None), FunctionDef('use', arguments([], [arg('self', None, None), arg('player', None, None), arg('game', None, None)], None, [], [], None, []), [Expr(Call(Attribute(Call(Name('super', Load()), [], []), 'use', Load()), [Name('player', Load()), Name('game', Load())], [])), If(Compare(Attribute(Name('player', Load()),'mana', Load()), [Lt()], [Constant(8, None)]), [AugAssign(Attribute(Name('player', Load()),'mana', Store()), Add(), Constant(2, None))], [Assign([Attribute(Name('player', Load()),'mana', Store())], Constant(10, None), None)])], [], None, None)], [])], [])
  ```
  ```python
  #wrong prediction (example of error after training)
  Module([ClassDef('Innervate', [Name('SpellCard', Load())], [], [FunctionDef('__init__', arguments([], [arg('self', None, None)], None, [], [], None, []), [Expr(Call(Attribute(Call(Name('super', Load()), [], []), '__init__', Load()), [Constant('Innervate', None), Constant(0, None), Attribute(Name('CHARACTER_CLASS', Load()), 'DRUID', Load()), Attribute(Name('CARD_RARITY', Load()), 'FREE', Load())], []))], [], None, None), FunctionDef('use', arguments([], [arg('self', None, None), arg('player', None, None), arg('game', None, None)], None, [], [], None, []), [Expr(Call(Attribute(Call(Name('super', Load()), [], []), 'use', Load()), [Name('player', Load()), Name('game', Load())], [])), For(Compare(Attribute(Name('player', Load()),'maxa', Load()), [Lt()], [Constant(10, None)]), [AugAssign(Attribute(Name('player', Load()),'mana', Store()), Add(), Constant(2, None))], Exign([Name(Name('player', Load()),'mana', Store())], Constant(None, None), None)],], [], None, None)], [])], [])
  ```
  
  
  ## Intended uses & limitations
  
  HearthStone card code synthesis. 
  
  ## Training and evaluation data
  
  See split of [hearthstone](https://huggingface.co/datasets/dvitel/hearthstone) dataset
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 2e-05
  - train_batch_size: 4
  - eval_batch_size: 4
  - seed: 17
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - num_epochs: 200
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch  | Step  | Validation Loss | Exact Match | Bleu   | Codebleu | Ngram Match Score | Weighted Ngram Match Score | Syntax Match Score | Dataflow Match Score | Chrf    |
  |:-------------:|:------:|:-----:|:---------------:|:-----------:|:------:|:--------:|:-----------------:|:--------------------------:|:------------------:|:--------------------:|:-------:|
  | 0.3871        | 11.94  | 1600  | 0.1043          | 0.0152      | 0.9499 | 0.8549   | 0.8089            | 0.8089                     | 0.8653             | 0.9366               | 95.4674 |
  | 0.0752        | 23.88  | 3200  | 0.0784          | 0.1212      | 0.9640 | 0.8874   | 0.8525            | 0.8526                     | 0.8929             | 0.9516               | 96.7978 |
  | 0.0448        | 35.82  | 4800  | 0.0717          | 0.1364      | 0.9693 | 0.9077   | 0.8782            | 0.8782                     | 0.9069             | 0.9674               | 97.2100 |
  | 0.0308        | 47.76  | 6400  | 0.0752          | 0.1364      | 0.9702 | 0.9061   | 0.8808       