readme: ---
  license: apache-2.0
  tags:
  - distigpt2
  - hearthstone
  metrics:
  - bleu
  - dvitel/codebleu
  - exact_match
  - chrf
  datasets:
  - dvitel/hearthstone
  model-index:
  - name: h0
    results:
    - task:
        type: text-generation
        name: Python Code Synthesis
      dataset:
        type: dvitel/hearthstone
        name: HearthStone
        split: test
      metrics:
        - type: exact_match
          value: 0.30303030303030304
          name: Exact Match
        - type: bleu
          value: 0.8850182403024257
          name: BLEU        
        - type: dvitel/codebleu
          value: 0.677852377992836
          name: CodeBLEU                
        - type: chrf
          value: 91.00848749530383
          name: chrF     
  ---
  
  # h3
  
  This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on [hearthstone](https://huggingface.co/datasets/dvitel/hearthstone) dataset.
  [GitHub repo](https://github.com/dvitel/nlp-sem-parsing/blob/master/h3.py).
  It achieves the following results on the evaluation set:
  - Loss: 0.2782
  - Exact Match: 0.2879
  - Bleu: 0.9121
  - Codebleu: 0.7482
  - Ngram Match Score: 0.7504
  - Weighted Ngram Match Score: 0.7583
  - Syntax Match Score: 0.7673
  - Dataflow Match Score: 0.7169
  - Chrf: 93.1064
  
  ## Model description
  
  DistilGPT2 fine-tuned on HearthStone dataset for 200 epochs. \
  Related to [dvitel/h0](https://huggingface.co/dvitel/h0) but with preprocessing which anonymizes classes and function variables (Local renaming). \
  [dvitel/h2](https://huggingface.co/dvitel/h2) implements global renaming where all names are removed. Global renaming showed worse results compared to local renaming.
  
  Example of generated code with mistake on last eval iteration (EV L - gold labels, EV P - prediction):
  ```python
  EV L class CLS0(MinionCard):
          def __init__(self):
                  super().__init__('Darkscale Healer', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Heal(2), CharacterSelector()))
          def create_minion(self, v0):
                  return Minion(4, 5)
  EV P class CLS0(MinionCard):
          def __init__(self):
                  super().__init__('Darkscale Healer', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Heal(2), CharacterSelector())
          def create_minion(self, v0):
                  return Minion(4, 5)
  
  EV L class CLS0(WeaponCard):
          def __init__(self):
                  super().__init__('Fiery War Axe', 2, CHARACTER_CLASS.WARRIOR, CARD_RARITY.FREE)
          def create_weapon(self, v0):
                  return Weapon(3, 2)
  EV P class CLS0(WeaponCard):
          def __init__(self):
                  super().__init__('Fiery War Axe', 2, CHARACTER_CLASS.WARRIOR, CARD_RARITY.FREE,
          def create_weapon(self, v0):
                  return Weapon(3, 2)
  
  EV L class CLS0(MinionCard):
          def __init__(self):
                  super().__init__('Frostwolf Warlord', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Give([Buff(ChangeAttack(Count(MinionSelector()))), Buff(ChangeHealth(Count(MinionSelector())))]), SelfSelector()))
          def create_minion(self, v0):
                  return Minion(4, 4)
  EV P class CLS0(MinionCard):
          def __init__(self):
                  super().__init__('Frostwolf Warlord', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Give([Buff(ChangeAttack(Count(MinionSelector(),), Buff(ChangeHealth(Count(MinionSelector()))))]),), SelfSelector()))
          def create_minion(self, v0):
                  return Minion(4, 4)
  
  EV L class CLS0(SpellCard):
          def __init__(self):
                  super().__init__('Hellfire', 4, CHARACTER_CLASS.WARLOCK, CARD_RARITY.FREE)
          def use(self, v0, v1):
                  super().use(v0, v1)
                  v2 = copy.copy(v1.other_player.minions)
                  v2.extend(v1.current_player.minions)
                  v2.append(v1.other_player.hero)
                  v2.append(v1.current_player.hero)
                  for v3 in v2:
                          v3.damage(v0.effective_spell_damage(3), self)
  EV P class CLS0(SpellCard):
          def __init__(self):
                  super().__init__('Hellfire', 4, CHARACTER_CLASS.WARLOCK, CARD_RARITY.FREE,
          def use(self, v0, v1):
                  super().use(v0, v1)
                  v2 = copy.copy(v1.other_player.minions)
                  v2.extend(v1.current_player.minions)
                  for.append(v1.other_player.hero)
                  for.append(v1.other_player.hero)
                  for v3 in v2:
                                  .damage(v0.effective_spell_damage(3), self)
  ```
  
  ## Intended uses & limitations
  
  HearthStone card code synthesis. 
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 2e-05
  - train_batch_size: 4
  - eval_batch_size: 4
  - seed: 17
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - num_epochs: 200
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch  | Step  | Validation Loss | Exact M