readme: # Dialog-KoELECTRA
  
  Github : [https://github.com/skplanet/Dialog-KoELECTRA](https://github.com/skplanet/Dialog-KoELECTRA)
  
  ## Introduction
  
  **Dialog-KoELECTRA** is a language model specialized for dialogue. It was trained with 22GB colloquial and written style Korean text data. Dialog-ELECTRA model is made based on the [ELECTRA](https://openreview.net/pdf?id=r1xMH1BtvB) model. ELECTRA is a method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. 
  
  <br>
  
  ## Released Models
  
  We are initially releasing small version pre-trained model.
  The model was trained on Korean text. We hope to release other models, such as base/large models, in the future.
  
  | Model | Layers | Hidden Size | Params | Max<br/>Seq Len | Learning<br/>Rate | Batch Size | Train Steps  |
  | :---: | :---: | :---: | :---: | :---:  | :---: | :---:  | :---:  |
  | Dialog-KoELECTRA-Small | 12 | 256 | 14M | 128 | 1e-4 | 512 | 700K |
  
  <br>
  
  ## Model Performance
  
  Dialog-KoELECTRA shows strong performance in conversational downstream tasks.
  
  |               | **NSMC**<br/>(acc) | **Question Pair**<br/>(acc) | **Korean-Hate-Speech**<br/>(F1) | **Naver NER**<br/>(F1) | **KorNLI**<br/>(acc) | **KorSTS**<br/>(spearman) | 
  | :--------------------- | :----------------: | :--------------------: | :----------------: | :------------------: | :-----------------------: | :-------------------------: | 
  | DistilKoBERT           |       88.60        |            92.48          |                 60.72                 |      84.65          |        72.00         |           72.59           |   
  | **Dialog-KoELECTRA-Small** |     **90.01**      |           **94.99**             |               **68.26**               |  **85.51**         |      **78.54**       |         **78.96**         |    
  
  <br>
  
  ## Train Data
  
  
  <table class="tg">
  <thead>
    <tr>
      <th class="tg-c3ow"></th>
      <th class="tg-c3ow">corpus name</th>
      <th class="tg-c3ow">size</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="tg-c3ow" rowspan="4">dialog</td>
      <td class="tg-0pky"><a href="https://aihub.or.kr/aidata/85" target="_blank" rel="noopener noreferrer">Aihub Korean dialog corpus</a></td>
      <td class="tg-c3ow" rowspan="4">7GB</td>
    </tr>
    <tr>
      <td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Spoken corpus</a></td>
    </tr>
    <tr>
      <td class="tg-0pky"><a href="https://github.com/songys/Chatbot_data" target="_blank" rel="noopener noreferrer">Korean chatbot data</a></td>
    </tr>
    <tr>
      <td class="tg-0pky"><a href="https://github.com/Beomi/KcBERT" target="_blank" rel="noopener noreferrer">KcBERT</a></td>
    </tr>
    <tr>
      <td class="tg-c3ow" rowspan="2">written</td>
      <td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Newspaper corpus</a></td>
      <td class="tg-c3ow" rowspan="2">15GB</td>
    </tr>
    <tr>
      <td class="tg-0pky"><a href="https://github.com/lovit/namuwikitext" target="_blank" rel="noopener noreferrer">namuwikitext</a></td>
    </tr>
  </tbody>
  </table>
  
  <br>
  
  ## Vocabulary
  
  We applied morpheme analysis using [huggingface_konlpy](https://github.com/lovit/huggingface_konlpy) when creating a vocabulary dictionary.
  As a result of the experiment, it showed better performance than a vocabulary dictionary created without applying morpheme analysis.
  <table>
  <thead>
    <tr>
      <th>vocabulary size</th>
      <th>unused token size</th>
      <th>limit alphabet</th>
      <th>min frequency</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>40,000</td>
      <td>500</td>
      <td>6,000</td>
      <td>3</td>
    </tr>
  </tbody>
  </table>
  
  <br>
  