readme: ---
  language: 
  - ja
  thumbnail: 
  tags:
  - xlnet
  - lm-head
  - causal-lm
  license:
  - apache-2.0
  datasets:
  - Japanese_Business_News
  metrics:
  ---
  
  # XLNet-japanese
  
  ## Model description
  This model require Mecab and senetencepiece with XLNetTokenizer.
  See details https://qiita.com/mkt3/items/4d0ae36f3f212aee8002
  
  This model uses NFKD as the normalization method for character encoding.
  Japanese muddle marks and semi-muddle marks will be lost.
  
  *日本語の濁点・半濁点がないモデルです*
  
  #### How to use
  
  ```python
  from fugashi import Tagger 
  
  from transformers import (
      pipeline,
      XLNetLMHeadModel,
      XLNetTokenizer
  )
  
  class XLNet():
      def __init__(self):
          self.m = Tagger('-Owakati') 
          self.gen_model = XLNetLMHeadModel.from_pretrained("hajime9652/xlnet-japanese")
          self.gen_tokenizer = XLNetTokenizer.from_pretrained("hajime9652/xlnet-japanese")
           
      def generate(self, prompt="福岡のご飯は美味しい。コンパクトで暮らしやすい街。"):
          prompt = self.m.parse(prompt)
          inputs = self.gen_tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
          prompt_length = len(self.gen_tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
          outputs = self.gen_model.generate(inputs, max_length=200, do_sample=True, top_p=0.95, top_k=60)
          generated = prompt + self.gen_tokenizer.decode(outputs[0])[prompt_length:]
          return generated
  ```
  
  #### Limitations and bias
  This model's training use the Japanese Business News.
  
  
  # Important matter
  The company that created and published this model is called Stockmark.
  This repository is for use by HuggingFace and not for infringement.
  See this documents https://qiita.com/mkt3/items/4d0ae36f3f212aee8002
  published by https://github.com/mkt3
  