---
local: bertjapanese
sections:
- local: overview
  title: Overview
- local: transformers.BertJapaneseTokenizer
  title: BertJapaneseTokenizer
title: BertJapanese
---
<script>
import Tip from "./Tip.svelte";
import Youtube from "./Youtube.svelte";
import Docstring from "./Docstring.svelte";
import CodeBlock from "./CodeBlock.svelte";
import CodeBlockFw from "./CodeBlockFw.svelte";
import DocNotebookDropdown from "./DocNotebookDropdown.svelte";
import IconCopyLink from "./IconCopyLink.svelte";
export let fw: "pt" | "tf"
</script>
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

<h1 id="bertjapanese">BertJapanese</h1>

<h2 id="overview">Overview</h2>

The BERT models trained on Japanese text.

There are models with two different tokenization methods:

- Tokenize with MeCab and WordPiece. This requires some extra dependencies, [fugashi](https://github.com/polm/fugashi) which is a wrapper around [MeCab](https://taku910.github.io/mecab/).
- Tokenize into characters.

To use *MecabTokenizer*, you should `pip install transformers["ja"]` (or `pip install -e .["ja"]` if you install
from source) to install dependencies.

See [details on cl-tohoku repository](https://github.com/cl-tohoku/bert-japanese).

Example of using a model with MeCab and WordPiece tokenization:

```python
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer

>>> bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese")
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese")

>>> ## Input Japanese Text
>>> line = "吾輩は猫である。"

>>> inputs = tokenizer(line, return_tensors="pt")

>>> print(tokenizer.decode(inputs["input_ids"][0]))
[CLS] 吾輩 は 猫 で ある 。 [SEP]

>>> outputs = bertjapanese(**inputs)
```

Example of using a model with Character tokenization:

```python
>>> bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese-char")
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-char")

>>> ## Input Japanese Text
>>> line = "吾輩は猫である。"

>>> inputs = tokenizer(line, return_tensors="pt")

>>> print(tokenizer.decode(inputs["input_ids"][0]))
[CLS] 吾 輩 は 猫 で あ る 。 [SEP]

>>> outputs = bertjapanese(**inputs)
```

Tips:

- This implementation is the same as BERT, except for tokenization method. Refer to the [documentation of BERT](bert) for more usage examples.

This model was contributed by [cl-tohoku](https://huggingface.co/cl-tohoku).

<h2 id="transformers.BertJapaneseTokenizer">BertJapaneseTokenizer</h2>

<div class="docstring">

<docstring><name>class transformers.BertJapaneseTokenizer</name><anchor>transformers.BertJapaneseTokenizer</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L72</source><parameters>[{"name": "vocab_file", "val": ""}, {"name": "do_lower_case", "val": " = False"}, {"name": "do_word_tokenize", "val": " = True"}, {"name": "do_subword_tokenize", "val": " = True"}, {"name": "word_tokenizer_type", "val": " = 'basic'"}, {"name": "subword_tokenizer_type", "val": " = 'wordpiece'"}, {"name": "never_split", "val": " = None"}, {"name": "unk_token", "val": " = '[UNK]'"}, {"name": "sep_token", "val": " = '[SEP]'"}, {"name": "pad_token", "val": " = '[PAD]'"}, {"name": "cls_token", "val": " = '[CLS]'"}, {"name": "mask_token", "val": " = '[MASK]'"}, {"name": "mecab_kwargs", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **vocab_file** (`str`) --
  Path to a one-wordpiece-per-line vocabulary file.
- **do_lower_case** (`bool`, *optional*, defaults to `True`) --
  Whether to lower case the input. Only has an effect when do_basic_tokenize=True.
- **do_word_tokenize** (`bool`, *optional*, defaults to `True`) --
  Whether to do word tokenization.
- **do_subword_tokenize** (`bool`, *optional*, defaults to `True`) --
  Whether to do subword tokenization.
- **word_tokenizer_type** (`str`, *optional*, defaults to `"basic"`) --
  Type of word tokenizer.
- **subword_tokenizer_type** (`str`, *optional*, defaults to `"wordpiece"`) --
  Type of subword tokenizer.
- **mecab_kwargs** (`str`, *optional*) --
  Dictionary passed to the `MecabTokenizer` constructor.</paramsdesc><paramgroups>0</paramgroups></docstring>

Construct a BERT tokenizer for Japanese text, based on a MecabTokenizer.




</div>
