Tora-12B

• Model was implemented via ChatVector

Tora-12B = cyberagent/Mistral-Nemo-Japanese-Instruct-2408 + (cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b - mistralai/Mistral-Nemo-Base-2407)

Model license

model	license
cyberagent/Mistral-Nemo-Japanese-Instruct-2408	apache-2.0
cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b	apache-2.0
mistralai/Mistral-Nemo-Base-2407	apache-2.0

Benchmark

• Benchmark was conducted via Open Japanese LLM Leaderboard

N-shot	AVG	AVG (NLI)	AVG (QA)	AVG (RC)	AVG (MC)	AVG (EL)	AVG (FA)	AVG (MR)	AVG (MT)	AVG (HE)	AVG (CG)	AVG (SUM)
4	0.5036	0.6189	0.4522	0.8868	0.7748	0.5251	0.1908	0.684	0.8132	0.4971	0.0	0.0972
0	0.3728	0.5746	0.204	0.8228	0.6261	0.0171	0.1148	0.56	0.798	0.2861	0.0	0.0972

Reference

@misc{OJLL,
  author = {Miyao, Yusuke and Ishida, Shigeki and Okamoto, Takumi and Han, Namgi and Mousterou, Akim and Fourrier, Clémentine and Hayashi, Toshihiro and Tachibana, Yuichiro},
  title = {Open Japanese LLM Leaderboard},
  year = {2024},
  publisher = {OJLL},
  howpublished = "\url{https://huggingface.co/spaces/llm-jp/open-japanese-llm-leaderboard}"
}
@misc{llmjp2024llmjpcrossorganizationalprojectresearch,
      title={LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs},
      author={LLM-jp and : and Akiko Aizawa and Eiji Aramaki and Bowen Chen and Fei Cheng and Hiroyuki Deguchi and Rintaro Enomoto and Kazuki Fujii and Kensuke Fukumoto and Takuya Fukushima and Namgi Han and Yuto Harada and Chikara Hashimoto and Tatsuya Hiraoka and Shohei Hisada and Sosuke Hosokawa and Lu Jie and Keisuke Kamata and Teruhito Kanazawa and Hiroki Kanezashi and Hiroshi Kataoka and Satoru Katsumata and Daisuke Kawahara and Seiya Kawano and Atsushi Keyaki and Keisuke Kiryu and Hirokazu Kiyomaru and Takashi Kodama and Takahiro Kubo and Yohei Kuga and Ryoma Kumon and Shuhei Kurita and Sadao Kurohashi and Conglong Li and Taiki Maekawa and Hiroshi Matsuda and Yusuke Miyao and Kentaro Mizuki and Sakae Mizuki and Yugo Murawaki and Ryo Nakamura and Taishi Nakamura and Kouta Nakayama and Tomoka Nakazato and Takuro Niitsuma and Jiro Nishitoba and Yusuke Oda and Hayato Ogawa and Takumi Okamoto and Naoaki Okazaki and Yohei Oseki and Shintaro Ozaki and Koki Ryu and Rafal Rzepka and Keisuke Sakaguchi and Shota Sasaki and Satoshi Sekine and Kohei Suda and Saku Sugawara and Issa Sugiura and Hiroaki Sugiyama and Hisami Suzuki and Jun Suzuki and Toyotaro Suzumura and Kensuke Tachibana and Yu Takagi and Kyosuke Takami and Koichi Takeda and Masashi Takeshita and Masahiro Tanaka and Kenjiro Taura and Arseny Tolmachev and Nobuhiro Ueda and Zhen Wan and Shuntaro Yada and Sakiko Yahata and Yuya Yamamoto and Yusuke Yamauchi and Hitomi Yanaka and Rio Yokota and Koichiro Yoshino},
      year={2024},
      eprint={2407.03963},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.03963},
}