Answers.Body: <p>Of course now there has been a huge development:
  Huggingface published <a href="https://github.com/huggingface/pytorch-transformers" rel="nofollow noreferrer">pytorch-transformers</a>, a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including many pretrained (mostly English or multilingual) models (<a href="https://huggingface.co/pytorch-transformers/" rel="nofollow noreferrer">docs here</a>). It also includes one German BERT model. SpaCy offers a <a href="https://github.com/explosion/spacy-pytorch-transformers" rel="nofollow noreferrer">convenient wrapper</a> (<a href="https://explosion.ai/blog/spacy-pytorch-transformers" rel="nofollow noreferrer">blog post</a>).</p>
  
  <p>Update: Now, Salesforce published the English model <a href="https://github.com/salesforce/ctrl" rel="nofollow noreferrer">CTRL</a>, which allows for use of "control codes" that influence the style, genre and content of the generated text.</p>
  
  <p>For completeness, here is the old, now less relevant version of my answer:</p>
  
  <hr>
  
  <p>Since I posed the question, I found this <strong>pretrained German language model</strong>:
  <a href="https://lernapparat.de/german-lm/" rel="nofollow noreferrer">https://lernapparat.de/german-lm/</a></p>
  
  <p>It is an instance of a <a href="https://github.com/Bachfischer/german2vec" rel="nofollow noreferrer">3-layer "averaged stochastic descent weight-dropped" LSTM</a> which was implemented based on an implementation by <a href="https://github.com/salesforce/awd-lstm-lm" rel="nofollow noreferrer">Salesforce</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-17T18:54:35.393Z
Answers.Text: As a note: `transformers` (previously known as `pytorch-transformers`) currently also supports Tensorflow.
Answers.UserDisplayName: Bram Vanroy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-29T13:52:59.557Z
Answers.LastActivityDate: 2019-09-23T06:47:46.260Z
Answers.LastEditDate: 2019-09-23T06:47:46.260Z
Answers.UserDisplayName: Lutz Büch
Answers.Body: <p>This will depend to some extent on what you want to do with the language models.</p>
  
  <p>Some possible resources are:</p>
  
  <p>TensorFlow offers 3 pre-trained language models in the <a href="https://github.com/tensorflow/models/tree/master/research" rel="nofollow noreferrer">research package</a>. </p>
  
  <p>Cafe's <a href="https://github.com/BVLC/caffe/wiki/Model-Zoo" rel="nofollow noreferrer">ModelZoo</a> has a single pre-trained model that does video -> captions.</p>
  
  <p>Other packages like Cafe2  offer pre-trained models, but the documentation does not suggest any of them are suitable for language.</p>
  
  <p>Failing this, a good approach might be to email the authors of a paper that adopts an approach you like. Some (but far from all) researchers will be happy to share their models, which you can then use as a starting point for your own.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-27T06:31:39.280Z
Answers.Text: I mean __language model__ in the [standard sense](https://en.wikipedia.org/wiki/Language_model). I updated my question accordingly.
Answers.UserDisplayName: Lutz Büch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-27T09:27:02.987Z
Answers.Text: Hmm. It looks like you found a few. I suspect DuttA is correct: if these work well, they are extremely valuable right now. No one is likely to publish then free of change. Your best bet is to contact the authors of relevant papers and see if they'll share something with you.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T15:06:42.263Z
Answers.Text: Yeah, in the meantime, HuggingFace has established https://huggingface.co/models, where many research papers and other projects publish their models.
Answers.UserDisplayName: Lutz Büch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-25T13:08:58.910Z
Answers.LastActivityDate: 2018-08-25T13:08:58.910Z
Answers.UserDisplayName: John Doucette
Body: <p>Where can I find (more) pre-trained <a href="https://en.wikipedia.org/wiki/Language_model" rel="nofollow noreferrer">language models</a>? I am especially interested in <strong>neural network-based</strong> models for <strong>English and German</strong>.</p>
  
  <p>I am aware only of <a href="https://github.com/tensorflow/models/tree/master/research/lm_1b" rel="nofollow noreferrer">Language Model on One Billion Word Benchmark</a> and <a href="https://github.com/lverwimp/tf-lm" rel="nofollow noreferrer">TF-LM: TensorFlow-based Language Modeling Toolkit</a>.</p>
  
  <p>I am surprised not to find a greater wealth of models for different frameworks and languages.</p>
  
Tags: <neural-networks><natural-language-processing><bert><gpt><language-model>
Title: Where can I find pre-trained language models in English and German?
UserDisplayName: Lutz Büch
ThreadText: # Where can I find pre-trained language models in English and German?
  
  (Asked by: username_0 on 2018-08-23)
  
  Where can I find (more) pre-trained [language models](https://en.wikipedia.org/wiki/Language_model)? I am especially interested in **neural network-based** models for **English and German**.
  
  I am aware only of [Language Model on One Billion Word Benchmark](https://github.com/tensorflow/models/tree/master/research/lm_1b) and [TF-LM: TensorFlow-based Language Modeling Toolkit](https://github.com/lverwimp/tf-lm).
  
  I am surprised not to find a greater wealth of models for different frameworks and languages.
  
  **Tags:** neural-networks, natural-language-processing, bert, gpt, language-model
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-08-29. Score: 1 | Accepted answer
  
  Of course now there has been a huge development:
  Huggingface published [pytorch-transformers](https://github.com/huggingface/pytorch-transformers), a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including many pretrained (mostly English or multilingual) models ([docs here](https://huggingface.co/pytorch-transformers/)). It also includes one German BERT model. SpaCy offers a [convenient wrapper](https://github.com/explosion/spacy-pytorch-transformers) ([blog post](https://explosion.ai/blog/spacy-pytorch-transformers)).
  
  Update: Now, Salesforce published the English model [CTRL](https://github.com/salesforce/ctrl), which allows for use of "control codes" that influence the style, genre and content of the generated text.
  
  For completeness, here is the old, now less relevant version of my answer:
  
  ---
  
  Since I posed the question, I found this **pretrained German language model**:
  <https://lernapparat.de/german-lm/>
  
  It is an instance of a [3-layer "averaged stochastic descent weight-dropped" LSTM](https://github.com/Bachfischer/german2vec) which was implemented based on an implementation by [Salesforce](https://github.com/salesforce/awd-lstm-lm).
  
  > **Comments:**
  > **username_1**: As a note: `transformers` (previously known as `pytorch-transformers`) currently also supports Tensorflow.  
  
  ---
  
  ### Answer by username_2 on 2018-08-25. Score: 0
  
  This will depend to some extent on what you want to do with the language models.
  
  Some possible resources are:
  
  TensorFlow offers 3 pre-trained language models in the [research package](https://github.com/tensorflow/models/tree/master/research).
  
  Cafe's [ModelZoo](https://github.com/BVLC/caffe/wiki/Model-Zoo) has a single pre-trained model that does video -> captions.
  
  Other packages like Cafe2 offer pre-trained models, but the documentation does not suggest any of them are suitable for language.
  
  Failing this, a good approach might be to email the authors of a paper that adopts an approach you like. Some (but far from all) researchers will be happy to share their models, which you can then use as a starting point for your own.
  
  > **Comments:**
  > **username_0**: I mean __language model__ in the [standard sense](https://en.wikipedia.org/wiki/Language_model). I updated my question accordingly.  
  > **username_2**: Hmm. It looks like you found a few. I suspect DuttA is correct: if these work well, they are extremely valuable right now. No one is likely to publish then free of change. Your best bet is to contact the authors of relevant papers and see if they'll share something with you.  
  > **username_0**: Yeah, in the meantime, HuggingFace has established https://huggingface.co/models, where many research papers and other projects publish their models.  
  
  ---
  
  