How to contribute to “Write with transformer”?
## 🚀 I would like to contribute to a French version of this App

I’m French, I write short stories, and I’m also a software engineer

## Motivation

I’ll retire in 6 months and I wanted to build such an app before I stumbled on your demo.

## Additional context

https://www.linkedin.com/in/mauceri/

What is it that you can contribute? The only (yet impressive) thing that is going on is language modeling. Can you contribute a pre-trained French model for one of the frameworks? That's (as far as I know) the only way to contribute. <|||||>Thanks Bram, I’m going to investigate what the cost could be for XLNet on clevergrid https://www.clevergrid.io/?pk_campaign=ga-gpu-1&pk_source=adwords&pk_medium=sem&pk_content=gpuasaservicefr&gclid=CjwKCAjwibzsBRAMEiwA1pHZrvm8ozRMrbcDR7YoYiKqsq6gEnPo9AecJwjKzBxa8L-4_hB6ny4uARoCwfMQAvD_BwE

Envoyé de mon iPad

> Le 28 sept. 2019 à 09:44, Bram Vanroy <notifications@github.com> a écrit :
> 
> What is it that you can contribute? The only (yet impressive) thing that is going on is language modeling. Can you contribute a pre-trained French model for one of the frameworks? That's (as far as I know) the only way to contribute.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.
<|||||>Hi all,
We (ovh) are open to calculate it for free.<|||||>Not sure if a new French language model is still necessary after Camembert has been introduced.<|||||>That's awesome news, @jqueguiner! Let us know if we can help.

@BramVanroy To work well with Write With Transformer, we would want more like a FR-pretrained GPT-2-like model. CamemBERT wouldn't do on generation out of the box.

See also the more specific issue: https://github.com/huggingface/transformers/issues/1356<|||||>For generation CamemBERT is of no use I think...

Envoyé de mon iPad

> Le 22 nov. 2019 à 14:02, Bram Vanroy <notifications@github.com> a écrit :
> 
> ﻿
> Not sure if a new French language model is still necessary after Camembert has been introduced.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
<|||||>Yes, CamemBERT is awesome, but for WWT we need a FR-Pretrained GPT-2 model!

Envoyé de mon iPad

> Le 22 nov. 2019 à 14:55, Julien Chaumond <notifications@github.com> a écrit :
> 
> ﻿
> That's awesome news, @jqueguiner! Let us know if we can help.
> 
> @BramVanroy To work well with Write With Transformer, we would want more like a FR-pretrained GPT-2-like model. CamemBERT wouldn't do on generation out of the box.
> 
> See also the more specific issue: #1356
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
<|||||>Camembert doesnt offer generation, only syntax analysis and masking due to the nature of the network. Multiple mask generation (<mask><mask><mask>) gives really uggly results as you cna test here : https://market-place.ai.ovh.net/#!/apis/43323c37-59e7-4092-b23c-3759e7c09288/pages/94d31892-4e64-446f-9318-924e64346f9e

IMO we should start training using OSCAR dataset 
https://traces1.inria.fr/oscar/

@julien-c yes we can start with a collab GPT2 french training ipynb together then I'll prepare the env for a DGX1 or something similar. I didn't train a GPT2 before. IS it scaling over multiple GPU's ? do we need horovod adaptation ?<|||||>Oops, sorry everyone. I thought this was a general French model question. My bad. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
