Is there an implementation of BERT architecture in PyTorch that I can modify here?
Hello Team,

  Firstly, thanks for this amazing repo.

I am doing my own research and I want access to a native implementation of BERT in PyTorch so I can modify the architecture and play with it by including a few of my own modules.

Is that possible with the codebase in HuggingFace repo here?

Thanks
Hi! Yes, you can modify the BERT architecture as you please, it's self contained. It's in the [modeling_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py) file.<|||||>Thanks @LysandreJik , can you also confirm if this implementation supports multi gpu training? <|||||>This implementation is a PyTorch model, so it supports everything a PyTorch model can do :) GPU, Multi-GPU, TPU, you name it.<|||||>Hello @LysandreJik  and team, I am looking at the script `run_language_modeling.py` at https://github.com/huggingface/transformers/tree/master/examples/language-modeling . 

I saw that the example uses WikiText-2 dataset for example. If I want to fine-tune BERT on my own dataset, how should the dataset be structured? Should I mask the words myself or is there some DataLoader that will do that?

I downloaded the WikiText data and I saw an example chunk of text is 
```
 = Robert <unk> =

 Robert <unk> is an English film , television and theatre actor . He had a guest @-@ starring role on the television series The Bill in 2000 . This was followed by a starring role in the play Herons written by Simon Stephens , which was performed in 2001 at the Royal Court Theatre . He had a guest role in the television series Judge John <unk> in 2002 . In 2004 <unk> landed a role as " Craig " in the episode " Teddy 's Story " of the television series The Long Firm ; he starred alongside actors Mark Strong and Derek Jacobi . He was cast in the 2005 theatre productions of the Philip Ridley play Mercury Fur , which was performed at the Drum Theatre in Plymouth and the <unk> <unk> Factory in London . He was directed by John <unk> and starred alongside Ben <unk> , Shane <unk> , Harry Kent , Fraser <unk> , Sophie Stanton and Dominic Hall .
 In 2006 , <unk> starred alongside <unk> in the play <unk> written by Mark <unk> . He appeared on a 2006 episode of the television series , Doctors , followed by a role in the 2007 theatre production of How to Curse directed by <unk> <unk> . How to Curse was performed at Bush Theatre in the London Borough of <unk> and Fulham . <unk> starred in two films in 2008 , <unk> <unk> by filmmaker Paris <unk> , and <unk> Punch directed by <unk> Blackburn . In May 2008 , <unk> made a guest appearance on a two @-@ part episode arc of the television series Waking the Dead , followed by an appearance on the television series <unk> in November 2008 . He had a recurring role in ten episodes of the television series <unk> in 2010 , as " <unk> Fletcher " . <unk> starred in the 2011 film <unk> directed by Paris <unk> .

 = = Career = =
```

In my case I have a large set of text files. Just text files with free text inside it. Can someone point me to a document/resource that lets me understand how should the input be for masked language modelling pretraining using BERT?

I plan to use the `https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py` file contents, modify the layers a bit based on my architecture decisions and train it on my own dataset using masked language modeling where random words are masked and I predict them back.

Any help is appreciated. Thanks<|||||>Hi @abhisheksgumadi, this is a very interesting and rather broad question. Could you ask it on the forums over on https://discuss.huggingface.co? Thanks a lot!