github.com/pytorch/fairseq
  

  
    
  
	  
  

  
    
      In the vocab of bart.bpe.bpe.decoder,  what does Ġ mean for those words prefixed with 'Ġ'? 17
    

    
      
        opened 
        
          
        
        Feb 17, 2020
      
      

        
          closed 
        
          
        
        Feb 21, 2020
      
        

      
        
          
          v381654729
        
      
    

    
        question
    
  


  
    ## ❓ Questions and Help


#### What is your question?

In the vocab of bart….bpe.bpe.decoder,  what does Ġ mean for those words prefixed with 'Ġ'?

#### Code

    keywords = { "checkpoint_file" : "model.pt"}
    bart = BARTModel.from_pretrained(config_translation_zh_en.bart_english_directory,   **keywords )
    # bart.bpe.bpe.decoder is a dict, and it contains many 'strange' words like 'Ġthe' 'Ġand' 'Ġof' and also many normal words like 'playing' 'bound' etc. 
![image](https://user-images.githubusercontent.com/43415394/74647258-52024680-51b6-11ea-97dd-4631e0404d17.png)


bart.bpe.bpe.decoder is a dict, and it contains many 'strange' words like 'Ġthe' 'Ġand' 'Ġof' and also many normal words like 'playing' 'bound' etc. At first glance, words prefixed with Ġ seem to be complete words and words not prefixed with Ġ are incomplete words (like BertTokenizer's ##-prefixed words ) like 'ing' 'th' and ''re'. However, a high proportion of words not prefixed with Ġ seem not incomplete words, like frequency, relation, etc. 

#### Question

What does Ġ mean for those words prefixed with 'Ġ'?  And how does the tokenizer of BART tokenize with the set of 50265-word vocabulary (bart.task.source_dictionary, 50257 (integer) words of which map to bart.bpe.bpe.decoder)? I haven't found documentations on the Internet or in the codes.