readme: ---
  language: vi
  tags:
  - bert
  - roberta
  - xlmroberta
  - vietnam
  - vietnamese
  - wiki
  license: apache-2.0
  datasets:
  - wikipedia
  ---
  
  # Vi-XLM-RoBERTa base model (uncased)
  MODEL IS BEING TRAINED (Pending...)
  - Progress: ▓▓░░░░░░░░ 25.00%
  - Epochs: 10/40 (2022-12-06)
  
  ## Logs
  <a href="https://wandb.ai/anhdungitvn/test/reports/perplexity-22-10-21-21-03-10---VmlldzoyODMwOTQ3?accessToken=lk98cqcsbmcdl4ftjw8vu2bfoc6ifigal1p2db3zo27cc7xwzedfivk3o4aeei3h">Perplexity</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/Training-loss-22-10-21-21-04-12---VmlldzoyODMwOTU4?accessToken=8kfl35d92xxa1eockvrf08p8r65zhlrm1dxfl7vhu5avc3cfozl897n030t1q1f6">Train Loss</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/eval_loss-22-10-21-21-04-24---VmlldzoyODMwOTYx?accessToken=3efesxgcurawkzl93anh4w4hz28c1fl5oexrnwb0mo0pz118fz6mappkpr434tp9">Eval Loss</a> | 
  <a href="https://wandb.ai/anhdungitvn/test/reports/lr-22-10-21-21-03-57---VmlldzoyODMwOTU1?accessToken=ggggf7fe4c4rvhexuonybf3m6mhrdn8tz7o4qjqwbml6k0h9f0953zpsbrympktq">Learning Rate</a> | 
  <a href="https://wandb.ai/anhdungitvn/test/reports/global_step-22-10-21-21-04-36---VmlldzoyODMwOTYy?accessToken=cjd65vgz6uviswqqi5tbud14g9v4bxhjpuccal4hqmrm1h4vqfxithrf4wg0u7xq">Global Step</a>
  
  ## Gradients by layers
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-embeddings-LayerNorm-weight-22-12-06-20-31-20---VmlldzozMDk0NTkx?accessToken=vepursxxx3s66rvrnwtiep9jhx8vjkoid1oyk3gd83b6ertq6r70qhotjg9ne36s">Embedding</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-0-attention-output-LayerNorm-weight-22-12-06-20-30-40---VmlldzozMDk0NTg4?accessToken=x975ir3hd7smhqm64dahbgtv76uwklmgf87j7izkkjbcgu41ed9zcfkk3rzb1nmm">L0-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-1-attention-output-LayerNorm-weight-22-12-06-20-30-24---VmlldzozMDk0NTg2?accessToken=gwqxn1asm08k2jrdhnpa2oiprley2cmpqgnazjxctorq53yfb0zq5yqq8p46ubzq">L1-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-2-attention-output-LayerNorm-weight-22-12-06-20-30-09---VmlldzozMDk0NTgz?accessToken=b6kr19k4qutc12i1hsfki4jryr8ou5pzbmlkvyq6zwxj25tw1txe5vsd8t4li4i3">L2-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-3-attention-output-LayerNorm-weight-22-12-06-20-29-49---VmlldzozMDk0NTgx?accessToken=12klldmz2xpfjcw0bi6a5ggqe3ebtcdq9yismzaz562xajxb696zqcgvsdxp3ext">L3-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-4-attention-output-LayerNorm-weight-22-12-06-20-29-28---VmlldzozMDk0NTc2?accessToken=qgbz1njt7vwi0aspw7bh4n6ad4i0bctctgedq7qmsd4lds9l758ibbm5g4ndfs9f">L4-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-5-attention-output-LayerNorm-weight-22-12-06-20-29-11---VmlldzozMDk0NTc1?accessToken=wytlftyuujtgpmxxwk6q3kd30zjre3qiloa6w6h1aw6eb2169udu3lwtgbbyhd8x">L5-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-6-attention-output-LayerNorm-weight-22-12-06-20-28-53---VmlldzozMDk0NTcx?accessToken=k2ni3ba1xx50vpeto3xpnn1hc0ypvxd7im5mrk1pq47krl64fc765pjr1zt14eyc">L6-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-7-attention-output-LayerNorm-weight-22-12-06-20-28-37---VmlldzozMDk0NTY3?accessToken=kwsawb1wu60ecq1pnvtiim7sumqqca9uqn7rcv2o0ize794h49cgvchv680nju1z">L7-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-8-attention-output-LayerNorm-weight-22-12-06-20-28-20---VmlldzozMDk0NTY0?accessToken=uaqu2tqx39ki634m6j3fywcs76tzyp3q9i1riw9af52pehym9p3ukyamsyldq6te">L8-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-9-attention-output-LayerNorm-weight-22-12-06-20-27-57---VmlldzozMDk0NTYz?accessToken=8fvpijkrfvaywa909woxz38qa0dj8jsm6wyjd3tpubemoakzf3u84p3yel3g5b4a">L9-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-10-attention-output-LayerNorm-weight-22-12-06-20-26-45---VmlldzozMDk0NTU3?accessToken=qv18nauts75orcu6b3lpt227g3aa88wfnpy2lnckwn771pp5j4fn270lrnrxvaie">L10-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-roberta-encoder-layer-11-attention-output-LayerNorm-weight-22-12-06-20-25-42---VmlldzozMDk0NTUz?accessToken=6xc2nhzbbctc7o0h8ztaspy8m1ha9r1qoldav2e1rilvwl1yu48frels9134ittl">L11-Attention</a> |
  <a href="https://wandb.ai/anhdungitvn/test/reports/gradients-lm_head-layer_norm-weight-22-12-06-20-32-36---VmlldzozMDk0NTk4?accessToken=t2zroq4rqqu12sak0v8hohmkkqgjuqfgni6apqtearwxa01d3yppglhtux91y4ga">LM Head</a> 
  
  
  This model is Vietnamese XLM Robert, base, uncased. 
  
  ## Model description
  
  
    
  ## Intended uses & limitations
  
  You can use the raw model for either masked language modeling, but it's mostly intended to
  be fine-tuned on a downstream task. 
  
  ### How to