readme: # 基于trocr(beit+roberta)实现对中文场景文字识别
  trocr原地址(https://github.com/microsoft/unilm/tree/master/trocr)
  ## 实现功能
  - [x]  单行/多行文字/横竖排文字识别
  - [x]  不规则文字（印章，公式等）
  - [ ]  表格识别
  - [ ]  模型蒸馏/DML(协作学习)
  - [ ]  Prompt Learning
  ## 环境编译
  ```
  docker build --network=host -t trocr-chinese:latest .
  docker run --gpus all -it -v /tmp/trocr-chinese:/trocr-chinese trocr-chinese:latest bash
  ```
  1. Set up Python for macOS, see https://developer.apple.com/metal/tensorflow-plugin/
  2. Install requirements `python -m pip install -r requirements.txt`
  3. Install Pillow `python -m pip install pillow`
  4. Upgrade Numpy `python -m pip install numpy --upgrade`
  5. Install PyTorch `conda install pytorch torchvision torchaudio -c pytorch`
  6. Set envioronmental variable so PyTorch can fallback to CPU: `conda env config vars set PYTORCH_ENABLE_MPS_FALLBACK=1`
  7. Reactivate environment: `source ~/miniconda/bin/activate`
  8. Generate custom vocab:
  ```
  python gen_vocab.py \
         --dataset_path "dataset/*/*.txt" \
         --cust_vocab ./cust-data/vocab.txt
  ```
  9. Download pretrained weights from https://pan.baidu.com/s/1rARdfadQlQGKGHa3de82BA, password: 0o65
  10. Initialize weights for fine-tuning
  Make sure you are using 4.15.0 version of transformers by running `pip install transformers==4.15.0`.
  ```
  python init_custdata_model.py --cust_vocab ./cust-data/vocab.txt --pretrain_model ./weights --cust_data_init_weights_path ./cust-data/weights
  ```
  11. Train
  To enable M1 GPU support, install the dev version of transformers by running `pip install git+https://github.com/huggingface/transformers`.
  ```
  pip install git+https://github.com/huggingface/transformers@3be028bc9d4b2cce9539b940f17052f333284684
  ```
  In Dec 21, 2022, the dev version that's working for me is `transformers-4.26.0.dev0`. Later stable releases may have M1 GPU support built-in so you don't need to install the dev version.
  If you are running the whole procedure again, remember to reinstall the older transformers version as instructed in step 10. Otherwise, the weights initialized will not be in the correct format and you will see miserable accuracy rate, likely due to breaking changes involving how tokenization is done.
  ```
  python train.py --cust_data_init_weights_path ./cust-data/weights --checkpoint_path ./checkpoint/trocr-custdata --dataset_path "./dataset/*/*.jpg" --per_device_train_batch_size 8
  ```
  
  ## Optimize inference
  Install dependencies
  ```
  python -m pip install optimum
  conda install onnxruntime -c conda-forge
  ```
  
  Convert to ONNX
  ```
  python -m transformers.onnx --model=checkpoint/trocr-custdata-8000/last --feature=vision2seq-lm onnx/ --atol 1e-3
  ```
  
  ## 训练
  ### 初始化模型到自定义训练数据集
  #### 字符集准备参考cust-data/vocab.txt
  ```
  vocab.txt
  1
  2
  ...
  a
  b
  c
  ```
  ```[python]
  python gen_vocab.py \
         --dataset_path "dataset/cust-data/0/*.txt" \
         --cust_vocab ./cust-data/vocab.txt
  
  ```
  ### 初始化自定义数据集模型
  #### 下载预训练模型trocr模型权重
  链接: https://pan.baidu.com/s/1rARdfadQlQGKGHa3de82BA  密码: 0o65
  ```
  python init_custdata_model.py \   
      --cust_vocab ./cust-data/vocab.txt \  
      --pretrain_model ./weights \
      --cust_data_init_weights_path ./cust-data/weights
      
  ## cust_vocab 词库文件   
  ## pretrain_model 预训练模型权重   
  ## cust_data_init_weights_path 自定义模型初始化模型权重保存位置   
  
  ```
  
  ### 训练模型
  #### 数据准备,数据结构如下图所示
  ```
  dataset/cust-data/0/0.jpg
  dataset/cust-data/0/0.txt
  ...
  dataset/cust-data/100/10000.jpg
  dataset/cust-data/100/10000.txt
  ```
  
  #### 训练模型
  ```
  python train.py \
         --cut_data_init_weights_path ./cust-data/weights \
         --checkpoint_path ./checkpoint/trocr-custdata \
         --dataset_path "./dataset/cust-data/*/*.jpg" \
         --per_device_train_batch_size 8 \
         --CUDA_VISIBLE_DEVICES 1
  ```
  
  #### 评估模型
  ##### 拷贝checkpoint/trocr-custdata训练完成的pytorch_model.bin 到 ./cust-data/weights 目录下
  
  ```[python]
  python eval.py \
      --dataset_path "./data/cust-data/test/*/*.jpg" \
      --cust_data_init_weights_path ./cust-data/weights    
  ```
  
  ## 测试模型
  ```
  ## 拷贝训练完成的pytorch_model.bin 到 ./cust-data/weights 目录下
  index = 2300 ##选择最好的或者最后一个step模型
  cp ./checkpoint/trocr-custdata/checkpoint-$index/pytorch_model.bin ./cust-data/weights
  python app.py --cust_data_init_weights_path ./cust-data/weights --test_img test/test.jpg
  ```
  
  ## 预训练模型
  | 模型        | cer(字符错误率)           | acc(文本行)  | 下载地址  |训练数据来源 |训练耗时(GPU:3090) | 
  | ------------- |:-------------:| -----:|-----:|-----:|-----:|
  | hand-write(中文手写)      |0.011 | 0.940 |[hand-write](https://pan.baidu.com/s/19f7iu9tLHkcT_zpi3UfqLQ)  密码: punl |[数据集地址](https://aistudio.baidu.com/aistudio/datasetdetail/102884/0) |8.5h(10epoch)|
  | seal-ocr(印章识别)      |0.006 | 0.956 | 整理后开放下载|- |
  | im2latex(数学公式识别)      |- | - |- |[im2latex](https://zenodo.org/record/56198#.YkniL25Bx_S) ||
  | TAL_OCR_TABLE(表格识别)     |- | - |- |[TAL_OCR_TABLE](https://ai.100tal.com/dataset) |
  | TAL_OCR_MATH(小学低年级算式数据集)|- | - |- | [TAL_OCR_MATH](https://ai.100tal.com/dataset) |
  | TAL_OCR_CHN(手写中文数据集)|0.0455 | 0.674（标注质量不太高,例如：test_64/552.jpg 标注值:蝶恋花, 实际值:欧阳修  ） |[TAL_OCR_CHN](https://pan.baidu.com/s/1GbyVpbiKyVN1nJipvH2fpA)  密码: 9kd8 |