readme: # WudaoSailing
  
  WudaoSailing is a package for pretraining chinese Language Model and finetune tasks. Now it supports GLM, Bert, T5, Cogview and Roberta models.
  
  
  
  
  ## Get Started
  ### Docker Image
  We prepare two docker images based on CUDA 10.2 and CUDA 11.2. You can build images from the docker file [docs/docker/cuda102.dockerfile](docs/docker/cuda102.dcokerfile) or pull the pre-built images from Docker Hub and run with docker v19.03+
    ```shell
    nvidia-docker run -id  --hostname=V100  --network=host\
    --ipc=host --shm-size=16gb --name=deepspeed-cuda \  
    -e NVIDIA_VISIBLE_DEVICES=0,1,2,3 \
    -v /DATA/disk1/docker/containers/:/data deepspeed/cuda102:lastest
    ```
    or replace `cuda102` with `cuda112`.
  
    ```shell
      docker build -f cuda102.dockerfile  -t deepspeed/cuda102 .
    ```
  
  ### Clone this repo
    ```shell
    git clone https://github.com/wangguojim/WudaoSailing.git
    cd WudaoSailing
    pip install -r requirements.txt
    ```
  
  ## GLM
  
  We show some examples based on GLM model.
  
  ### finetuene
  We provide scripts for finetuning GLM on some downstream tasks.
  
  #### SuperGLUE
  
  - Download the [SuperGlue](https://super.gluebenchmark.com/tasks) data and check the experiment setup in 
    [examples/glm/scripts/ds_finetune_superglue.sh](xamples/glm/scripts/ds_finetune_superglue.sh). Note that `DATA_ROOT, CHECKPOINT_PATH, SAVE_PATH` 
    need to be changed to your local path. You may also change the `batch-size` and `nproc_per_node` according to your 
    available hardware.
  
  - Run the following script for text similarity finetune task (use the afqmc dataset as an example)
  
  ```
  cd examples/glm/ 
  bash scripts/ds_finetune_superglue.sh\
       config/model_blocklm_large_chinese.sh\
       config_tasks/task_afqmc.sh
  ```
  
  
  - Run the following script for text classification finetune task  (use the thunews and thunews dataset as an example)
  
  ```
  cd examples/glm/ 
  bash scripts/ds_finetune_superglue.sh\
       config/model_blocklm_large_chinese.sh\
       config_tasks/task_tnews.sh
  ```
  
  - Run the following script  for causal inference finetune task (use the COPA dataset as an example)
  
  ```
  cd examples/glm/ 
  bash scripts/ds_finetune_superglue.sh\
       config/model_blocklm_large_chinese.sh\
       config_tasks/task_copa.sh
  ```
    
  - To apply GLM to a new NLU dataset with cloze-filling finetuning, implement a `DataProcessor` in
    [examples/glm/tasks/superglue/dataset.py](examples/glm/tasks/superglue/dataset.py) for data loading and add a `PVP` in 
    [examples/glm/tasks/superglue/pvp.py](examples/glm/tasks/superglue/pvp.py) for the cloze question. More details can be found 
    [here](examples/glm/tasks/superglue/README.md).
  
  
  
  #### Blank Filling (Interactive)
  * Change `CHECKPOINT_PATH` to your local path. Run the following script
  ```
  bash config/generate_block.sh\
       config/model_blocklm_large_chinese.sh
  ```
  ##### Example1 (Entity Prediction):
  
  Context: 凯旋门位于意大利米兰市古城堡旁。1807年为纪念[MASK]而建，门高25米，顶上矗立两武士青铜古兵车铸像。
  
  GLM:拿破仑军队攻克米兰城
  
  ##### Example2 (Sentence Prediction)
  Context: 工业互联网（Industrial Internet）是新一代信息通信技术与工业经济深度融合的新型基础设施、应用模式和工业生态，通过对人、机、物、系统等的全面连接，构建起覆盖全产业链、全价值链的全新制造和服务体系，为工业乃至产业数字化、网络化、智能化发展提供了实现途径，是第四次工业革命的重要基石。[sMASK]它以网络为基础、平台为中枢、数据为要素、安全为保障，既是工业数字化、网络化、智能化转型的基础设施，也是互联网、大数据、人工智能与实体经济深度融合的应用模式，同时也是一种新业态、新产业，将重塑企业形态、供应链和产业链。当前，工业互联网融合应用向国民经济重点行业广泛拓展，形成平台化设计、智能化制造、网络化协同、个性化定制、服务化延伸、数字化管理六大新模式，赋能、赋智、赋值作用不断显现，有力的促进了实体经济提质、增效、降本、绿色、安全发展。
  
  GLM: 工业互联网是制造业技术、管理、模式的重大变革,是推动互联网、大数据、人工智能和实体经济深度融合的重要载体,是建设制造强国和网络强国的重要基础。
  
  ##### Example3 (Long Text Generation)
  Context: 问题:高斯所在的国家有什么汽车品牌?答案:[gMASK]
  
  GLM:答案:[gMASK]<|startofpiece|>德国奔驰、德国大众、别克、沃尔沃、斯柯达、本田、雪铁龙.
  
  
  ### Ptuning
  Run the following script to integrate p-tuning with GLM:
  ```shell
  cd algutils/ptuning/ 
  bash finetune_zy.sh
  ```
  
  ### Pretrain
  Run the following script to pre-train the GLM-Large model
  ```shell
  cd examples/glm/ 
  bash scripts/ds_pretrain_nvidia.sh config/ds_block_large.sh
  ```
  
  The script [examples/glm/config/ds_pretrain_nvidia.sh](examples/glm/config/ds_pretrain_nvidia.sh) launches the training program with DeepSpeed. You should change `NUM_WORKERS` and `NUM_GPUS_PER_WORKER` to the number of workers and the number of gpus per worker. Also change `HOST_FILE_PATH` to the path to an OpenMPI-style hostfile. More details about DeepSpeed launcher can be found [here](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node).
  
  The file [examples/glm/config/ds_block_large.sh](examples/glm/config/ds_block_large.sh) defines the hyperparameters for pretraining. Most of the arguments are fairly self-explanatory. Specifically, `--train-data` can be multiple keywords defined in `NAMED_CORPORA` in [data_utils/corpora.py](data_utils/corpora.py). The hyperparameters of the optimizer are defined in the corresponding json file under `config`. The semantics of the json file can be found [here](https://www.deepspeed.ai/docs/config-json).
  
  
  
  ## Bert
  
  We show some examples based on GLM model.
  
  ### Pretrain
  Run the following script to pre-train the Bert model
  ```shell
  cd examples/bert/ 
  python q