readme: ---
  language: en
  tags:
  - Clsssification
  - science
  license: apache-2.0
  datasets:
  - tensorflow
  - numpy
  - keras
  - pandas
  - openpyxl
  - gensin
  - contractions
  - nltk
  - spacy
  thumbnail: https://github.com/Marcosdib/S2Query/Classification_Architecture_model.png
  metrics:
  - accuracy
  ---
  
  ![MCTIimg](https://antigo.mctic.gov.br/mctic/export/sites/institucional/institucional/entidadesVinculadas/conselhos/pag-old/RODAPE_MCTI.png)
  
  
  # MCTI Text Classification Task (uncased)
  
  Disclaimer: The Brazilian Ministry of Science, Technology, and Innovation (MCTI) has partially supported this project.
  
  The model [NLP MCTI Classification Multi](https://huggingface.co/spaces/unb-lamfo-nlp-mcti/NLP-W2V-CNN-Multi) is part of the project [Research Financing Product Portfolio (FPP)](https://huggingface.co/unb-lamfo-nlp-mcti) and focuses 
  on the Text Classification task, exploring different machine learning strategies to classify a small amount 
  of long, unstructured, and uneven data to find a proper method with good performance. Pre-training and word embedding 
  solutions were used to learn word relationships from other datasets with considerable similarity and a larger scale. 
  Then, using the acquired resources, based on the dataset available in the MCTI, transfer learning plus deep learning 
  models were applied to improve the understanding of each sentence. 
  
  ## According to the abstract,
  
  "The research results serve as a successful case of artificial intelligence in a federal government application".
  More details about the project, architecture model, training model, and classifications process can be found in the article
  ["Using transfer learning to classify long unstructured texts with small amounts of labeled data"](https://www.scitepress.org/Link.aspx?doi=10.5220/0011527700003318).
  
  ## Model description
  
  The work consists of a machine learning model with word embedding and Convolutional Neural Network (CNN).
  For the project, a Convolutional Neural Network (CNN) was chosen, as it presents better accuracy in empirical
  comparison with 3 other different architectures: Neural Network (NN), Deep Neural Network (DNN), and Long-Term 
  Memory (LSTM).
  
  As the input data is composed of unstructured and nonuniform texts, it is essential to normalize the data to study 
  little insights and valuable relationships to work with their best features. In this way, learning is 
  facilitated and allows the gradient descent to converge more quickly.
  
  The first layer of the model is a pre-trained Word2Vec embedding layer as a method of extracting features from the data that can 
  replace one-hot coding with dimensional reduction. The pre-training of this model is explained further in this document.
  
  After the embedding layer, there is the CNN classification model. The architecture of the CNN network is composed of a 50% dropout layer followed by two 1D convolution layers 
  associated with a MaxPooling layer. After maximum grouping, a dense layer of size 128 is added and connected to 
  a 50% dropout which finally connects to a flattened layer and the final sort dense layer. The dropout layers 
  helped avoid network overfitting by masking part of the data so that the network learned to create 
  redundancies in analyzing the inputs.
  
  ![CNN Classification Model](https://raw.githubusercontent.com/chap0lin/WEBIST2022/master/Assets/cnn_model.png)
  
  ## Model variations
  
  Table 1 below presents the results of several implementations with different architectures, highlighting the 
  accuracy, f1-score, recall, and precision results obtained in each network training. 
  
  Table 1: Results of experiments
  | Model                  | Accuracy | F1-score | Recall | Precision |
  |------------------------|----------|----------|--------|-----------|
  | Keras Embedding + SNN  |    92.47 |    88.46 |  79.66 |    100.00 |
  | Keras Embedding + DNN  |    89.78 |    84.41 |  77.81 |     92.57 |
  | Keras Embedding + CNN  |    93.01 |    89.91 |  85.18 |     95.69 |
  | Keras Embedding + LSTM |    93.01 |    88.94 |  83.32 |     95.54 |
  | Word2Vec + SNN         |    89.25 |    83.82 |  74.15 |     97.10 |
  | Word2Vec + DNN         |    90.32 |    86.52 |  85.18 |     88.70 |
  | Word2Vec + CNN         |    92.47 |    88.42 |  80.85 |     98.72 |
  | Word2Vec + LSTM        |    89.78 |    84.36 |  75.36 |     95.81 |
  | Longformer + SNN       |    61.29 |        0 |      0 |         0 |
  | Longformer + DNN       |    91.93 |    87.62 |  80.37 |     97.62 |
  | Longformer + CNN       |    94.09 |    90.69 |  83.41 |    100.00 |
  | Longformer + LSTM      |    61.29 |        0 |      0 |         0 |
  
  Table 2 below shows the times required for training each epoch, the data validation execution time and the weight of the deep learning 
  model associated with each implementation.
  
  Table 2: Results of Training time epoch, Validation time, and Weight
  | Model                  |  Training time epoch(s) | Validation time (s) | Weight(MB) |
  |------------------------|:-----------------------:|:-------------------:|:----------