<repo_name>adriancervero/fake-news-detector<file_sep>/README.md
# Fake News Detector
This project consists of the building of a machine learning model capable of detecting fake news. 
The model will be deployed through a web application using Flask.
## Dataset
The corpus for the model training was obtained from the next [article](https://www.researchgate.net/publication/333098973_Detection_of_fake_news_in_a_new_corpus_for_the_Spanish_language).
\
\
<NAME>., <NAME>., <NAME>., & <NAME>. (2019). 
Detection of fake news in a new corpus for the Spanish language. Journal of Intelligent & Fuzzy Systems, 36(5), 4869-4876.
\
\
And it is available in this repository: https://github.com/jpposadas/FakeNewsCorpusSpanish

## Preprocessing
First, I have created a preprocessing function for prepare the text before feed the model with it. This function remove punctuation and tokenize the words
of each text. Optionally can remove stopwords and lemmatize the text via parameter. This function will be the first step in the pipeline process. Next
step will be the vectorizer that transform the text words in an array of counts of each word of the corpus, with a vocabulary of 3000 words. After this process
data is prepared for the model.

## Model Building
In order to find the best model for this purpose I have tried several classification models such as Logistic Regression, Support Vector Machine, Random Forest
and Boosting. Also, I have evaluated three different feature representation:
* Bag of words
* N-gram (bigrams and trigrams)
\
\
I have tried all posible combinations of these setups to find the best accuracy on the training test using GridSearchCV from Scikit-Learn python module.

## Results
                        LogReg  SVM   RF    Boosting
    CV Score(Accuracy): 0.82    0.77  0.79  0.80
    
    Test Score (Using LogReg): 0.78 

## Deployment

I used Flask for the web application and Heroku for deploying it.

Available on:
https://fake-news-detector-by-acervero.herokuapp.com/
<file_sep>/preprocessing.py
from sklearn.base import BaseEstimator, TransformerMixin
import spacy
import numpy as np
from spacy.lang.en.stop_words import STOP_WORDS
nlp = spacy.load('./models/es_core_news_sm-2.3.1')

class TextPreprocessing(BaseEstimator, TransformerMixin):
    def __init__(self, remove_punctuation=True, remove_stopwords=True,
                lemmatization=True, lowercase=True):
        self.remove_punctuation = remove_punctuation
        self.remove_stopwords = remove_stopwords
        self.lemmatization = lemmatization
        self.lowercase = lowercase

    def fit(self, X, y=None):
        return self
    def transform(self, X, y=None):
        X_transformed = []
        for idx, text in enumerate(X):
            prepared_text = ""
            if self.lowercase:
                text = text.lower()
            doc = nlp(text, disable=['tagger','parser', 'ner', 'textcat'])
            for word in doc:
                if self.remove_stopwords and word.is_stop:
                    continue
                if self.remove_punctuation and word.is_punct:
                    continue
                if self.lemmatization:
                    word = word.lemma_
                prepared_text += " " + str(word)
            X_transformed.append(prepared_text)
        return np.array(X_transformed)
<file_sep>/app.py
from flask import Flask,request, url_for, redirect, render_template, jsonify
from preprocessing import TextPreprocessing
import pickle

app = Flask(__name__)


def predict_news(text):
    y = pipeline.predict([text])
    return y
def loadModels(name_file):
    with open('models/'+name_file+'.tk', 'rb') as pickle_file:
            model = pickle.load(pickle_file)
    return model

pipeline = loadModels('pipeline')

@app.route('/')
def home():
    return render_template("home.html")

@app.route('/predict', methods=['POST'])
def predict():
    new_request = request.form
    y = predict_news(new_request['News'])
    return render_template('home.html',pred='News is {}'.format(y))

if __name__ == '__main__':
    app.run(debug=True)
<file_sep>/requirements.txt
Flask==1.1.2
numpy==1.19.4
scikit_learn==0.23.2
spacy==2.3.4
gunicorn==20.0.4
