Here we use to refer to the label of the tweet, and to refer the features (words). What C X we needs are: Number of tweets: the number of all tweets. Number of the number of unique labels. C: Number of the number of tweets with label This should be a global C = c: c. variable and can be used with the previous number we have to compute label prior probabilities. Number of in = the number of words that are in all the tweets with label X C c: c. Number of = in = the number of word that is in all the tweets with X x C c: x label This number and the previous number we have are combined to c. compute the probabilities of a single word under a label x c. the total number of unique words in all tweets. This is actually the size of |V|: our vocabulary. Using the above data, we can get the prior probabilities and likelihoods. It is obvious that only one mapper and reducer job is not enough. And our goal is to build the prediction model, which means we only need to test the training phase. In general there are two jobs: the word job and the label job. For each job there is a map function and a reduce function. The word job is to compute the likelihood of each word, and the label job is to compute the prior probability of each label. Let us take a look at each job in details. The Word Job
Here we use C to refer to the label of the tweet, and X to refer the features (words). What we needs are: Number of tweets: the number of all tweets. Number of C: the number of unique labels. Number of C = c: the number of tweets with label c. This should be a global variable and can be used with the previous number we have to compute label prior probabilities. Number of X in C = c: the number of words that are in all the tweets with label c. Number of X = x in C = c: the number of word x that is in all the tweets with label c. This number and the previous number we have are combined to compute the probabilities of a single word x under a label c. |V|: the total number of unique words in all tweets. This is actually the size of our vocabulary. Using the above data, we can get the prior probabilities and likelihoods. It is obvious that only one mapper and reducer job is not enough. And our goal is to build the prediction model, which means we only need to test the training phase. In general there are two jobs: the word job and the label job. For each job there is a map function and a reduce function. The word job is to compute the likelihood of each word, and the label job is to compute the prior probability of each label. Let us take a look at each job in details. The Word Job
ZhaoyuLi2014.docx