Answers.Body: <p>It sounds like you are trying to do some kind of <a href="https://en.wikipedia.org/wiki/Semi-supervised_learning" rel="nofollow noreferrer">semi-supervised learning</a>. In semi-supervised learning, some data points are labelled (you know which class they belong to), and others are not. There are classification algorithms designed specifically for this kind of problem, like a <a href="http://www.cs.cmu.edu/~guestrin/Class/10701-S06/Slides/tsvms-pca.pdf" rel="nofollow noreferrer">transductive-SVM</a>. I personally have not found these techniques to be more effective than simply discarding the unlabelled data and treating my problem as purely supervised, but YMMV.</p>
  
  <p>TFIDF remains fairly popular, as do ngram-based approaches. A more modern vectorization to consider might be <a href="https://en.wikipedia.org/wiki/Word2vec" rel="nofollow noreferrer">word2vec</a>, which translated something like a bag-of-words style vector into a more meaningful feature space for words.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-22T14:06:58.003Z
Answers.Text: Thanks. Actually, in my case all the data is unlabelled, hence the term 'clustering'. What I want to do is separate them into a predefined number of categories. There is a number of factors that I do now how to handle: 1. how to ensure a good starting position for the clusters b) how to enable refinement of clusters' definitions over time. As for the vectorisation techniques, I will definitely look into word2vec.
Answers.UserDisplayName: Jan Parzydło
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-22T15:41:42.050Z
Answers.Text: @JanParzydło The standard clustering technique is K-Means clustering. (https://en.wikipedia.org/wiki/K-means_clustering). This may be slow if you have high-dimensional data, but can use k-medoids (https://en.wikipedia.org/wiki/K-medoids) with a pre-computed kernel instead if this is a problem.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-08T14:41:28.217Z
Answers.LastActivityDate: 2018-08-08T14:41:28.217Z
Answers.UserDisplayName: John Doucette
Answers.Body: <blockquote>
    <p>I would also like to be able to start with a predefined set of classes (or clusters/centroids) as I know for a fact what the types of those emails will be.</p>
  </blockquote>
  
  <p>This is not a clustering problem, but a semi-supervised learning problem. If you don't have labeled data yet, then create some labels. You might also want to look into "active learning".</p>
  
  <p>One approach is:</p>
  
  <ol>
  <li>For each category, create 5 labeled samples</li>
  <li>Train a classifier on them (e.g. tf-idf features and a small neural network)</li>
  <li>Let the neural network label your dataset</li>
  <li>Check the labels where it was most confident for all classes and the ones where the probabilities for all classes were most evenly spread. Use this to quickly create more labels.</li>
  <li>Maybe Amazon mechanical Turk is an option to quickly generate more labels</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-08T05:49:08.947Z
Answers.LastActivityDate: 2018-10-08T05:49:08.947Z
Answers.UserDisplayName: Martin Thoma
Body: <p>I have a dataset of unlabelled emails that fall into distinct categories (around a dozen). I want to be able to classify them along with new ones to come in the future in a dynamic matter. I know that there are dynamic clustering techniques that allow the clusters to evolve over time ('dynamic-means' being one of them). However, I would also like to be able to start with a predefined set of classes (or clusters/centroids), as I know for a fact what the types of those emails will be.</p>
  <p>Furthermore, I need some guidance in terms of what vectorisation technique to use for my type of data. Would creating a term matrix using TF-IDF be sufficient? I assume that the data I am dealing with could be differentiated on the basis of keyword occurrence, but I cannot tell to what degree. Are there more sophisticated vectorisation techniques based more on the text semantics? Are they worth exploring?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T23:46:41.337Z
Comments.Text: `that allow the clusters to evolve over time` Just to confirm, does that mean that you want to add more categories in the future?
Comments.UserDisplayName: cantordust
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-22T14:14:46.140Z
Comments.Text: Thanks for you response. Like I said in the comment below - I have a rough idea what the types of emails will be and want to cluster them around this idea. First of all, I believe I should not start the clustering procedure with a random set of centroids, but rather predefined ones. Secondly, since I can only try to define my entry points for the algorithm basing on the current dataset, I think it would be reasonable to let the centroids move around as new datapoints arrive. I think that I'd like to stick with a fixed number of clusters for now.
Comments.UserDisplayName: Jan Parzydło
Tags: <unsupervised-learning><algorithm-request><clustering>
Title: What techniques to explore for dynamic clustering of documents (emails)?
UserDisplayName: Jan Parzydło
ThreadText: # What techniques to explore for dynamic clustering of documents (emails)?
  
  (Asked by: username_0 on 2018-07-03)
  
  I have a dataset of unlabelled emails that fall into distinct categories (around a dozen). I want to be able to classify them along with new ones to come in the future in a dynamic matter. I know that there are dynamic clustering techniques that allow the clusters to evolve over time ('dynamic-means' being one of them). However, I would also like to be able to start with a predefined set of classes (or clusters/centroids), as I know for a fact what the types of those emails will be.
  
  Furthermore, I need some guidance in terms of what vectorisation technique to use for my type of data. Would creating a term matrix using TF-IDF be sufficient? I assume that the data I am dealing with could be differentiated on the basis of keyword occurrence, but I cannot tell to what degree. Are there more sophisticated vectorisation techniques based more on the text semantics? Are they worth exploring?
  
  **Tags:** unsupervised-learning, algorithm-request, clustering
  
  > **Comments:**
  > **username_1**: `that allow the clusters to evolve over time` Just to confirm, does that mean that you want to add more categories in the future?  
  > **username_0**: Thanks for you response. Like I said in the comment below - I have a rough idea what the types of emails will be and want to cluster them around this idea. First of all, I believe I should not start the clustering procedure with a random set of centroids, but rather predefined ones. Secondly, since I can only try to define my entry points for the algorithm basing on the current dataset, I think it would be reasonable to let the centroids move around as new datapoints arrive. I think that I'd like to stick with a fixed number of clusters for now.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-08-08. Score: 1
  
  It sounds like you are trying to do some kind of [semi-supervised learning](https://en.wikipedia.org/wiki/Semi-supervised_learning). In semi-supervised learning, some data points are labelled (you know which class they belong to), and others are not. There are classification algorithms designed specifically for this kind of problem, like a [transductive-SVM](http://www.cs.cmu.edu/~guestrin/Class/10701-S06/Slides/tsvms-pca.pdf). I personally have not found these techniques to be more effective than simply discarding the unlabelled data and treating my problem as purely supervised, but YMMV.
  
  TFIDF remains fairly popular, as do ngram-based approaches. A more modern vectorization to consider might be [word2vec](https://en.wikipedia.org/wiki/Word2vec), which translated something like a bag-of-words style vector into a more meaningful feature space for words.
  
  > **Comments:**
  > **username_0**: Thanks. Actually, in my case all the data is unlabelled, hence the term 'clustering'. What I want to do is separate them into a predefined number of categories. There is a number of factors that I do now how to handle: 1. how to ensure a good starting position for the clusters b) how to enable refinement of clusters' definitions over time. As for the vectorisation techniques, I will definitely look into word2vec.  
  > **username_2**: @username_0 The standard clustering technique is K-Means clustering. (https://en.wikipedia.org/wiki/K-means_clustering). This may be slow if you have high-dimensional data, but can use k-medoids (https://en.wikipedia.org/wiki/K-medoids) with a pre-computed kernel instead if this is a problem.  
  
  ---
  
  ### Answer by username_3 on 2018-10-08. Score: 0
  
  > I would also like to be able to start with a predefined set of classes (or clusters/centroids) as I know for a fact what the types of those emails will be.
  
  This is not a clustering problem, but a semi-supervised learning problem. If you don't have labeled data yet, then create some labels. You might also want to look into "active learning".
  
  One approach is:
  
  1. For each category, create 5 labeled samples
  2. Train a classifier on them (e.g. tf-idf features and a small neural network)
  3. Let the neural network label your dataset
  4. Check the labels where it was most confident for all classes and the ones where the probabilities for all classes were most evenly spread. Use this to quickly create more labels.
  5. Maybe Amazon mechanical Turk is an option to quickly generate more labels
  
  
  ---
  
  