Body: <p>I have a task where I need to take &quot;data points&quot; which consist of collections of items. Each item needs to be categorised according to predefined categories. That's the easy part - my solution is to train a deep neural network with cross entropy loss. By the way, the reason I don't classify each item separately is because they acquire their meaning when they come together as a set.</p>
  <p>The hard part is that each of these items also have a cluster label. Each cluster can only have items of one category in it, and there can be any number of clusters. Unsupervised clustering methods (applied after the neural network does the categorisation) work fairly well, but not well-enough for my needs. I'd like to:</p>
  <p>A. Make use of the fact that I have the ground truth labelling for these clusters</p>
  <p>B. Leverage my deep neural network because a lot of the &quot;reasoning&quot; required to solve the classification task will be conducive to the clustering task.</p>
  <p>Answers which address at least one of those are useful to me.</p>
  <p><strong>EDIT</strong></p>
  <p>I realised I might be confusing people with this concept of cluster &quot;labels&quot;. To clarify, this is no different than the standard way a classical unsupervised clustering algorithm might return its results. If I have N data points and feed them to a clustering algo, the algo might return N labels, one for each data point, and each of which are integers in [0, C-1] where C is the number of clusters. In my example we have the labels for a training dataset and want to make use of them during training. We <strong>cannot</strong> use softmax + cross-entropy loss because the cluster labels are permutation invariant.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-02T23:12:15.683Z
Comments.Text: Is there a reason you can't create two separate output layers, one that tells you the category and the other that tells you the cluster?  Both outputs would use crossentropy loss.  In effect, this would be a supervised clustering, if you will.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T08:52:15.063Z
Comments.Text: @DavidHoelzer  I had a similar question on [stats](https://stats.stackexchange.com/questions/543335/what-are-some-machine-learning-frameworks-for-supervised-clustering). I think I somehow confused people with my wording. The cluster "labels" are permutation invariant, so cross-entropy loss won't work.
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T09:03:26.763Z
Comments.Text: Just to spell it out for anyone who comes across this, the reason permutation invariance is a problem is because we shouldn't punish the network for outputting labels [1, 1, 0, 0] when the ground truth is [0, 0, 1, 1]. These label sets are equivalent in terms of clustering.
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T11:13:04.483Z
Comments.Text: @user253751 it certainly will impact training. It will raise similar issues to training a dog vs canis_lupis binary classification task on a dataset of dog images. There are no distinguising traits, so the network will never learn your labels. It will just learn to predict the distribution of dog:canis_lupis randomly (unless there's a leak, and there's some way of distinguishing).
  
  In the clustering task, say I always label the biggest cluster (whatever it is) with 0s, then the network would probably just learn to mostly predict 0s... (and maybe it will learn some vague notion of clusters)
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T11:23:35.533Z
Comments.Text: It's not exactly the same... but hopefully you get the point. There's just no concrete way to say which cluster should be 0 and which should be 1 and... so on for each input. So it will just "confuse" the model.
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T12:29:34.233Z
Comments.Text: If they are permutation invariant, why not adjust that so that they aren't?  Otherwise, the clusters are effectively random from the point of view of a function to map from features to predictions, which seems to be what @alexander-soare is saying.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T12:44:48.993Z
Comments.Text: @DavidHoelzer because I don't have a rules based way of doing it. It would be like randomly assigning canis_lupis and dog labels to pictures of dogs. I could come up with a scheme, but then I'm inventing a scheme and trying to make the model learn it which is unwanted overhead, and may not even work.
  
  I must say... I've gotten a lot of push back on this here and in the stats one. Wondering if I've gone mad... :P
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T12:47:13.880Z
Comments.Text: There IS a way of doing this though, and it's not overly-exotic or new. That is, self-supervised learning. The loss function is a distance metric between descriptors, and we want descriptors of items in the same cluster to be close, and ... yeah the opposite thing too. My only beef with that is that I will then have to do classical clustering in feature space anyway. Would like to know if there's a way to output labels rather than descriptors.
Comments.UserDisplayName: Alexander Soare
Tags: <neural-networks><supervised-learning><clustering>
Title: What are some machine learning frameworks for supervised clustering?
UserDisplayName: Alexander Soare
ThreadText: # What are some machine learning frameworks for supervised clustering?
  
  (Asked by: username_0 on 2021-09-02)
  
  I have a task where I need to take "data points" which consist of collections of items. Each item needs to be categorised according to predefined categories. That's the easy part - my solution is to train a deep neural network with cross entropy loss. By the way, the reason I don't classify each item separately is because they acquire their meaning when they come together as a set.
  
  The hard part is that each of these items also have a cluster label. Each cluster can only have items of one category in it, and there can be any number of clusters. Unsupervised clustering methods (applied after the neural network does the categorisation) work fairly well, but not well-enough for my needs. I'd like to:
  
  A. Make use of the fact that I have the ground truth labelling for these clusters
  
  B. Leverage my deep neural network because a lot of the "reasoning" required to solve the classification task will be conducive to the clustering task.
  
  Answers which address at least one of those are useful to me.
  
  **EDIT**
  
  I realised I might be confusing people with this concept of cluster "labels". To clarify, this is no different than the standard way a classical unsupervised clustering algorithm might return its results. If I have N data points and feed them to a clustering algo, the algo might return N labels, one for each data point, and each of which are integers in [0, C-1] where C is the number of clusters. In my example we have the labels for a training dataset and want to make use of them during training. We **cannot** use softmax + cross-entropy loss because the cluster labels are permutation invariant.
  
  **Tags:** neural-networks, supervised-learning, clustering
  
  > **Comments:**
  > **username_1**: Is there a reason you can't create two separate output layers, one that tells you the category and the other that tells you the cluster?  Both outputs would use crossentropy loss.  In effect, this would be a supervised clustering, if you will.  
  > **username_0**: @username_1  I had a similar question on [stats](https://stats.stackexchange.com/questions/543335/what-are-some-machine-learning-frameworks-for-supervised-clustering). I think I somehow confused people with my wording. The cluster "labels" are permutation invariant, so cross-entropy loss won't work.  
  > **username_0**: Just to spell it out for anyone who comes across this, the reason permutation invariance is a problem is because we shouldn't punish the network for outputting labels [1, 1, 0, 0] when the ground truth is [0, 0, 1, 1]. These label sets are equivalent in terms of clustering.  
  > **username_0**: @user253751 it certainly will impact training. It will raise similar issues to training a dog vs canis_lupis binary classification task on a dataset of dog images. There are no distinguising traits, so the network will never learn your labels. It will just learn to predict the distribution of dog:canis_lupis randomly (unless there's a leak, and there's some way of distinguishing).  In the clustering task, say I always label the biggest cluster (whatever it is) with 0s, then the network would probably just learn to mostly predict 0s... (and maybe it will learn some vague notion of clusters)  
  > **username_0**: It's not exactly the same... but hopefully you get the point. There's just no concrete way to say which cluster should be 0 and which should be 1 and... so on for each input. So it will just "confuse" the model.  
  > **username_1**: If they are permutation invariant, why not adjust that so that they aren't?  Otherwise, the clusters are effectively random from the point of view of a function to map from features to predictions, which seems to be what @username_0 is saying.  
  > **username_0**: @username_1 because I don't have a rules based way of doing it. It would be like randomly assigning canis_lupis and dog labels to pictures of dogs. I could come up with a scheme, but then I'm inventing a scheme and trying to make the model learn it which is unwanted overhead, and may not even work.  I must say... I've gotten a lot of push back on this here and in the stats one. Wondering if I've gone mad... :P  
  > **username_0**: There IS a way of doing this though, and it's not overly-exotic or new. That is, self-supervised learning. The loss function is a distance metric between descriptors, and we want descriptors of items in the same cluster to be close, and ... yeah the opposite thing too. My only beef with that is that I will then have to do classical clustering in feature space anyway. Would like to know if there's a way to output labels rather than descriptors.  
  
  ---
  
  ## 0 answers
  
  