Answers.Body: <p>An interesting question. I would start by finding <code>n</code> nearest neighbors of each data point, then calculate their center of mass <code>c</code> and the point's distance <code>d</code> to its nth nearest neighbor. The smaller the <code>d</code> is the larger the density is around a given point. You could then iteratively step every point away from their <code>c</code> in an inverse proportion to the distance <code>d</code> with a suitable step size. This would spread out the clusters.</p>
  
  <p>But this won't help you transform any new points in the dataset, maybe you can learn this arbitrary mapping <code>R^n -&gt; R^n</code> by using an other neural network and apply it to new samples?</p>
  
  <p>This is the first ad-hoc idea which came to my mind. It would be interesting to see a 2D animation of this.</p>
  
  <p>A more rigorous approach might be a <a href="https://blog.keras.io/building-autoencoders-in-keras.html" rel="nofollow noreferrer">variational autoencoder</a>, you can embed the data in a lower dimensional space with approximately normal distribution. But it doesn't guarantee that clusters would be as spread out as you'd like. An alternative loss function would help with that, for example every point's distance to their original nth closest neighbor should be as close to one as possible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T03:58:38.307Z
Answers.Text: Thanks for your advise. I also considered about using k-nearest neighbor(kNN) based scheme, but as you know it includes max (or min) function in it. So I think it is very hard to use kNN related metric as a loss function to train VAE, even though the goal mapping itself do not have to be a differentiable one. But maybe some relaxed version of it, such as stochastic NN could be a naive replacement.
Answers.UserDisplayName: Jun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T21:28:49.333Z
Answers.Text: After studying neural networks more I recently came up with a new idea (might not be novel): train an autoencoder (it doesn't matter if the encoded "code" dimension is smaller or same size as the input) and add extra penalty to the "code" activations if they are too close on known dissimilar inputs. I am about to write a blog article about this at some point.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T21:37:15.090Z
Answers.Text: Too late to edit the previous comment. Surely this is not a novel idea, it combines an autoencoder with a contrastive loss. But wikipedia's article on contractive autoencoder mentions only positive samples, which makes it more of a denoising autoencoder IMO.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-20T14:06:29.457Z
Answers.LastActivityDate: 2020-05-20T14:06:29.457Z
Answers.UserDisplayName: NikoNyrh
Body: <p>Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other.  Some of them are already far from each other, but some of them can be positioned very closely.  </p>
  
  <p>I want to get a certain mapping function that can separate such points that are close to each other, while still preserving the points that are already far away from each other. </p>
  
  <p>I do not care what is the form of the mapping. Since the mapping will be employed as pre-processing, it does not have to be differentiable or even continuous.  </p>
  
  <p>I think this problem is somewhat similar to 'minimizing the maximum distance ratio between the points'. Maybe this problem can be understood as stretching the crushed graph to a sphere-like isotropic graph. </p>
  
  <p>I googled it for an hour, but it seems that the people are usually interested in selecting the points that have such nice characteristics from a bunch of data, rather than mapping an existing vector points to a better one.</p>
  
  <p>So, in conclusion, I could not find anything useful.   </p>
  
  <p>Maybe you can think 'the neural network will naturally learn it while solving classification problem'. But it failed. Because it is already struggling with too many burdens. So, this is why I want to help my network with pre-processing.  </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-20T20:43:13.657Z
Comments.Text: isn't it what kernels do?
Comments.UserDisplayName: Alireza
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T03:54:19.767Z
Comments.Text: which kernel do you mean?
Comments.UserDisplayName: Jun
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T07:58:34.173Z
Comments.Text: The general concept of kernel trick in kernel methods in machine learning. such as SVM. the kernels do what you're saying: transforming to a new hyperspace which a discriminative hyperplane can easier separate classes.
Comments.UserDisplayName: Alireza
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-19T16:58:20.387Z
Comments.Text: this is interesting, but, why do you want to do this?
Comments.UserDisplayName: carlo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-17T17:59:43.873Z
Comments.Text: "vector points" means "set of points described as vectors" ? Is a 1D, 2D, 3D, ... problem ? The maximum distance will be reached send each point to the infinite (except one of them at origin). Do you have restrictions for the base space ?
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-17T18:01:03.287Z
Comments.Text: related: https://math.stackexchange.com/questions/339649/maximizing-distance-between-points
Comments.UserDisplayName: pasaba por aqui
Tags: <deep-learning><data-preprocessing>
Title: Can I find a mapping that minimizes the maximum distance ratio of certain vectors?
UserDisplayName: Jun
ThreadText: # Can I find a mapping that minimizes the maximum distance ratio of certain vectors?
  
  (Asked by: username_0 on 2020-05-19)
  
  Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other. Some of them are already far from each other, but some of them can be positioned very closely.
  
  I want to get a certain mapping function that can separate such points that are close to each other, while still preserving the points that are already far away from each other.
  
  I do not care what is the form of the mapping. Since the mapping will be employed as pre-processing, it does not have to be differentiable or even continuous.
  
  I think this problem is somewhat similar to 'minimizing the maximum distance ratio between the points'. Maybe this problem can be understood as stretching the crushed graph to a sphere-like isotropic graph.
  
  I googled it for an hour, but it seems that the people are usually interested in selecting the points that have such nice characteristics from a bunch of data, rather than mapping an existing vector points to a better one.
  
  So, in conclusion, I could not find anything useful.
  
  Maybe you can think 'the neural network will naturally learn it while solving classification problem'. But it failed. Because it is already struggling with too many burdens. So, this is why I want to help my network with pre-processing.
  
  **Tags:** deep-learning, data-preprocessing
  
  > **Comments:**
  > **username_1**: isn't it what kernels do?  
  > **username_0**: which kernel do you mean?  
  > **username_1**: The general concept of kernel trick in kernel methods in machine learning. such as SVM. the kernels do what you're saying: transforming to a new hyperspace which a discriminative hyperplane can easier separate classes.  
  > **username_2**: this is interesting, but, why do you want to do this?  
  > **username_3**: "vector points" means "set of points described as vectors" ? Is a 1D, 2D, 3D, ... problem ? The maximum distance will be reached send each point to the infinite (except one of them at origin). Do you have restrictions for the base space ?  
  > **username_3**: related: https://math.stackexchange.com/questions/339649/maximizing-distance-between-points  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2020-05-20. Score: 1
  
  An interesting question. I would start by finding `n` nearest neighbors of each data point, then calculate their center of mass `c` and the point's distance `d` to its nth nearest neighbor. The smaller the `d` is the larger the density is around a given point. You could then iteratively step every point away from their `c` in an inverse proportion to the distance `d` with a suitable step size. This would spread out the clusters.
  
  But this won't help you transform any new points in the dataset, maybe you can learn this arbitrary mapping `R^n -> R^n` by using an other neural network and apply it to new samples?
  
  This is the first ad-hoc idea which came to my mind. It would be interesting to see a 2D animation of this.
  
  A more rigorous approach might be a [variational autoencoder](https://blog.keras.io/building-autoencoders-in-keras.html), you can embed the data in a lower dimensional space with approximately normal distribution. But it doesn't guarantee that clusters would be as spread out as you'd like. An alternative loss function would help with that, for example every point's distance to their original nth closest neighbor should be as close to one as possible.
  
  > **Comments:**
  > **username_0**: Thanks for your advise. I also considered about using k-nearest neighbor(kNN) based scheme, but as you know it includes max (or min) function in it. So I think it is very hard to use kNN related metric as a loss function to train VAE, even though the goal mapping itself do not have to be a differentiable one. But maybe some relaxed version of it, such as stochastic NN could be a naive replacement.  
  > **username_4**: After studying neural networks more I recently came up with a new idea (might not be novel): train an autoencoder (it doesn't matter if the encoded "code" dimension is smaller or same size as the input) and add extra penalty to the "code" activations if they are too close on known dissimilar inputs. I am about to write a blog article about this at some point.  
  > **username_4**: Too late to edit the previous comment. Surely this is not a novel idea, it combines an autoencoder with a contrastive loss. But wikipedia's article on contractive autoencoder mentions only positive samples, which makes it more of a denoising autoencoder IMO.  
  
  ---
  
  