Answers.Body: <p>Firstly you claim that geometric formulation of machine learning (M)L is not fundamental and the same ML problem can be represented in different spaces of different dimensions. However, the essence of contemporary deep learning is built from two simple principles known as feature learning and local gradient-descent. Many
  high-dimensional learning tasks previously thought to be beyond reach, such as computer vision, playing Go, or protein folding, are in fact feasible after adapted learned features capture the appropriate notion of <em>regularity</em> for each task, such as semantic similarity (GPT and BERT large variant pretrained token embeddings have different dimensions and representations, but both share regularity of semantic similarity) and group symmetry (translational and rotational equivariance &amp; invariance of CNNs).</p>
  <p>Similarly for the geometric feature of distance, each ML task will have inductive bias for such feature if it's really relevant with domain knowledge. Therefore though features can be represented in many seemingly different ways, for most real world ML problems their predicted or generated data have many intrinsic regularities as well to ensure stability and applicability.</p>
  <p>Besides geometric formulation, ML has many other formulations such as probabilistic (Bayesian) and PAC formulation where sample data are drawn from some unknown probability distribution and ML becomes the task of inferring or approximating distributions rather than finding geometric shapes such as Normalizing Flows or the task of updating a prior (latent) distribution to a posterior using Bayes' theorem such as VAEs.</p>
  <p>For reinforcement learning problems, sequential decision-theoretic formulation is the standard, and in fact classification and regression ML problems can be viewed as one-shot decision problems as well, not just as geometric lines or hyper-surfaces separating or going through data which is hardly applicable for cost sensitive loss with imbalanced data, for instance. Traditional knowledge base learning problems are usually formulated in purely symbolic rules and logic such as <a href="https://en.wikipedia.org/wiki/Inductive_logic_programming" rel="nofollow noreferrer">Inductive Logic Programming (ILP)</a> where symbolic rules are learned from samples. Finally there's information-theoretic <a href="https://en.wikipedia.org/wiki/Minimum_description_length" rel="nofollow noreferrer">Minimum Description Length (MDL)</a> formulation of some ML problems where the goal is to find the model that compresses the observed data most effectively.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-31T07:04:53.393Z
Answers.LastActivityDate: 2024-12-31T07:26:58.230Z
Answers.LastEditDate: 2024-12-31T07:26:58.230Z
Answers.UserDisplayName: cinch
Body: <p><strong>Note: I am not asking what is geometric formulation of machine learning. I am searching alternative formulations that are not reduced to machine learning on manifolds</strong></p>
  <p>Machine learning is usually formulated as a geometric problem (e.g. <a href="https://www.youtube.com/watch?v=c8WkLLtOo_U" rel="nofollow noreferrer">[1]</a>, <a href="https://mathoverflow.net/questions/450858/geometric-formulation-of-the-subject-of-machine-learning">[2]</a>). The data are represented as points in some manifold where coordinates are features, and the task to find another geometric object related to the data. This object can be a line going through the data in case of regression or a hyper-surface separating subsets of the data in case of classification.</p>
  <p>I couldn't find a formulation of machine learning that is not reduced to such geometry.</p>
  <p>To me, it looks like geometric formulation is not fundamental. The space in which data are represented is not unique. It depends on the feature set created for the data. For the same data many (potentially infinite) different feature sets can be created, so the same machine learning problem can be represented in different spaces of different dimensions.</p>
  <p>For example, I work in chemistry and material science. There is a lot of machine learning here, mapping compounds to points in so called 'chemical space' where most predictions fail. I keep asking in seminars and conferences what is this chemical space, its structure, dimension, topology. It looks like there is no definite answer except 'it depends on the descriptor (feature set, a vector of numbers, corresponding to a compound, etc, graphs are also there)'.</p>
  <p>There are thousands of papers about different descriptors, and even multi-volume books devoted solely to descriptors to choose from. According to some descriptors, two molecules are very similar, yet one is benign, another is a poison. According to another descriptor, they're different. That's why I say 'not fundamental' - any feature set says very little about chemical compounds they are supposed to represent.</p>
  <p>Another issue is that geometric formulation requires the notion of distance. Different types of distances can be used which gives different, even contradicting predictions. It is often unknown whether distance and other geometric constructs can make sense at all.</p>
  <p>Is the distance between graphene and amorphous silicon less or greater than the distance between boron nitride and amorphous carbon? Is there triangle inequality in that space? What is greater, the distance between copper oxide and molybdenum dichalcogenide plus the distance between molybdenum dichalcogenide and silicon carbide or the distance between copper oxide and silicon carbide? I think, I can construct features of these substances and distances between them that the answer can be anything you want.</p>
  <p>One way to solve this in chemistry is to continue searching for a better feature set. But then there are materials like VO<sub>2</sub> and TiO<sub>2</sub> whose electron densities are almost indistinguishable, so any set of features would map them into points that are next to each other, yet one of them is a conductor, while another is an insulator.</p>
  <p>So, my question is: Maybe instead of searching for a better feature set that maps data to points in space and designing a distance function that measures similarities between data, I could use some other machine learning paradigm that does not reduce a machine learning problem to a geometric problem?</p>
  <p>Is mapping to geometric objects, and getting inference based on their similarity is the only possible paradigm in machine learning? Are there formulations of machine learning alternative to the geometric one?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-31T10:01:01.127Z
Comments.Text: This question is similar to: [What is geometric deep learning?](https://ai.stackexchange.com/questions/11166/what-is-geometric-deep-learning). If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-31T11:09:27.670Z
Comments.Text: @nbro That question is totally different. That question asks "What is geometric deep learning". I am not asking what is geometric formulation, I know what it is. Moreover, in my question I reiterate what it is. I am asking what are alternative formulations if any. The question you link does not discuss alternative formulations
Comments.UserDisplayName: Vladislav Gladkikh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-31T12:17:00.227Z
Comments.Text: I know you're not asking about a geometric formulation of deep learning. You're asking the opposite. The linked question, however, tells you that geometric deep learning deals with graphs and graphs are non-Euclidean data, which means that many algorithms and models that assume an Euclidean space don't directly work on graphs. The name geometric DL is a bit misleading. (The comment above was generated automatically, btw, when you vote to close a post as a duplicate.)
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-31T12:20:16.503Z
Comments.Text: Your post makes some arguable assumptions, like "geometric formulation is not fundamental." and then you say that the spaces are not unique. I don't see the relation between the two. You can have spaces like $R$, $R^2$, $R^3$, etc., and what's the problem with that? Why would you need to have only one unique space for what you call "geometric formulation" to be fundamental?
Comments.UserDisplayName: nbro
Tags: <machine-learning><reference-request><math><ai-basics>
Title: Is there a non-geometric formulation of machine learning
UserDisplayName: Vladislav Gladkikh
ThreadText: # Is there a non-geometric formulation of machine learning
  
  (Asked by: username_0 on 2024-12-31)
  
  **Note: I am not asking what is geometric formulation of machine learning. I am searching alternative formulations that are not reduced to machine learning on manifolds**
  
  Machine learning is usually formulated as a geometric problem (e.g. [[1]](https://www.youtube.com/watch?v=c8WkLLtOo_U), [[2]](https://mathoverflow.net/questions/450858/geometric-formulation-of-the-subject-of-machine-learning)). The data are represented as points in some manifold where coordinates are features, and the task to find another geometric object related to the data. This object can be a line going through the data in case of regression or a hyper-surface separating subsets of the data in case of classification.
  
  I couldn't find a formulation of machine learning that is not reduced to such geometry.
  
  To me, it looks like geometric formulation is not fundamental. The space in which data are represented is not unique. It depends on the feature set created for the data. For the same data many (potentially infinite) different feature sets can be created, so the same machine learning problem can be represented in different spaces of different dimensions.
  
  For example, I work in chemistry and material science. There is a lot of machine learning here, mapping compounds to points in so called 'chemical space' where most predictions fail. I keep asking in seminars and conferences what is this chemical space, its structure, dimension, topology. It looks like there is no definite answer except 'it depends on the descriptor (feature set, a vector of numbers, corresponding to a compound, etc, graphs are also there)'.
  
  There are thousands of papers about different descriptors, and even multi-volume books devoted solely to descriptors to choose from. According to some descriptors, two molecules are very similar, yet one is benign, another is a poison. According to another descriptor, they're different. That's why I say 'not fundamental' - any feature set says very little about chemical compounds they are supposed to represent.
  
  Another issue is that geometric formulation requires the notion of distance. Different types of distances can be used which gives different, even contradicting predictions. It is often unknown whether distance and other geometric constructs can make sense at all.
  
  Is the distance between graphene and amorphous silicon less or greater than the distance between boron nitride and amorphous carbon? Is there triangle inequality in that space? What is greater, the distance between copper oxide and molybdenum dichalcogenide plus the distance between molybdenum dichalcogenide and silicon carbide or the distance between copper oxide and silicon carbide? I think, I can construct features of these substances and distances between them that the answer can be anything you want.
  
  One way to solve this in chemistry is to continue searching for a better feature set. But then there are materials like VO2 and TiO2 whose electron densities are almost indistinguishable, so any set of features would map them into points that are next to each other, yet one of them is a conductor, while another is an insulator.
  
  So, my question is: Maybe instead of searching for a better feature set that maps data to points in space and designing a distance function that measures similarities between data, I could use some other machine learning paradigm that does not reduce a machine learning problem to a geometric problem?
  
  Is mapping to geometric objects, and getting inference based on their similarity is the only possible paradigm in machine learning? Are there formulations of machine learning alternative to the geometric one?
  
  **Tags:** machine-learning, reference-request, math, ai-basics
  
  > **Comments:**
  > **username_1**: This question is similar to: [What is geometric deep learning?](https://ai.stackexchange.com/questions/11166/what-is-geometric-deep-learning). If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.  
  > **username_0**: @username_1 That question is totally different. That question asks "What is geometric deep learning". I am not asking what is geometric formulation, I know what it is. Moreover, in my question I reiterate what it is. I am asking what are alternative formulations if any. The question you link does not discuss alternative formulations  
  > **username_1**: I know you're not asking about a geometric formulation of deep learning. You're asking the opposite. The linked question, however, tells you that geometric deep learning deals with graphs and graphs are non-Euclidean data, which means that many algorithms and models that assume an Euclidean space don't directly work on graphs. The name geometric DL is a bit misleading. (The comment above was generated automatically, btw, when you vote to close a post as a duplicate.)  
  > **username_1**: Your post makes some arguable assumptions, like "geometric formulation is not fundamental." and then you say that the spaces are not unique. I don't see the relation between the two. You can have spaces like $R$, $R^2$, $R^3$, etc., and what's the problem with that? Why would you need to have only one unique space for what you call "geometric formulation" to be fundamental?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-31. Score: 1
  
  Firstly you claim that geometric formulation of machine learning (M)L is not fundamental and the same ML problem can be represented in different spaces of different dimensions. However, the essence of contemporary deep learning is built from two simple principles known as feature learning and local gradient-descent. Many
  high-dimensional learning tasks previously thought to be beyond reach, such as computer vision, playing Go, or protein folding, are in fact feasible after adapted learned features capture the appropriate notion of *regularity* for each task, such as semantic similarity (GPT and BERT large variant pretrained token embeddings have different dimensions and representations, but both share regularity of semantic similarity) and group symmetry (translational and rotational equivariance & invariance of CNNs).
  
  Similarly for the geometric feature of distance, each ML task will have inductive bias for such feature if it's really relevant with domain knowledge. Therefore though features can be represented in many seemingly different ways, for most real world ML problems their predicted or generated data have many intrinsic regularities as well to ensure stability and applicability.
  
  Besides geometric formulation, ML has many other formulations such as probabilistic (Bayesian) and PAC formulation where sample data are drawn from some unknown probability distribution and ML becomes the task of inferring or approximating distributions rather than finding geometric shapes such as Normalizing Flows or the task of updating a prior (latent) distribution to a posterior using Bayes' theorem such as VAEs.
  
  For reinforcement learning problems, sequential decision-theoretic formulation is the standard, and in fact classification and regression ML problems can be viewed as one-shot decision problems as well, not just as geometric lines or hyper-surfaces separating or going through data which is hardly applicable for cost sensitive loss with imbalanced data, for instance. Traditional knowledge base learning problems are usually formulated in purely symbolic rules and logic such as [Inductive Logic Programming (ILP)](https://en.wikipedia.org/wiki/Inductive_logic_programming) where symbolic rules are learned from samples. Finally there's information-theoretic [Minimum Description Length (MDL)](https://en.wikipedia.org/wiki/Minimum_description_length) formulation of some ML problems where the goal is to find the model that compresses the observed data most effectively.
  
  
  ---
  
  