Answers.Body: <p>Regression with LLMs is definitely possible. Assuming you use a GPT-like model, you can either</p>
  <ol>
  <li>train the transformer from scratch on the regression task, or</li>
  <li>first pre-train the transformer on a general task, and then transfer learn the regression task by replacing the final linear layer.</li>
  </ol>
  <p>Which option is more appropriate depends on the kind of regression task and how well that task is generalizable. For example, if you want to assess how positive a text is, you can better do that using option 2. However, a super-specific regression task is likely easier to do through option 1.</p>
  <p>Check out <a href="https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work">this</a> post for more general knowledge on transformers, and a bit of context about 'pretraining' and 'transfer learning'.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T03:02:59.770Z
Answers.Text: Thanks! Does the answer change if the data is mostly numerical and categorical data where not much text is involved?
Answers.UserDisplayName: sharkeater123
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T06:00:54.387Z
Answers.Text: You can still use transformers for numerical data. Neural nets have never been veey good at categorical data, but it is possible. The approach is still the same. If its generalizable then use 1 otherwise 2
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T14:05:57.493Z
Answers.Text: Would you recommend LLMs for tabular data in general? I feel like there are easier ways to accomplish the task at hand. The post was helpful, are there any other resources you can share? Thanks!
Answers.UserDisplayName: sharkeater123
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T14:55:37.357Z
Answers.Text: Transformers are very general architectures. Although they were introduced for NLP they can be used for static data as well (tho arguably you wouldnt need positional encoding then). However, if you have static data, simple MLP architectures can get you started faster probably. For graphs, which you mention, you might actually wanna look into graph-neural-networks. They are super interesting and might be more suited for your problem.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T16:00:21.013Z
Answers.Text: Awesome, thanks for answering. I was told to use GPT-4 to work on predicting edge weights, but I can't imagine it being super helpful which is why I asked. To be clear, applying GPT to static data prediction would take a lot of setup right?
Answers.UserDisplayName: sharkeater123
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T08:10:58.727Z
Answers.Text: It is possible, but for graphs its simply not the right architecture
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T14:50:08.883Z
Answers.Text: The overall graph structure doesn't mean much. It's more so about the attributes associated with two nodes and the edge. It's essentially just predicting a price based on numerical and categorical features. GPT is not the right architecture for this?
Answers.UserDisplayName: sharkeater123
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T15:27:28.663Z
Answers.Text: Theres many architectures that can be used for such a task. If you'd like specific advice on  how to approach a problem, i'd recommend starting a new question, detailing your dataset/use-case/target and asking what kind of architecture would be most suitable. This question was opened for whether transformers can be used for regression tasks, not for advice on a specific problem (although that was the intention behind it).
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-17T21:24:21.990Z
Answers.LastActivityDate: 2023-05-17T21:24:21.990Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I want to use LLMs to predict edge weights in a graph based on attributes between two nodes. Is this even possible? If not, what would you recommend?</p>
  <p>I tried to look up uses of LLM in regression tasks, but haven't had much luck finding anything helpful.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-27T16:29:12.307Z
Comments.Text: A better question might be, why? are other well-understood approaches not sufficient? Large _language_ models don't seem like a good place to start, unless somehow the weights will depend on understanding text.
Comments.UserDisplayName: Sean Owen
Tags: <neural-networks><machine-learning><regression><large-language-models>
Title: Is it possible to use LLMs for regression tasks?
UserDisplayName: sharkeater123
ThreadText: # Is it possible to use LLMs for regression tasks?
  
  (Asked by: username_0 on 2023-05-17)
  
  I want to use LLMs to predict edge weights in a graph based on attributes between two nodes. Is this even possible? If not, what would you recommend?
  
  I tried to look up uses of LLM in regression tasks, but haven't had much luck finding anything helpful.
  
  **Tags:** neural-networks, machine-learning, regression, large-language-models
  > **Comments:**
  > **username_1**: A better question might be, why? are other well-understood approaches not sufficient? Large _language_ models don't seem like a good place to start, unless somehow the weights will depend on understanding text.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-17. Score: 3 | Accepted answer
  
  Regression with LLMs is definitely possible. Assuming you use a GPT-like model, you can either
  
  1. train the transformer from scratch on the regression task, or
  2. first pre-train the transformer on a general task, and then transfer learn the regression task by replacing the final linear layer.
  
  Which option is more appropriate depends on the kind of regression task and how well that task is generalizable. For example, if you want to assess how positive a text is, you can better do that using option 2. However, a super-specific regression task is likely easier to do through option 1.
  
  Check out [this](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) post for more general knowledge on transformers, and a bit of context about 'pretraining' and 'transfer learning'.
  
  > **Comments:**
  > **username_0**: Thanks! Does the answer change if the data is mostly numerical and categorical data where not much text is involved?  
  > **username_2**: You can still use transformers for numerical data. Neural nets have never been veey good at categorical data, but it is possible. The approach is still the same. If its generalizable then use 1 otherwise 2  
  > **username_0**: Would you recommend LLMs for tabular data in general? I feel like there are easier ways to accomplish the task at hand. The post was helpful, are there any other resources you can share? Thanks!  
  > **username_2**: Transformers are very general architectures. Although they were introduced for NLP they can be used for static data as well (tho arguably you wouldnt need positional encoding then). However, if you have static data, simple MLP architectures can get you started faster probably. For graphs, which you mention, you might actually wanna look into graph-neural-networks. They are super interesting and might be more suited for your problem.  
  > **username_0**: Awesome, thanks for answering. I was told to use GPT-4 to work on predicting edge weights, but I can't imagine it being super helpful which is why I asked. To be clear, applying GPT to static data prediction would take a lot of setup right?  
  > **username_2**: It is possible, but for graphs its simply not the right architecture  
  > **username_0**: The overall graph structure doesn't mean much. It's more so about the attributes associated with two nodes and the edge. It's essentially just predicting a price based on numerical and categorical features. GPT is not the right architecture for this?  
  > **username_2**: Theres many architectures that can be used for such a task. If you'd like specific advice on  how to approach a problem, i'd recommend starting a new question, detailing your dataset/use-case/target and asking what kind of architecture would be most suitable. This question was opened for whether transformers can be used for regression tasks, not for advice on a specific problem (although that was the intention behind it).  
  
  ---
  
  