Answers.Body: <p>In the context of <a href="https://ai.stackexchange.com/a/10624/2444">self-supervised learning</a> (which is also used in NLP), a <strong>downstream task</strong> is the task that you actually want to solve. This definition makes sense if you're familiar with transfer learning or self-supervised learning, which are also used for NLP. In particular, in transfer learning, you first pre-train a model with some &quot;general&quot; dataset (e.g. ImageNet), which does <em>not</em> represent the task that you want to solve, but allows the model to learn some &quot;general&quot; features. Then you fine-tune this pre-trained model on the dataset that represents the actual problem that you want to solve. This latter task/problem is what would be called, in the context of self-supervised learning, a <strong>downstream task</strong>. In <a href="https://ai.stackexchange.com/a/10624/2444">this answer</a>, I mention these downstream tasks.</p>
  <p>In the same book that you quote, the author also writes (section <strong>14.6.2 Extrinsic evaluations</strong>, p. 339 of the book)</p>
  <blockquote>
  <p>Word representations contribute to downstream tasks like <strong>sequence labeling</strong> and <strong>document classification</strong> by enabling generalization across words. The use of distributed representations as features is a form of semi-supervised learning, in which performance on a supervised learning problem is augmented by learning distributed representations from unlabeled data (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010). These <strong>pre-trained</strong> word representations can be used as features in a linear prediction model, or as the input layer in a neural network, such as a Bi-LSTM tagging model (§ 7.6). Word representations can be evaluated by the performance of the downstream systems that consume them: for example, GloVe embeddings are convincingly better than Latent Semantic Analysis as features in the downstream task of <strong>named entity recognition</strong> (Pennington et al., 2014). Unfortunately, extrinsic and intrinsic evaluations do not always point in the same direction, and the best word representations for one downstream task may perform poorly on another task (Schnabel et al., 2015).</p>
  <p>When word representations are updated from labeled data in the downstream task, they are said to be <strong>fine-tuned</strong>.</p>
  </blockquote>
  <p>So, to me, after having read this section of the book, it seems that the author is using the term &quot;downstream task&quot; as it's used in self-supervised learning. Examples of downstream tasks are thus</p>
  <ol>
  <li>sequence labeling</li>
  <li>documentation classification</li>
  <li>named entity recognition</li>
  </ol>
  <p>Tasks like training a model to learn word embeddings are not downstream tasks, because these tasks are not really the ultimate tasks that you want to solve, but they are solved in order to solve other tasks (i.e. the downstream tasks).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-08T08:01:06.270Z
Answers.Text: Your answer is correct. **Features
  are transformations on input data that facilitate a downstream algorithm, like a classifier, to produce correct outcomes on new data**  It is given [here](https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=34)
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-23T20:19:01.110Z
Answers.Text: Are there any upstream tasks as opposed to downstream tasks?
Answers.UserDisplayName: b.g.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-27T15:07:15.570Z
Answers.LastActivityDate: 2021-10-08T11:45:55.273Z
Answers.LastEditDate: 2021-10-08T11:45:55.273Z
Answers.UserDisplayName: nbro
Body: <p>The following paragraph is from page no 331 of the textbook <a href="https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf#page=350" rel="noreferrer">Natural Language Processing</a> by <em>Jacob Eisenstein</em>. It mentions about certain type of tasks called as downstream tasks. But, it provide no further examples or details regarding these tasks.</p>
  <blockquote>
  <p>Learning algorithms like perceptron and conditional random fields
  often perform better with discrete feature vectors. A simple way to
  obtain discrete representations from distributional statistics is by
  clustering, so that words in the same cluster have similar
  distributional statistics. This can help in <strong>downstream tasks</strong>, by
  sharing features between all words in the same cluster. However, there
  is an obvious tradeoff: if the number of clusters is too small, the
  words in each cluster will not have much in common; if the number of
  clusters is too large, then the learner will not see enough examples
  from each cluster to generalize.</p>
  </blockquote>
  <p>Which tasks in artificial intelligence or NLP are called as downstream tasks?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-08T08:08:50.560Z
Comments.Text: [downstream](https://www.lexico.com/en/definition/downstream)
Comments.UserDisplayName: hanugm
Tags: <natural-language-processing><terminology>
Title: Which tasks are called as downstream tasks?
UserDisplayName: hanugm
ThreadText: # Which tasks are called as downstream tasks?
  
  (Asked by: username_0 on 2021-06-26)
  
  The following paragraph is from page no 331 of the textbook [Natural Language Processing](https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf#page=350) by *Jacob Eisenstein*. It mentions about certain type of tasks called as downstream tasks. But, it provide no further examples or details regarding these tasks.
  
  > Learning algorithms like perceptron and conditional random fields
  > often perform better with discrete feature vectors. A simple way to
  > obtain discrete representations from distributional statistics is by
  > clustering, so that words in the same cluster have similar
  > distributional statistics. This can help in **downstream tasks**, by
  > sharing features between all words in the same cluster. However, there
  > is an obvious tradeoff: if the number of clusters is too small, the
  > words in each cluster will not have much in common; if the number of
  > clusters is too large, then the learner will not see enough examples
  > from each cluster to generalize.
  
  Which tasks in artificial intelligence or NLP are called as downstream tasks?
  
  **Tags:** natural-language-processing, terminology
  
  > **Comments:**
  > **username_0**: [downstream](https://www.lexico.com/en/definition/downstream)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-27. Score: 15 | Accepted answer
  
  In the context of [self-supervised learning](https://ai.stackexchange.com/a/10624/2444) (which is also used in NLP), a **downstream task** is the task that you actually want to solve. This definition makes sense if you're familiar with transfer learning or self-supervised learning, which are also used for NLP. In particular, in transfer learning, you first pre-train a model with some "general" dataset (e.g. ImageNet), which does *not* represent the task that you want to solve, but allows the model to learn some "general" features. Then you fine-tune this pre-trained model on the dataset that represents the actual problem that you want to solve. This latter task/problem is what would be called, in the context of self-supervised learning, a **downstream task**. In [this answer](https://ai.stackexchange.com/a/10624/2444), I mention these downstream tasks.
  
  In the same book that you quote, the author also writes (section **14.6.2 Extrinsic evaluations**, p. 339 of the book)
  
  > Word representations contribute to downstream tasks like **sequence labeling** and **document classification** by enabling generalization across words. The use of distributed representations as features is a form of semi-supervised learning, in which performance on a supervised learning problem is augmented by learning distributed representations from unlabeled data (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010). These **pre-trained** word representations can be used as features in a linear prediction model, or as the input layer in a neural network, such as a Bi-LSTM tagging model (§ 7.6). Word representations can be evaluated by the performance of the downstream systems that consume them: for example, GloVe embeddings are convincingly better than Latent Semantic Analysis as features in the downstream task of **named entity recognition** (Pennington et al., 2014). Unfortunately, extrinsic and intrinsic evaluations do not always point in the same direction, and the best word representations for one downstream task may perform poorly on another task (Schnabel et al., 2015).
  >
  > When word representations are updated from labeled data in the downstream task, they are said to be **fine-tuned**.
  
  So, to me, after having read this section of the book, it seems that the author is using the term "downstream task" as it's used in self-supervised learning. Examples of downstream tasks are thus
  
  1. sequence labeling
  2. documentation classification
  3. named entity recognition
  
  Tasks like training a model to learn word embeddings are not downstream tasks, because these tasks are not really the ultimate tasks that you want to solve, but they are solved in order to solve other tasks (i.e. the downstream tasks).
  
  > **Comments:**
  > **username_0**: Your answer is correct. **Features are transformations on input data that facilitate a downstream algorithm, like a classifier, to produce correct outcomes on new data**  It is given [here](https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=34)  
  > **username_2**: Are there any upstream tasks as opposed to downstream tasks?  
  
  ---
  
  