Answers.Body: <p>After reading different sources and writing my own minimal DT code I realized, that the DT is basically &quot;picking up&quot; the best pieces of trajectories given offline data.</p>
  <p>Anyway a relatively new <em>online decision transformer</em> appeared in the RL world, which makes it more interesting then the original decision transformer</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-07T13:17:38.297Z
Answers.LastActivityDate: 2024-07-07T13:17:38.297Z
Answers.UserDisplayName: Dave
Body: <p>I'm studying the <a href="https://sites.google.com/berkeley.edu/decision-transformer" rel="nofollow noreferrer">Decision Transformer</a> for some offline reinforcement learning tasks. The basic idea is to collect a huge quantity of data generated by a real experimental device (let's say an arm manipulator) and then apply offline reinforcement learning for determining a policy, which maximize for instance the current consumption.</p>
  <p>After reading the work of <a href="https://arxiv.org/pdf/2005.01643.pdf" rel="nofollow noreferrer">Sergey Levine</a> et al. and look of some of his videos (really good by the way), I thought at offline reinforcement learning as a new possibility to learn a policy from collected real data, avoiding to use a simulator or something like that. But as stated in the link above at page 29:</p>
  <p><em>&quot;A reasonable question we might ask in regard to datasets for offline RL is: in which situations might we actually expect offline RL to yield a policy that is significantly better than any trajectory in the training set? While we cannot expect offline RL to discover actions that are better than any action illustrated in the data, we can expect it to effectively utilize the compositional structure inherent in any temporal process. This idea is illustrated in Figure 4: if the dataset contains a subsequence illustrating a way to arrive at state 2 from state 1, as well as a separate subsequence illustrating how to arrive at state 3 from state 2, then an effective offline RL method should be able to learn how to arrive at state 3 from state 1, which might provide for a substantially higher final reward than any of the subsequences in the dataset. When we also consider the capacity of
  neural networks to generalize, we could imagine this sort of “transitive induction” taking place on a portion of the state variables, effectively inferring potentially optimal behavior from highly suboptimal components. This capability can be evaluated with benchmarks that explicitly provide data containing this structure, and the D4RL benchmark suite provides a range of tasks that exercise this capability.&quot;</em> <a href="https://arxiv.org/pdf/2005.01643.pdf" rel="nofollow noreferrer">Sergey Levine</a></p>
  <p>So as far as I understood, it is (at the moment) only auspicable to determine an offline policy by &quot;combining&quot; subsequences together in order to achieve a desired task or to reach a desired return.</p>
  <p>Then I move over and discovered the <a href="https://sites.google.com/berkeley.edu/decision-transformer" rel="nofollow noreferrer">Decision Transformer</a>, which look very elegant and powerful, since they would basically replace the whole RL concept with a single Transformer.
  Despite the fact, that the benchmark results of the DT reported in the paper are in some cases better than CQL and BC (respectively for Conservative Q Learning and Behavioral Cloning), I have to admit, that DTs just combine &quot;trajectory pieces&quot; and do not determine a &quot;new&quot; policy for a specific task.</p>
  <p>Is it right? Or am I missing something?</p>
  <p>I'm looking for a good reason, to apply DTs in my tasks in order to determine an optimal policy (offline) compared to conventional offline methods.</p>
  
Tags: <transformer>
Title: Decision Transformer: more than a "trajectory picking" algorithm?
UserDisplayName: Dave
ThreadText: # Decision Transformer: more than a "trajectory picking" algorithm?
  
  (Asked by: username_0 on 2024-04-20)
  
  I'm studying the [Decision Transformer](https://sites.google.com/berkeley.edu/decision-transformer) for some offline reinforcement learning tasks. The basic idea is to collect a huge quantity of data generated by a real experimental device (let's say an arm manipulator) and then apply offline reinforcement learning for determining a policy, which maximize for instance the current consumption.
  
  After reading the work of [Sergey Levine](https://arxiv.org/pdf/2005.01643.pdf) et al. and look of some of his videos (really good by the way), I thought at offline reinforcement learning as a new possibility to learn a policy from collected real data, avoiding to use a simulator or something like that. But as stated in the link above at page 29:
  
  *"A reasonable question we might ask in regard to datasets for offline RL is: in which situations might we actually expect offline RL to yield a policy that is significantly better than any trajectory in the training set? While we cannot expect offline RL to discover actions that are better than any action illustrated in the data, we can expect it to effectively utilize the compositional structure inherent in any temporal process. This idea is illustrated in Figure 4: if the dataset contains a subsequence illustrating a way to arrive at state 2 from state 1, as well as a separate subsequence illustrating how to arrive at state 3 from state 2, then an effective offline RL method should be able to learn how to arrive at state 3 from state 1, which might provide for a substantially higher final reward than any of the subsequences in the dataset. When we also consider the capacity of
  neural networks to generalize, we could imagine this sort of “transitive induction” taking place on a portion of the state variables, effectively inferring potentially optimal behavior from highly suboptimal components. This capability can be evaluated with benchmarks that explicitly provide data containing this structure, and the D4RL benchmark suite provides a range of tasks that exercise this capability."* [Sergey Levine](https://arxiv.org/pdf/2005.01643.pdf)
  
  So as far as I understood, it is (at the moment) only auspicable to determine an offline policy by "combining" subsequences together in order to achieve a desired task or to reach a desired return.
  
  Then I move over and discovered the [Decision Transformer](https://sites.google.com/berkeley.edu/decision-transformer), which look very elegant and powerful, since they would basically replace the whole RL concept with a single Transformer.
  Despite the fact, that the benchmark results of the DT reported in the paper are in some cases better than CQL and BC (respectively for Conservative Q Learning and Behavioral Cloning), I have to admit, that DTs just combine "trajectory pieces" and do not determine a "new" policy for a specific task.
  
  Is it right? Or am I missing something?
  
  I'm looking for a good reason, to apply DTs in my tasks in order to determine an optimal policy (offline) compared to conventional offline methods.
  
  **Tags:** transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-07-07. Score: 0 | Accepted answer
  
  After reading different sources and writing my own minimal DT code I realized, that the DT is basically "picking up" the best pieces of trajectories given offline data.
  
  Anyway a relatively new *online decision transformer* appeared in the RL world, which makes it more interesting then the original decision transformer
  
  
  ---
  
  