<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>
<p dir="auto">A programmatic means (property) to disambiguate when a <code class="notranslate">_LRScheduler</code> should be stepped.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">Currently, there are a variety of <code class="notranslate">_LRScheduler</code>s found in <code class="notranslate">torch.optim.lr_scheduler</code>. While the majority should be stepped per epoch (e.g., <code class="notranslate">LambdaLR</code>, <code class="notranslate">MultiplicativeLR</code>, <code class="notranslate">StepLR</code>, etc), some are designed to be stepped per training batch (<code class="notranslate">CyclicLR</code>, <code class="notranslate">OneCycleLR</code>, etc). Unfortunately, there is no way (programmatically) to identify which category an arbitrary <code class="notranslate">_LRScheduler</code> falls into without either maintaining a hard-coded list (<g-emoji class="g-emoji" alias="cry" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f622.png">😢</g-emoji>) or parsing the docstring (<g-emoji class="g-emoji" alias="cry" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f622.png">😢</g-emoji>  <g-emoji class="g-emoji" alias="cry" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f622.png">😢</g-emoji>). This makes correctly injecting arbitrary <code class="notranslate">_LRScheduler</code>s difficult to do in an extensible fashion.</p>
<h2 dir="auto">Pitch</h2>
<p dir="auto">Add two boolean properties to every <code class="notranslate">_LRScheduler</code>:</p>
<ul dir="auto">
<li>steps_on_batch</li>
<li>steps_on_epoch</li>
</ul>
<p dir="auto">Currently, neither symbol is used in <a href="https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html" rel="nofollow"><code class="notranslate">torch.optim.lr_scheduler</code></a>, so this should be a strictly additive change as far as I'm aware.</p>
<h2 dir="auto">Alternatives</h2>
<p dir="auto">Every scheduler that exists in PyTorch today exclusively steps per batch or per epoch. Therefore, either of the two properties alone would currently satisfy the aforementioned motivation. However, should a scheduler be created that is stepped at both stages, this option would fail. Later adding the second property would be a pseudo-backwards incompatible change as significant amounts of code would likely include <code class="notranslate">if not sched.step_epoch: ...</code> to determine if the scheduler should be stepped per batch (or vice versa).</p>
<p dir="auto">Given the minimal cost of adding both, the flexibility it enables, and the potential to mitigate future challenges, I have opted to propose both properties.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vincentqb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vincentqb">@vincentqb</a></p>
<p dir="auto">I'm not very sure that what is the parameter "tgt" means.<br>
The document says it is the sequence of decoder. However, I have a question that since I have the sequence to decoder, thus I have the result of decoder, why do I have to use transformer for inferencing?<br>
For example, I have a seq2seq task that translate "I love you" from English to Franch "Je t'aime", what is the parameter of "tgt"?<br>
Looking forward answering. Thanks!</p>