title: regex as tokenizer - string beginning with delimiter
body: <p>sregex_token_iterator works almost perfectly as a tokenizer when the index of the submatch is specified to be -1. But unfortunately it doesn't work well with strings that begin with delimiters e.g:</p>
  
  <pre><code>#include &lt;string&gt;
  #include &lt;regex&gt;
  #include &lt;iostream&gt;
  using namespace std;
  
  int main()
  {
      string s("--aa---b-c--d--");
      regex r("-+");
  
      for (sregex_token_iterator it = sregex_token_iterator(s.begin(), s.end(), r, -1); it != sregex_token_iterator(); ++it)
      {
          cout &lt;&lt; (string) *it &lt;&lt; endl;
      }
  
      return 0;
  }
  </code></pre>
  
  <p>prints out:</p>
  
  <blockquote>
    <p>&nbsp;<br>
    <code>aa</code><br>
    <code>b</code><br>
    <code>c</code><br>
    <code>d</code></p>
  </blockquote>
  
  <p>(Note the leading empty line).</p>
  
  <p>So note that it actually handles trailing delimeters well (as it doesn't print an extra empty line).</p>
  
  <p>Reading the standard it seems like there is a clause for specifically handling trailing delimeter to work well i.e:</p>
  
  <p><code>[re.tokiter] no 4.</code></p>
  
  <blockquote>
    <p>If the end of sequence is reached (position is equal to the end of sequence iterator), the iterator becomes equal to the end-of-sequence iterator value, unless the sub-expression being enumerated has index -1, in which case the iterator enumerates one last sub-expression that contains all the characters from the end of the last regular expression match to the end of the input sequence being enumerated, provided that this
    <strong>would not be an empty sub-expression.</strong></p>
  </blockquote>
  
  <p>Does anyone know what's the reason for this seemingly asymmetric behaviour being specified?</p>
  
  <p>And lastly, is there an elegant solution to make this work? (such that we don't have empty entries at all).</p>