title: How to parse python code line-by-line until an expression is complete
body: <p>I have a multiline string of text, and I want to parse a portion of python code in a line-by-line manner, such that I have a list of string with each item representing its own python statement. </p>
  
  <p>Unfortunately, I just built an AST of the entire text because portions of the text will not contain valid Python syntax. Each python statement may also span multiple lines. However, for each valid statement, I do know which line it starts on. However, I can't distinguish between invalid syntax and continuations of previous lines. </p>
  
  <p>For example I might have something like this (I'll add a comment denoting which lines I know start valid syntax, and which lines are either invalid syntax or a continuation of the previous statement)</p>
  
  <pre><code>foo = bar()                 # valid-start 
  this = (                    # valid-start
  'perfectly valid syntax'    # unknown
  )                           # unknown
  44x=but-this-is-bad-syntax  # unknown
  </code></pre>
  
  <p>The desired output here is a list of tuples, with the first item denoting if the statement is valid python or junk, and the second item being the text corresponding to that statement. </p>
  
  <pre><code>[
      ('PY', 'foo = bar()'),
      ('PY', """this = (                    
      'perfectly valid syntax'    
      )"""),                           
      ('JUNK', '44x=but-this-is-bad-syntax')
  ]
  </code></pre>
  
  <p>One solution I considered was checking to see if the parenthesis were balanced, but this becomes tricky when strings are involved (I also haven't convinced myself this works in all cases yet).</p>
  
  <pre><code>foo = bar()                 # valid-start 
  this = '''                  # valid-start
      this is still ()        # unknown
      perfectly valid )))     # unknown
  '''                         # unknown
  z = '''                     # unknown
    even though this is valid # unknown
    syntax, I don't want this # unknown
    line grouped'''           # unknown
  44x=but-this-is-bad-syntax  # unknown
  a = 1                       # valid-start
  </code></pre>
  
  <p>should produce something like this output:</p>
  
  <pre><code>[
      ('PY', 'foo = bar()'),
      ('PY', """this = '''                  
      this is still ()        
      perfectly valid )))     
      '''"""),                           
      ('JUNK', "z = '''"),
      ('JUNK', "even though this is valid"),
      ('JUNK', "syntax, I don't want this"),
      ('JUNK', "line grouped'''"),
      ('JUNK', "line grouped"),
      ('JUNK', "44x=but-this-is-bad-syntax"),
      ('PY', "a = 1"),
  ]
  </code></pre>
  
  <p>Note in the last example, the line starting with <code>z = '''</code> is marked as unknown. Even though continuing with it would still produce valid syntax, I want to stop parsing the statement starting with <code>this = '''</code> exactly once it becomes valid syntax, (i.e. <code>z = '''</code> would not be included)</p>
  
  <p>Does anyone have an idea on how this might be done?</p>
  
  <p>Would a pyparsing solution, that simply checks for balanced parenthesis while taking strings into account be sufficient? The idea is I would define a grammar that accepts balanced parenthesis / square brackets / curly braces, where the body the nesting could be any sequence of characters or a string (which may contain parens, but those wont be counted towards balance). Then, I would parse lines with this grammar until the reconstructed lines exactly equaled the original lines. </p>
  
  <p>Does anyone see a problem with the previous approach / does anyone have a simpler method for doing this that doesn't involve a dependency on pyparsing?</p>
  
  <h2>EDIT</h2>
  
  <p>Based on the answer from @rici I came up with a function that can take a list of lines, and returns True if the lines form a complete statement and False otherwise.  </p>
  
  <pre><code>import tokenize
  from six.moves import cStringIO as StringIO
  
  def is_complete_statement(lines):
      """
      Checks if the lines form a complete python statment.
      """
      try:
          stream = StringIO()
          stream.write('\n'.join(lines))
          stream.seek(0)
          for t in tokenize.generate_tokens(stream.readline):
              pass
      except tokenize.TokenError as ex:
          message = ex.args[0]
          if message.startswith('EOF in multi-line'):
              return False
          raise
      else:
          return True
  </code></pre>