
> -----Original Message-----
> From: Jean-Paul Calderone [mailto:exarkun@...]
> Sent: Saturday, May 13, 2006 1:37 PM
> To: Paul McGuire
> Cc: pyparsing-users@...
> Subject: Re: [Pyparsing] RE: Pyparsing unicode
>
<snip>
>
> I took your lead and produced the attached stand-alone test
> based on a couple actions in Imaginary. It includes a
> slightly different version of targetString (one that uses
> removeQuotes) and shows two different failures. Any insights
> you have would be appreciated.
>
> Jean-Paul
>
Great test case!
Ok, here is the problem: in 1.4.2, I expanded the concept of parse actions
to support a list of parse actions, to be executed in a chain, instead of
just one. Every call to setParseAction *adds* the current action to the
expression's list of parse actions (yes, I see that this is *not* intuitive,
and I will fix this! Instead of expanding the behavior of setParseAction, I
should have added a new method, something like addParseAction, or something
that clearly implies that we are adding the current parse action to whatever
other actions have already been defined). Because you are calling
setParseAction on the global quotedString, the results get pared back again
and again. I might also adjust removeQuotes to only remove the first and
last characters if they are in fact quote characters - I'm not sure about
this yet, though.
Here are 2 possible workarounds for you:
1. use a copy of quotedString in your definition of qstr inside
targetString. Change:
qstr = pyparsing.quotedString.setParseAction(pyparsing.removeQuotes)
To:
qstr =
pyparsing.quotedString.copy().setParseAction(pyparsing.removeQuotes)
2. Take care to only attach removeQuotes to quotedString *once*, perhaps
right after calling import.
import pyparsing
pyparsing.quotedString.setParseAction(pyparsing.removeQuotes)
Then inside targetString, just assign qstr as:
qstr = pyparsing.quotedString
Sorry for the confusion! Let me know what you settle on doing.
-- Paul
On Sat, 13 May 2006 12:35:02 -0500, Paul McGuire
>
.)
Hey Paul,
Thanks for the quick response.
First, regarding the mutation of the toks argument, I believe stripper is the only function in Imaginary which does this. Since you pointed out pyparsing.removeQuotes, I'll just use that instead, and in the future I'll keep in mind that mutating that object might not be a good idea.
I took your lead and produced the attached stand-alone test based on a couple actions in Imaginary. It includes a slightly different version of targetString (one that uses removeQuotes) and shows two different failures. Any insights you have would be appreciated.
Jean-Paul
> -----Original Message-----
> From: Jean-Paul Calderone [mailto:exarkun@...]
> Sent: Saturday, May 13, 2006 11:41 AM
> To: Paul McGuire
> Subject: Re: Pyparsing unicode
>
> On Fri, 5 May 2006 15:49:10 -0500, Paul McGuire
> <paul@...> wrote:
> >J-P,
> >
> >How are things for you with this issue? Is upgrading to 1.4.2 an
acceptable
> >option for you? I'm seeing through Google that this is giving you a fair
> >amount of heartburn - I'm sorry for this, let me know if things are still
a
> >problem for you.
>
> Hey Paul,
>
> I finally got a chance to try out 1.4.2 with Imaginary. The
> unicode difficulties seem to be resolved (at least, my test
> suite no longer complains about getting str instead of
> unicode), but the previously broken tests still don't
> actually pass, and actually five new tests are now failing.
>
> I haven't looked at the failures closely yet, but it looks
> like they are mostly in areas which rely on quoted strings
> and are failing because instead of the content of the string
> being handed back an empty string is received instead.
>
> Maybe my "targetString" expression isn't being defined
> correctly? Here's the definition again:
>
> def stripper(s, loc, toks):
> toks = toks.asList()
> toks[0] = toks[0][1:-1]
> return toks
>
> def targetString(name):
> qstr = pyparsing.quotedString.setParseAction(stripper)
> return (
> pyparsing.Word(pyparsing.alphanums) ^
> qstr).setResultsName(name)
>
> Failures from the test suite mostly come out looking like this:
>
> twisted.trial.unittest.FailTest:
> ["> create pants 'pair of daisy dukes'", ' created.']
> did not match expected
> [u"> create pants 'pair of daisy dukes'", 'Pair of daisy dukes
created.']
> (Line 1)
>
> and:
>
> twisted.trial.unittest.FailTest:
> ["> create 'vending machine' vendy", "Can't find ."]
> did not match expected
> [u"> create 'vending machine' vendy", 'Vendy created.']
> (Line 1)
>
> I'll probably investigate this a bit further myself today,
> but if you have any hints, they'd be much appreciated.
>
> Feel free to CC the pyparsing list on your response, if you'd
> like to take this discussion back there.
>
> Thanks,
>
> Jean-Paul
>
>
Jean-Paul,.)
-- Paul
--------------
import pyparsing
def stripper(s, loc, toks):
toks = toks.asList()
toks[0] = toks[0][1:-1] # <--- assigning to toks
return toks
def targetString(name):
qstr = pyparsing.quotedString.setParseAction(stripper)
# stripper could be replaced with the removeQuotes parse action
# qstr = pyparsing.quotedString.setParseAction(pyparsing.removeQuotes)
return (
pyparsing.Word(pyparsing.alphanums) ^
qstr).setResultsName(name)
input = u""" this is a Unicode string with a 'quoted string' in it """
for toks,s,e in targetString("word").scanString(input):
print toks.asList()
--------------
Gives me the result:
[u'this']
[u'is']
[u'a']
[u'Unicode']
[u'string']
[u'with']
[u'a']
[u'quoted string']
[u'in']
[u'it']
Oh, here are the results under 1.3.3:
1.3.3
["'foo'"]
['foo']
And here are the results under 1.4.2 (the current released version):
1.4.2
[u"'foo'"]
[u'foo']
So I definitely see this problem exists under 1.3.3, but I'd rather not go
through a release on this old version track if I can help it. Is it a
problem for you to upgrade to 1.4.2?
-- Paul
Jean-Paul,
Here are my tests with the latest version:
from pyparsing import quotedString,__version__
print __version__
def stripper(s, loc, toks):
toks = toks.asList()
toks[0] = toks[0][1:-1]
return toks
input = u"'foo'"
print quotedString.parseString(input)
quotedString.setParseAction(stripper)
print quotedString.parseString(input)
Prints:
1.4.3
[u"'foo'"]
[u'foo']
Can you send me an example of how the behavior changes depending on usage?
So far, it looks like the current release does the right thing.
-- Paul
Jean-Paul -
My first thought is that this is a bug in pyparsing. I'll look into what
changed around the 1.3 time frame to see what may have caused this.
There is also a more recent version of pyparsing than 1.3.3, you might
download from SF and give it a try. I don't expect it to be different in
this respect tho.
-- Paul
Hey All,
I've been using PyParsing to handle commands in Imaginary (formerly Pottery). So far it's done most of the things I've asked of it, and I think I have some ideas to work around the rest, but the behavior with respect to unicode is a bit confusing.
In 1.2 (Ubuntu Breezy packaged version), I could parse a unicode string and get back a unicode string:
exarkun@...:~$ python
Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyparsing
>>> pyparsing.__version__
'1.2'
>>> pyparsing.quotedString.parseString(u"'foo'")
([u"'foo'"], {})
>>>
exarkun@...:~$
However, on upgrading to 1.3 (Ubuntu Dapper packaged version), this no longer appears to be the case:
exarkun@...:~$ python
Python 2.4.3 (#2, Apr 27 2006, 14:43:58)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyparsing
>>> pyparsing.__version__
'1.3.3'
>>> pyparsing.quotedString.parseString(u"'foo'")
(["'foo'"], {})
>>>
exarkun@...:~$
More confusing, this behavior seems to depend on the exact expression you use to parse a string: sometimes the result will come out as unicode, sometimes not. The exact expression I am using (created by the targetString function here <>) allows either quoted or unquoted strings and, frustratingly, if the quotes are supplied the result is a str, but if they are omitted the result is unicode.
I have considered wrapping my usage of PyParsing in an extra layer that does type-checking and decodes when appropriate, but this seems like a hackish work-around for a mis-feature of PyParsing, rather than the correct solution.
Is this a bug, am I mis-using PyParsing, or does PyParsing really just not differentiate between these two types?
Thanks in advance,
Jean-Paul