Opened 15 years ago

Last modified 6 years ago

#2490 enhancement new

xpath.queryForString fails on unicode XML

Reported by: akrherz Owned by:
Priority: low Milestone:
Component: words Keywords: unicode xpath
Cc: Ralph Meijer Branch:
Author:

Description (last modified by therve)

I'm getting this traceback:

File "./bot-twisted.py", line 133, in processMessage
    bstring = xpath.queryForString('/message/body', elem)
  File "/usr/lib/python2.3/site-packages/twisted/words/xish/xpath.py", line 275, in queryForString
    return internQuery(xpathstr).queryForString(elem)
  File "/usr/lib/python2.3/site-packages/twisted/words/xish/xpath.py", line 242, in queryForString
    self.baseLocation.queryForString(elem, result)
  File "/usr/lib/python2.3/site-packages/twisted/words/xish/xpath.py", line 115, in queryForString
    self.childLocation.queryForString(c, resultbuf)
  File "/usr/lib/python2.3/site-packages/twisted/words/xish/xpath.py", line 117, in queryForString
    resultbuf.write(str(elem))
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 177: ordinal not in range(128)

I'm using Twisted 2.5.0 (TwistedWords-0.5.0). thanks!

Attachments (2)

test_xpath_unicode.py (649 bytes) - added by Yury Yurevich 15 years ago.
unit-test, tests unicode in content and path
xpath.diff (1.6 KB) - added by Yury Yurevich 15 years ago.
quick&dirty patch for bug #2490

Download all attachments as: .zip

Change History (14)

comment:1 Changed 15 years ago by Yury Yurevich

Keywords: unicode xpath added
Priority: normalhigh

I'm getting this bug too.

Changed 15 years ago by Yury Yurevich

Attachment: test_xpath_unicode.py added

unit-test, tests unicode in content and path

Changed 15 years ago by Yury Yurevich

Attachment: xpath.diff added

quick&dirty patch for bug #2490

comment:2 Changed 15 years ago by Ralph Meijer

Cc: Ralph Meijer added

comment:3 Changed 15 years ago by akrherz

thanks j2a, this patch seems to be working for me :)

comment:4 Changed 14 years ago by akrherz

Any updates on fixing this bug for the next twisted release? thanks!

comment:5 Changed 14 years ago by therve

Description: modified (diff)

comment:6 Changed 14 years ago by therve

xpathparser.py must not be modified, xpathparser.g should be instead. Also, the provided patch only fixes one of the tests for me.

I have no idea if the approach is correct or not.

comment:7 Changed 14 years ago by Ralph Meijer

Milestone: Twisted-8.2

comment:8 Changed 13 years ago by Ralph Meijer

Milestone: Twisted-8.2
Priority: highlow
Type: defectenhancement

As therve said, this would require changing the xpathparser.g file from which xpathparser.py is generated. However, YAPPS 2 does not support unicode in its scanner, so we would need to find a way around that. A possibility might be patching the scanner class later on in the file.

Since PyXML also uses YAPPS for parsing XPath queries, we might learn from that. They couldn't use the generated Scanner as-is, either, by the way.

comment:9 Changed 13 years ago by powdahound

I noticed that another (ghetto) workaround for this is to call xpath.queryForStringList('/message/body', stanza)[0] since that flow never casts the data using str().

comment:10 Changed 11 years ago by <automation>

Owner: Ralph Meijer deleted

comment:11 Changed 6 years ago by Ralph Meijer <ralphm@…>

In 5980a7b:

Error: Processor CommitTicketReference failed
 does not appear to be a Git repository. See the log for more information.

comment:12 Changed 6 years ago by Ralph Meijer <ralphm@…>

In 3c7ef07:

Error: Processor CommitTicketReference failed
 does not appear to be a Git repository. See the log for more information.
Note: See TracTickets for help on using tickets.