[Twisted-Python] is twisted compatible with pickle?

[Christopher Armstrong]

>
> On Thu, Mar 28, 2013 at 6:24 PM, succer110 at tiscali.it <
> succer110 at tiscali.it> wrote:
>
>> I have made 2 application:
>> The client extract data from a sql server (10k lines), and send every
>> line pickled to a "collector" server via socket.
>> The server uses twisted and receive every line, unpikle it and store the
>> data in another sql server.
>>
>> Everytime i start sending data from client to server, in the first 200
>> line (everytime a different line) **the server** throws an exception:
>> SOMETIMES it something like:
>>
>>     Traceback (most recent call last):
>>       File "collector2.py", line 81, in dataReceived
>>         self.count,account = pickle.loads(data)
>>       File "/usr/lib/python2.6/pickle.py", line 1374, in loads
>>         return Unpickler(file).load()
>>       File "/usr/lib/python2.6/pickle.py", line 858, in load
>>         dispatch[key](self)
>>       File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop
>>         del self.stack[-1]
>>     IndexError: list assignment index out of range
>>
>>

> And my server:
>>
>>     def dataReceived(self, data):
>>     try:
>>     self.count,account = pickle.loads(data)
>>     except Exception as e:
>>     print "Eccezione:", e
>>     print self.count+1
>>     print  data
>>     print traceback.print_exc()
>>
>>
>>

On Thu, Mar 28, 2013 at 10:07 PM, David Reid <dreid at dreid.org> wrote:

> dataReceived gets called with any data that is available on the socket.
>  That might not be all data you sent on the other side.  To ensure complete
> "messages" are delivered your application has to specify some framing, such
> as Netstrings.
>
> See:
> http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.NetstringReceiver.html and
> the original specification of netstrings
> http://cr.yp.to/proto/netstrings.txt
>
> That being said, it's a very bad idea to send pickles over the network
> because unpickling can result in arbitrary code execution.
>
> Peruse some of the results of
> https://www.google.com/search?q=pickle+execute+arbitrary+code for
> examples of these dangers.
>
> -David
>
>
We also have a FAQ entry about this:

http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.dataReceivedcalledwithonlypartofthedataIcalledtransport.writewith



-- 
Christopher Armstrong
http://radix.twistedmatrix.com/
http://planet-if.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20130328/a7ba24c5/attachment.html>