<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><div style><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Mar 28, 2013 at 6:24 PM, <a href="mailto:succer110@tiscali.it" target="_blank">succer110@tiscali.it</a> <span dir="ltr"><<a href="mailto:succer110@tiscali.it" target="_blank">succer110@tiscali.it</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div class="h5"><div>I have made 2 application: </div>
<div>The client extract data from a sql server (10k lines), and send every line pickled to a "collector" server via socket.</div>
<div>The server uses twisted and receive every line, unpikle it and store the data in another sql server.</div><div><br></div><div>Everytime i start sending data from client to server, in the first 200 line (everytime a different line) **the server** throws an exception:</div>
<div>SOMETIMES it something like:</div><div><br></div><div> Traceback (most recent call last):</div><div> File "collector2.py", line 81, in dataReceived</div><div> self.count,account<span style="white-space:pre-wrap"> </span>= pickle.loads(data)</div>
<div> File "/usr/lib/python2.6/pickle.py", line 1374, in loads</div><div> return Unpickler(file).load()</div><div> File "/usr/lib/python2.6/pickle.py", line 858, in load</div><div> dispatch[key](self)</div>
<div> File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop</div><div> del self.stack[-1]</div><div> IndexError: list assignment index out of range</div><div><br></div></div></div></blockquote>
</div></div></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail_extra">
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div class="h5"><div></div><div>
And my server:</div><div><br></div><div>
def dataReceived(self, data):</div><div> <span style="white-space:pre-wrap"> </span>try:</div><div> <span style="white-space:pre-wrap"> </span>self.count,account<span style="white-space:pre-wrap"> </span>= pickle.loads(data) </div>
<div> <span style="white-space:pre-wrap"> </span>except Exception as e:</div><div> <span style="white-space:pre-wrap"> </span>print "Eccezione:", e</div><div> <span style="white-space:pre-wrap"> </span>print self.count+1</div>
<div> <span style="white-space:pre-wrap"> </span>print data</div><div> <span style="white-space:pre-wrap"> </span>print traceback.print_exc()</div><div><br></div><div><br></div></div></div></blockquote></div></div>
</blockquote><div><br></div><div><br></div><div>On Thu, Mar 28, 2013 at 10:07 PM, David Reid <span dir="ltr"><<a href="mailto:dreid@dreid.org" target="_blank">dreid@dreid.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">dataReceived gets called with any data that is available on the socket. That might not be all data you sent on the other side. To ensure complete "messages" are delivered your application has to specify some framing, such as Netstrings. <div>
<br></div><div>See: <a href="http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.NetstringReceiver.html" target="_blank">http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.NetstringReceiver.html</a> and the original specification of netstrings <a href="http://cr.yp.to/proto/netstrings.txt" target="_blank">http://cr.yp.to/proto/netstrings.txt</a><div>
<br></div><div>That being said, it's a very bad idea to send pickles over the network because unpickling can result in arbitrary code execution.</div><div><br></div><div>Peruse some of the results of <a href="https://www.google.com/search?q=pickle+execute+arbitrary+code" target="_blank">https://www.google.com/search?q=pickle+execute+arbitrary+code</a> for examples of these dangers.</div>
<div><br></div><div>-David</div></div></div><div class="gmail_extra"><br></div></blockquote><div><br></div><div>We also have a FAQ entry about this:</div><div><br></div><div><a href="http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.dataReceivedcalledwithonlypartofthedataIcalledtransport.writewith">http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.dataReceivedcalledwithonlypartofthedataIcalledtransport.writewith</a><br>
</div><div><br></div></div><div><br></div><div><br></div></div>-- <br>Christopher Armstrong<br><a href="http://radix.twistedmatrix.com/">http://radix.twistedmatrix.com/</a><br><a href="http://planet-if.com/">http://planet-if.com/</a><br>
<br>
</div></div>