[Twisted-Python] How to get protocol function do not block server?

[Manlio Perillo]

Norman Tindall ha scritto:
> Hello,
>    I am a newbie in twisted, sorry if this was asked thousands times
>    before.
> 
> I have wrote a simple protocol (over TCP sockets) with crc32.
> 
> Suppose one client made a handshake.. and then sends to server 10Mb of
> data... at the end of that 10Mb there is an crc32..
> 

You can require a max size for the packets.

A more general solution is to do the job in chunks:

>From Python documentation:
    print binascii.crc32("hello world")
    # Or, in two pieces:
    crc = binascii.crc32("hello")
    crc = binascii.crc32(" world", crc)
    print crc


Here is an example of computing the edonkey2k hash:

def waitFor(seconds):
    """Wait for the specified amount of seconds.
    To be used in a deferred generator.
    """

    from twisted.internet import reactor

    d = defer.Deferred()
    reactor.callLater(seconds, lambda: d.callback(None))

    return defer.waitForDeferred(d)


@defer.deferredGenerator
def getFileHash(path):
    """XXX TODO use a smaller file buffer size (8096 bytes?)
                use a thread?

    XXX TODO add support for md5, sha1 and crc32.
    """

    from Crypto.Hash import MD4


    # this is used as file buffer size, too
    CHUNK_SIZE = 9728000

    # XXX delay to give the reactor a chance
    DELAY = 0

    hashList = []

    fp = file(path, 'rb')
    while True:
        hash = MD4.new()
        data = fp.read(CHUNK_SIZE)
        yield waitFor(DELAY)

        if len(data) == 0:
            hash.update('')
            hashList.append(hash.digest())

        hash.update(data)
        yield waitFor(DELAY)

        hashList.append(hash.digest())

        if len(data) < CHUNK_SIZE:
            break

    # compute the hash of the hash list
    hash = MD4.new()
    hash.update(''.join(hashList))
    yield waitFor(DELAY)

    yield hash.hexdigest()

You can do the same, using cStringIO.

> crc32 is a pretty time consuming task on a slow machine..
> while server calculates crc32 of 10Mb sent by client one,
> other clients can not communicate with server
> (i tryed by myself to log in while transfer.. and i was wating about a
> 10 seconds)
> And also i have a UnCryptAFrame time consuming function.
> 

Check if the encryption can be done in chunks.




Regards  Manlio Perillo