t.w.stream : module documentation

Part of twisted.web2 View Source

The stream module provides a simple abstraction of streaming data. While Twisted already has some provisions for handling this in its Producer/Consumer model, the rather complex interactions between producer and consumer makes it difficult to implement something like the CompoundStream object. Thus, this API.

The IStream interface is very simple. It consists of two methods: read, and close. The read method should either return some data, None if there is no data left to read, or a Deferred. Close frees up any underlying resources and causes read to return None forevermore.

IByteStream adds a bit more to the API: 1) read is required to return objects conforming to the buffer interface. 2) .length, which may either an integer number of bytes remaining, or None if unknown 3) .split(position). Split takes a position, and splits the stream in two pieces, returning the two new streams. Using the original stream after calling split is not allowed.

There are two builtin source stream classes: FileStream and MemoryStream. The first produces data from a file object, the second from a buffer in memory. Any number of these can be combined into one stream with the CompoundStream object. Then, to interface with other parts of Twisted, there are two transcievers: StreamProducer and ProducerStream. The first takes a stream and turns it into an IPushProducer, which will write to a consumer. The second is a consumer which is a stream, so that other producers can write to it.
Interface IStream A stream of arbitrary data.
Interface IByteStream A stream which is of bytes.
Interface ISendfileableStream No interface docstring; 1/1 methods documented
Class SimpleStream Superclass of simple streams with a single buffer and a offset and length into that buffer.
Function mmapwrapper Python's mmap call sucks and ommitted the "offset" argument for no discernable reason. Replace this with a mmap module that has offset.
Class FileStream No class docstring; 1/3 methods documented
Class MemoryStream A stream that reads data from a buffer object.
Class CompoundStream A stream which is composed of many other streams.
Function readStream Pass a stream's data to a callback.
Function readAndDiscard Read all the data from the given stream, and throw it out.
Function readIntoFile Read a stream and write it into a file.
Function connectStream Connect a protocol constructed from a factory to stream.
Function fallbackSplit Undocumented
Class TruncaterStream Undocumented
Class PostTruncaterStream Undocumented
Class ProducerStream Turns producers into a IByteStream. Thus, implements IConsumer and IByteStream.
Class StreamProducer A push producer which gets its data by reading a stream.
Class ProcessStreamer Runs a process hooked up to streams.
Function generatorToStream Converts a generator function into a stream.
Class BufferedStream A stream which buffers its data to provide operations like readline and readExactly.
Function substream Undocumented
Class _StreamReader Process a stream's data using callbacks for data and stream finish.
Class _ProcessStreamerProtocol Undocumented
Class _StreamIterator Undocumented
Class _IteratorStream Undocumented
def mmapwrapper(*args, **kwargs): (source)
Python's mmap call sucks and ommitted the "offset" argument for no discernable reason. Replace this with a mmap module that has offset.
def readStream(stream, gotDataCallback): (source)

Pass a stream's data to a callback.

Returns Deferred which will be triggered on finish. Errors in reading the stream or in processing it will be returned via this Deferred.
def readAndDiscard(stream): (source)

Read all the data from the given stream, and throw it out.

Returns Deferred which will be triggered on finish.
def readIntoFile(stream, outFile): (source)

Read a stream and write it into a file.

Returns Deferred which will be triggered on finish.
def connectStream(inputStream, factory): (source)

Connect a protocol constructed from a factory to stream.

Returns an output stream from the protocol.

The protocol's transport will have a finish() method it should call when done writing.
def fallbackSplit(stream, point): (source)
Undocumented
def generatorToStream(fun): (source)

Converts a generator function into a stream.

The function should take an iterator as its first argument, which will be converted *from* a stream by this wrapper, and yield items which are turned *into* the results from the stream's 'read' call.

One important point: before every call to input.next(), you *MUST* do a "yield input.wait" first. Yielding this magic value takes care of ensuring that the input is not a deferred before you see it.
>>> from twisted.web2 import stream
>>> from string import maketrans
>>> alphabet = 'abcdefghijklmnopqrstuvwxyz'
>>>
>>> def encrypt(input, key):
...     code = alphabet[key:] + alphabet[:key]
...     translator = maketrans(alphabet+alphabet.upper(), code+code.upper())
...     yield input.wait
...     for s in input:
...         yield str(s).translate(translator)
...         yield input.wait
...
>>> encrypt = stream.generatorToStream(encrypt)
>>>
>>> plaintextStream = stream.MemoryStream('SampleSampleSample')
>>> encryptedStream = encrypt(plaintextStream, 13)
>>> encryptedStream.read()
'FnzcyrFnzcyrFnzcyr'
>>>
>>> plaintextStream = stream.MemoryStream('SampleSampleSample')
>>> encryptedStream = encrypt(plaintextStream, 13)
>>> evenMoreEncryptedStream = encrypt(encryptedStream, 13)
>>> evenMoreEncryptedStream.read()
'SampleSampleSample'
def substream(stream, start, end): (source)
Undocumented
API Documentation for Twisted, generated by pydoctor at 2011-10-27 16:02:37.