[Twisted-Python] The Anything Box

[William la Forge Jr]

Woops! I forgot to mention the URL:

http://compstrm.sourceforge.net

Also, here's a snippet from that page describing the project
a little better:

The Anything Box

This is a new kind of database. You can put anything into it, and make any
kind of query to access its contents. It works with structures and complexes
of structures using very scalable technology. And you can view current
state, past state, and changes over time. It is an extensible system that
begins with a rich set of metadata for describing data and operations, with
operations and data properties grouped into capabilities, which in turn are
used to construct data types.

The database is implemented over BSDDB, a transactional store that is also
used by MySQL, using 14 b-tree files to support a wide range of queries. The
basic storage element is (time, fileid, propertyid, oldValue, newValue), but
with additional support for document storage. The property/value pairs
assigned to a file for a given time are then computed based on a
reverse-chronological extraction process.

This Anything Box is fully integrated with Twisted, a fast asynchronous
server that supports both a web server and a very wide range of additional
protocols. The Anything Box itself is implemented using coroutines, making
the code more readable/maintainable than most asynchronous styles. It is
this coroutine technology, implemented as a computational stream, that the
CompStrm project name came from.

User Process Management

All user processes are coroutines, implemented for fast response rather than
throughput. Process management is handled through flow control, with
processes automatically terminated when no more output is desired. Any
process can interact with its user; loosely coupled (background) processes
will be implemented to pend when user input is requested.

Persistent Garbage Collection

Without some form of garbage collection, applications become much more
complex--especially when dealing with complex structures. Unfortunately,
garbage collection normally applies only to in-memory (non-persistent)
structures. The Anything Box implements persistent garbage collection by
using two different kinds of organization. First, data is organized into
directories based on data type, each directory having an associated
descriptor defining the type and its capabilities. Some data types are then
identified as well-known (protected) and are not subject to garbage
collection.

A second organization is by groups. Data elements with the member capability
are always tagged with the id of the id of the lead data element for their
group. References to a data element (from another element) are identified as
either weak or strong (non-weak). And when a strong reference to a data
element from another in the same group is destroyed, then the previously
referenced data element is a potential candidate for automatic garbage
collection--as all members of a group must be strongly referenced by another
element in the same group. (This of course places a design constraint on
applications, that weak references be used to avoid any strong cyclic
references.)

The TKI Shell

While the base-level operations are implemented in Python coroutines, the
Anything Box includes the TKI shell. TKI commands are operations defined by
the Anything Box, which in turn may be implemented as either Python
coroutines or TKI scripts. The shell itself is quite simple, supporting
argument passing to scripts, the assignment of property values to script
variables, and environment variables for navigating space and time and for
generating unique names. The richness of the TKI shell comes from the
numerous operations defined by the Anything Box, which includes support for
set values, lists and dictionaries, and for invoking (mapping) an operation
over the contents of a set, list or dictionary.