[Twisted-Python] Best place to put application code

[Jean-Paul Calderone]

On Fri, 2 Jan 2009 10:14:40 +0000, John Aherne <johnaherne at rocs.co.uk> wrote:
>One thing that has been puzzling me is where is the best place to put
>application code.

It depends. :)

>
>The case I am using is straightforward TCP server with client connections
>making simple requests and waiting for responses retrieved from database
>tables to be sent back.
>
>Reading the docs and looking at various examples provided in books and
>documentation has left me a bit confused.
>
>Is there a best practice for where application code should go - either the
>protcocol class or the factory class or somewhere else. Does it actually
>matter.
>
>Is there any downside to putting application code in the protocol or
>factory. What pitfalls are there for either approach.

Best practice for a Protocol class is to include code which is necessary
to interpret bytes which are received and turn them into a structured form
which is easier to deal with; code which starts from some structured form
and emits bytes to be sent should also be part of a Protocol class.

It is common practice to have a class which includes just these things and
then a subclass which adds application-specific logic based on top of this
functionality.  It is also common practice to connect a protocol which has
only these things, no application-specific code, and then have application
code elsewhere (in a free function, a method of a factory, another class's
method, user input, etc) make calls onto it.  Which of these approaches is
most well suited to a particular application depends.  For example, if the application code creates multiple connections with shared state adding the application logic to a Protocol subclass isn't a good approach.

>I see examples where application code appears in both classes, but the
>examples are very small so may not be indicative of what should be done.

Generally they're so small that there's no advantage to any approach over
any other, yes.

>In the docs I see reference to most of the code will be written in the
>protocol class, but that seems to be referring to actually writing protocols
>not application code. It also says that when the protocol needs to call
>application code to make it a method call -  not to mix protocol and
>application code. This could just mean creating some methods in the protocol
>class to handle the task.

Consider all of the code you write to be part of a library you're developing.
If you implement a protocol, then you've just written part of a library which
provides a slightly higher-level API for interacting with the network in some
way.  With that in hand, you can move on to some other part of your library
which uses that higher-level API to accomplish something even higher-level,
perhaps presenting yet another API to some other part of your application
which is higher-level still.  The motivation to not mix protocol and
application code is just the motivation to have clear boundaries in your
library to make as much of it reusable as possible.  If you have a protocol
implementation mixed together with application A, when you come along to
write application B which needs to use the same protocol, you'll have to
re-implement the protocol, or refactor your original implementation to move
the application A code elsewhere (of course, there's nothing wrong with
having to refactor your code - it's a common part of programming, and since
it's very difficult to predict the future, it's often best *not* to try to
anticipate your future requirements when writing code - just write what works
and is easily testable, and when your future requirements come along, deal
with them then; as you do this more and more, you'll probably get a sense of
how to structure your code to minimize the effort required for refactoring,
but aside from experience with this process, I don't know of any way to learn
this skill).

>However, if the application code needs to run for 10-12 seconds looking up
>database tables and accumulating results and waiting on  deferreds, should
>all this code reside in the protocol class or the factory.

I wouldn't take the duration of the task into consideration when trying to
decide where to put it.  I'd consider reusability, testability, simplicity,
and correctness.

>
>If I keep it in the protocol, then I already have my client connection to
>write back to. So that seems to be the place to keep the code.
>
>If I put the code in the factory, then I need to pass the client connection
>so it can write back to the client. Or is there another way of doing this I
>have missed.

This isn't much different from the trade-off you have to consider when you
decide to implement anything as two classes instead of one.  Since you can
no longer just use `self´ everywhere, you'll have to figure out how to get
a reference to the other instance that you need sometimes.  This shouldn't
be difficult though - just invoke a method on one class with an instance of
the other.

>.
>The factory seems to be the place where other classes can be passed in and
>the protocol can call them via self.factory. That seems to imply that
>application code should be put into the factory, but I can't see any way of
>passing back information from deferred results to the call from protocol.
>It's ok if it was just a simple method call that returns a result, but if
>the code has to run a series of deferreds then it will be the called method
>that will have the result and it will need a means of writing this back to
>the client. I don't think I can signal the protocol to say I now have the
>result.Of course I could easily be mistaken. So please correct.

This is just what Deferreds are for.  For example, let's consider a protocol
and factory for a game server lobby.  The protocol has a message type which
lets the server inform the client of the list of games which are currently
available to be joined.  The Protocol subclass sends this information to
the client when the connection is established, but it uses the factory
to actually find the list of games that are available.  Again, this
division is probably desirable for a number of reasons (it simplifies
unit testing, for example).

class GameFactory(ServerFactory):
    def availableGames(self):
        """
        Return a Deferred which will fire with a list of the games
        currently available to be joined.
        """
        return self.connectionPool.runQuery(
            "SELECT game_id FROM available_games")


class GameProtocol(Protocol):
    def connectionMade(self):
        """
        Send the available-games message so the client can pick one to
        join.
        """
        d = self.factory.availableGames()
        def cbGotGames(games):
            self.transport.write(" ".join(games))
        d.addCallback(cbGotGames)
        d.addErrback(log.err)

>
>Of course if I passs the client connection to the factory, then it can use
>this to write back. But that means passing around the client connection.
>Should I avoid doing that or is that not a problem.

It's an option, but I hope the above example will show how you can use
Deferreds to avoid needing to do this.

>
>I hope I have explained myself clearly. I'm just looking for some guidance
>and pointers to what is best to do or what is best to avoid.
>

Hope this helps,

Jean-Paul