[Twisted-Python] Testing with trial, adbapi questions

[Christian Simms]

I think your biggest hurdle is that you need to get used to using Deferred's
in adbapi and any functions/methods that call adbapi indirectly.  Unless you
use a fast embedded database like Sqlite, you have to use Deferred's because
there's network traffic under the covers to the database server. I'll try to
include a little advice below for each item:


On 6/25/07, Brendon Colby <brendoncolby at gmail.com> wrote:
>
> Greetings,
>
> I've begun moving an application I wrote from asynchat to twisted.
> I've got basic functionality, I've converted my DB stuff to adbapi and
> have begun writing unit tests. I have a few questions that I haven't
> been able to get answered through the API docs, twisted.words protocol
> code, mailing list posts etc. I'd be very grateful for some
> assistance!
>
> 1. Right now I'm defining my dbPool in my factory init, and running a
> method to load some data from the DB. I've copied the IRC
> BasicServerFunctionalityTestCase and modified it for my use:
>
> class BasicServerFunctionalityTestCase(unittest.TestCase):
>     def setUp(self):
>         self.f = StringIOWithoutClosing()
>         self.t = FileWrapperThatWorks(self.f) # I need this to return
> an IPv4Address!
>         self.p = sserverd.SServerProtocol()
>         self.p.factory = sserverd.SServerFactory(self.p)
>         self.p.makeConnection(self.t)
>         self.p.factory.dbPool.start()
>
>     def tearDown(self):
>         self.p.factory.dbPool.close()
>
>     def check(self, s):
>         self.assertEquals(self.f.getvalue(), s)
>
>     def testSendDeveloperID(self):
>         self.p.lineReceived('{%s:"PG"}\0' % sserverson.base10toN(104))
>         self.check("test\0")
>
> I've gotten past the database threads hanging open by running start()
> and close() manually. Now, my database callbacks that load necessary
> data appear to be getting called AFTER my test runs. I know they're
> getting called, but when the above test runs, the required data
> structure is empty. What am I missing here?


I think you have a race condition because (I assume)
self.p.lineReceivedindirectly did a database insert, and then the
self.check did a database select.  If that's true, you should make your test
use Deferred's like this:

  d = defer.succeed(None)
  d.addCallback(lambda out: self.p.lineReceived('{%s:"PG"}\0' %
sserverson.base10toN(104))
  d.addCallback(lambda out: self.check("test\0"))
  return d

You could also write this code using the new defer.inlineCallbacks (needs
python 2.5) or the older defer.deferredGenerator, but you might as well get
used to using callbacks/errbacks and Deferred's.


2. I'm having a bit of trouble wrapping my head around adbapi. Prior
> to it, one thing I would do is gather data and return a row, as in,
> the method "developerExists()" would check the DB and return row[0][0]
> or something (a bool). Now it seems as though I have to preload this
> data and check against that. Can I replicate this behavior with
> adbapi? Is there a better way than having to preload ALL my data up
> front? How can I get any data from the DB like this:
>
> data = getSomethingFromDB()
>
> I'm suspicious that I need to adopt a new paradigm here, but am having
> difficulty seeing past the old way of doing things!


You're right that you need a new paradigm, like this:

  def cb(data):
       # use the data here, in the callback function
  return getSomethingFromDB().addCallback(cb)

This assumes getSomethingFromDB() returns a Deferred which fires with the
result you wanted.

3. I have some methods (as I pointed out in #1) that load data. With
> adbapi, it appears that I have to do this:
>
>   def _loadDevelopers(self,results):
>     for result in results:
>       developerID = result[2]
>       ...
>       self.addDeveloper
> (Developer(authHost,authUrl,developerID,self,rc4key,
>         postHost,postUrl))
>
>   def loadDevelopers(self):
>     self.dbPool.runQuery("select * from developers where
> active=1").addCallback(
>     self._loadDevelopers).addErrback(printError)
>     # Before, I would just load my data here!
>
> Is this correct? i.e. a DB method that gets called, each having a
> corresponding callback method that does the real work? (This seems
> sort of a pain to me...) I feel as though I'm close to "getting it"
> (twisted in general, adbapi) but feel that I have yet to connect some
> key pieces of information.



This is basically correct, though loadDevelopers should probably return the
result of the runQuery() call.


4. I was digging through the adbapi code, and it appears that the only
> type of result set I can get is a tuple (because only cursors are
> used...). Is there a way I can get a dictionary returned, keys are
> column names? i.e. a MySqlDB.fetch_row(how=1). I'm just not seeing how
> I can do this with adbapi or, rather, how I can pass through to the
> MySqlDB module to get this type of result set.



If you can't find a MySQL-specific way to get a dictionary, you can always
write a generic python function which does so, and then always call that in
your code.  adbapi doesn't do this for you.

5. I've always run my application using DJB's daemontools
> (http://cr.yp.to/daemontools.html). Is there an incredibly compelling
> reason to switch to using twistd? It seems to have some pretty neat
> features, but I'm just not 100% sure I want to give up daemontools
> yet.


You really need to run your twisted app with twistd.  That's the abstraction
for launching twisted programs -- i.e., you define a variable named
"application" in your .tac file and then use twistd to start your app.  Of
course, there's nothing stopping you from using daemontools on top of it.

Thanks!
>
> Brendon Colby
>

Cheers,
Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070625/63d0a358/attachment.html>