twistd autoreload feature during development

[miracle2k]

I wonder if it would be feasible/possible to support a kind of "autoreload" in twistd, similar to how the development servers of various web frameworks do this (say, Django's runserver).

[exarkun]

I took a look at Django's auto-reloader. One interesting thing I noticed about it is that it has no Django dependency. I think it should work equally well for any Python program - including twistd. Unfortunately things are never really that easy. It wants to run things in a thread for some reason, which upsets reactor.run(). It also doesn't work as a nice simple command line wrapper, you have to write something like this:

$ cat autoreload-twistd.py
#!/usr/bin/python
import autoreload
import twisted.scripts.twistd as t
import sys
autoreload.main(t.run)

Still, this is pretty close, I think. It would be interesting to come up with a completely agnostic reloader that could be used by Django and Twisted (and anyone else).

I feel like this should be a project developed outside Twisted to ensure that other projects are comfortable adopting it.

[miracle2k]

Thanks, I didn't realize it was that straight forward.

I played around with it a bit now, and I get a "ValueError: signal only works in main thread" error; not sure if those were the thread-related reactor issues you've been referring to.

When the reactor is run in the main thread, you have the problem that a sys.exit() call is caught by it. There doesn't seem to be way to stop the reactor and cause twistd to exit with a specific code, which is how autoreload currently does things. Not sure if there is a good way to provide that option, or how useful it would be.

One option might be the script restarting itself once it detects a code change, i.e. first calling os.spawnve(), then reactor.stop(), but that seems hacky, and you need to rely in sys.argv not having changed, as well as the shutdown happening quickly enough so you don't have the new process requiring some still locked resource.

FWIW, what I am doing now is using a modified autoreload.py which runs the code_changed checker in a thread, and calls os._exit, rather than sys.exit (which won't work in a thread, of course). That means there's no clean shutdown, which I'm not really happy about, but it works well enough for me.

[exarkun]

There's a couple older ticket, #761 and #2182, for being able to specify the exit status of twistd somehow. If those were resolved, it sounds like you might be in pretty good shape.