Use Sphinx for Twisted Documentation

[khorn]

Twisted should use Sphinx for all long-form documentation (basically everything but API docs).

This ticket is for handling that conversion.

Comments, discussion, and reviews on this ticket should be about overall issues relating to the conversion.

Markup errors in the conversion branch should be handled in separate "markup review" tickets. This is so we can break up the task of manually reviewing all the markup and one person doesn't get stuck doing it all.

Markup review tickets will be listed in the comments of this ticket once there is a branch to review. (after the 10.1 release)

Download

The lore2sphinx tool can be downloaded from  https://bitbucket.org/khorn/lore2sphinx.

[exarkun]

Building the Lore-format documentation is currently part of the  continuous integration employed in Twisted's development process.

At some point we need to set up something that does the equivalent of the [ http://buildbot.twistedmatrix.com/builders/documentation/builds/512/steps/process-docs/logs/stdio Lore process docs step. I'd like to do this even before we've actually switched over, so we can use it to tell us about any problems in branches related to the conversion.

[thijs]

Replying to exarkun:

At some point we need to set up something that does the equivalent of the [ http://buildbot.twistedmatrix.com/builders/documentation/builds/512/steps/process-docs/logs/stdio Lore process docs step. I'd like to do this even before we've actually switched over, so we can use it to tell us about any problems in branches related to the conversion.

See #4225 that seems related.

[exarkun]

One of the steps for lore2sphinx is to run sphinx-quickstart. This prompts for a lot of information, some of which I don't have. I guessed and used defaults for a lot and got far enough so that I could run "make html" which threw out hundreds of warnings and errors, for example:

twisted-sphinx/source/projects/core/index.rst:: WARNING: document isn't included in any toctree
twisted-sphinx/source/projects/words/howto/im.rst:61: (ERROR/3) Unknown interpreted text role "api".
twisted-sphinx/source/projects/web2/examples/index.rst:66: WARNING: download file not readable: projects/web2/examples/auth/credsetup.py

So I think I must have done something wrong. Some more instructions would be helpful.

[khorn]

@exarkun:

The lore2sphinx repo includes an already-quickstarted sphinx project in the profiles/twisted directory. While this will need to be modified a bit when doing the "real" conversion, the intention was to provide a common baseline for people to test it against during development, though I don't think anyone but thijs ever did.

Also, keep in mind lore2sphinx was intended to be a "run it once for real" tool. Even then it's not perfect, and making it so would likely consume the rest of my useful programming career accounting for special cases, etc. When I run it and then use sphinx-build from either Sphinx 0.6.5 or Sphinx 1.0b2, I get just 13 errors (as of last week). My feeling is that those can be fixed manually.

[khorn]

#4553 was a duplicate of this ticket

[exarkun]

Also, keep in mind lore2sphinx was intended to be a "run it once for real" tool.

I know. You've mentioned this too many times for me to ever be able to forget it. :) Despite that, I don't see why we shouldn't be able to run it as many times as we want leading up to the actual conversion. And we need to deal with outstanding branches with changes to the lore source files somehow. It's possible that manually rewriting the docs is the easiest way to go here, but I don't think we should completely throw away the idea of being able to automate at least part of the conversion of such changes.

Even then it's not perfect, and making it so would likely consume the rest of my useful programming career accounting for special cases, etc.

I'm definitely not pushing for this.

When I run it and then use sphinx-build from either Sphinx 0.6.5 or Sphinx 1.0b2, I get just 13 errors (as of last week).

This is more interesting. I used Sphinx 1.0b2. Why did I get hundreds of errors? I suppose it must have had something to do with some of those arguments I gave to sphinx-quickstart. I'll try again with the pre-started project in the lore2sphinx repository.

[khorn]

I think using the pre-quickstarted project will help. What kinds of errors were you seeing?

The most likely possibility I can think of is that the docs are full of links to example files, which the fabfile and/or the l2sbuilder.py script copy as an intermediate step in the overall conversion process. If you skip this step, you'll see a whole lot of "I can't find this file" type warnings from Sphinx.

[exarkun]

the fabfile and/or the l2sbuilder.py script copy as an intermediate step

What step is this? I don't see it documented anywhere.

[khorn]

Well, it's documented in the fabfile and l2s_builder code :)

But I'm afraid that's about it.

[khorn]

okay, folks, the plan has changed slightly...

exarkun has created a buildbot which runs lore2sphinx on the current lore sources and creates Sphinx output.

quote from exarkun's email:

At last I've got a buildbot set up generating the sphinx docs.  The
build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the
containing directory is browseable).

In order to minimize the pain of the final conversion, we want to see if we can get the results of the buildbot to look as good as possible before "pulling the trigger".

So we'll be creating "chunk tickets" for that...I'll post the list here once I've created them.

[khorn]

Here's a list of the chunk tickets:

• projects/conch - #4566
• projects/core/development - #4567
• projects/core/howto (except ‘tutorial/’) - #4568
• projects/core/howto/tutorial - #4569
• projects/historic - #4570
• projects/lore - #4571
• projects/mail - #4572
• projects/names - #4573
• projects/pair - #4574
• projects/vfs - #4575
• projects/web (everything except web-in-60) - #4576
• projects/web/web-in-60 - #4577
• projects/web2 - #4578
• projects/words - #4579

[khorn]

NOTE: it needs to be possible to build the docs without the docs for vfs and web2, since they are not currently (and should not be) published to the website

It should be possible to handle this using one of the "conditional content" mechanisms in Sphinx.

[gtaylor]

Sorry if this isn't the right place to ask, but where can we go to learn more about helping with this?

[khorn]

@gtaylor:

Basically, the plan has remained the same. I think everyone just got busy (I certainly did). We finish reviewing and committing the "chunk" tickets listed in the comments above to make the Lore source convert as close to correctly as possible. Then we convert it to Sphinx using lore2sphiunx and get it into SVN. Then we set up a new set of "chunk" tickets for reviewing and fixing the RestructuredText source.

One possible obstacle is that the lore2sphinx buildbot seems to have broken sometime in the last couple of months. We need to see if we can get that fixed.

As for helping, you can try reviewing and fixing the tickets above, or perhaps post to the Twisted mailing list, which is probably an easier forum for discussions like this.

[thijs]

Another ticket: #4621 - lore2sphinx should make pretty links to included files.

[khorn]

OK, current status (following the Pycon 2011 sprints).

• all the markup tickets for modifying the lore sources to make the generated Sphinx sources look nicer are **done**

• the "apilinks" Sphinx extension has been moved into PyDoctor (as "apilinks_sphinxext.py"). We need to make sure that it will get installed with PyDoctor in a way that Sphinx will be able to find it (it needs to be on the Python path)

• the "traclinks" extension is available in sphinx-contrib

• there will likely be another extension which will be used to make the sphinx literalinclude directives look a bit nicer (have nice links to the included files)

remaining work:

• the twisted release automation needs to be updated to use Sphinx rather than Lore. This could be tricky...

[khorn]

(In [31151]) Branching to 'sphinx-conversion-4500'

[glyph]

exarkun pointed out that there should be no particular pressure to do this in time for the 11.0 release. Whenever it's completely done, even if that's just a day after the release is finished, we can re-generate the documentation from 11.0 and put it on the website immediately. There will be no need to do another release just to make that happen.

In fact, a separate announcement might be even better publicity-wise, so that we can emphasize the newly indexed documentation on its own.

[thijs]

Replying to khorn:

remaining work: * the twisted release automation needs to be updated to use Sphinx rather than Lore. This could be tricky...

So looking at the Lore2Sphinx timeline, shows there's 2 tickets left, this one and #4621. Is there a ticket for twisted release automation update? Anything else? Hopefully this can go into 11.1 :)

[thijs]

Replying to khorn:

* there will likely be another extension which will be used to make the sphinx literalinclude directives look a bit nicer (have nice links to the included files)

Which is reported in #4621.

[khorn]

Please review.

Note that so far this is only some changes to the release mgmt stuff. I'd like to get that reviewed before switching over the actual documentation.

[glyph]

Reviewing.

[glyph]

Hi khorn! Thanks for writing this branch.

1. First and foremost, I think that we should move this branch out to a different ticket. We can merge in some release-automation code to build the sphinx documentation before we actually commit all of the sphinx input, since reviewing the ReST sources may come up with other, unrelated issues. The more stuff we can get merged in small chunks, the better. Alternately we could open a different ticket for the ReST source migration, but this ticket seems to be pretty clearly aimed at that purpose. So you may want to respond to this review on a different ticket.
2. There's tons of trailing whitespace in this branch; it should be cleaned up.
3. SphinxBuilder.__doc__ isn't very clear.
   1. %SPHINXBUILD%? Give me a break. Hopefully nobody will ever actually run this on Windows (the release process definitely isn't supported there) so this syntax is pretty meaningless.
   2. The docstring should really say what the expectations for ALLSPHINXOPTS and BUILDDIR are.
   3. SphinxBuilder.build.__doc__ doesn't explain its parameters terribly well. What is the expected structure of docDir and buildDir? Where's the sphinx project supposed to be?
4. runCommand takes a list of strings, not a string. It happens to work as-is only because Popen (incorrectly) allows you to pass in a string. There are all kinds of potential problems with quoting this input.
5. Don't put # TODO markers in the code. If something needs to be done, file a new ticket. If not, just add an implementation note.
6. The comments about text/latex/pdf builders should similarly also be separate tickets.
7. Docstrings should always be """-style strings, not '''. Many of the tests use the latter style.
8. @ivar declarations ought to go in the class docstring, not method docstrings, so please move the ones from setUp. Thanks for thinking to document those, though :-).
9. For loops in tests are a bad idea. There's always the chance that you'll accidentally loop over nothing or somehow abort the loop early, and get a false positive. Even when the tests do their job correctly and fail on some bad data, a for loop means that when you get a failure you have to do forsensics to figure out which data element it failed on, since the traceback doesn't tell you. So, instead of writing tests like this:

   data = [[a, b], [c, d], [e, f]]
   for thunk in data:
       assertSomething(thunk)
       assertSomethingElse(thunk)
   

   write them like this:

   def doAsserts(thunk):
       assertSomething(thunk)
       assertSomethingElse(thunk)
   doAsserts([a, b])
   doAsserts([c, d])
   doAsserts([e, f])
   

10. SphinxBuilderTests docstring looks pretty incoherent: was this a debug / note to self that never got updated?
11. Some of the names are not coding-standard compliant. They should be confContent, indexContent, and test_failToBuild respectively.
12. A matter of personal taste: triple-quoted strings (aside from docstrings) should be indented so that their contents are further-indented than their enclosing suite. Since you're using textwrap to re-align them anyway, I figure that should be easy.
13. I'd really like it if this would include a command-line tool to update documentation in the right shape for the web site so that the release process doesn't develop another hole, which is currently plugged by an untested crappy thing on a wiki page. At the very least some new "documentation" (ha ha) of this form needs to be written.
14. As a minor detail,  the tests fail just about everywhere ;-) I believe this is because of the aforementioned string/list of strings Popen thing.

Thanks again Kevin for continuing your work on this. I am looking forward to putting this ticket behind us :-).

[khorn]

splitting out the release automation stuff to #5312