[Twisted-Python] Lore to Sphinx Conversion Progress Report 6

Kevin Horn kevin.horn at gmail.com
Sat Jul 10 16:17:03 EDT 2010

On Sat, Jul 10, 2010 at 2:40 AM, Tim Allen <screwtape at froup.com> wrote:

> On Sat, Jul 10, 2010 at 02:58:40AM -0000, exarkun at twistedmatrix.com wrote:
> > At last I've got a buildbot set up generating the sphinx docs.  The
> > build results can be seen here:
> >
> >   http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/
> >
> > (or with a different revision number for different revisions; the
> > containing directory is browseable).
> >
> > It would be great if everyone could go look around and report any
> > problems you see.
>     - In the Documentation Table Of Contents page, "Historical
>      Documents" is listed between "Twisted Core" and "Twisted Lore"
>      (presumably because of alphabetical order) but this seems
>      a strange place for it. Could it be moved to the bottom, or an
>      appendix, or something?

They're in alphabetical order.  The TOC page is generated dynamically by
looking at the file structure of the docs and that's the order file-globbing
lists them in.  We can fix this manually post-conversion.

>    - The bottom of every page has a "Download in other formats: Plain
>      Text" link that doesn't work; presumably scooped up from the Trac
>      templates. It should be removed (the Sphinx "Show Source" link is
>      in the right-hand sidebar anyway).

Yeah, this is a flaw in the trac theme.  Looks like I forgot to remove it.

>    - When the breadcrumb navigation at the top of the page gets too
>      long[1], it wraps to a second line, which looks kind of odd
>      because the nice gradient background image loops. Changing the CSS
>      background colour to be the same as the bottom of the gradient
>      image would probably help.
>    - Firefox's Error Console reports:
>        - a bogus "%" on line 56 of twistedtrac.css,
>        - bogus // comments on lines 559, 577, 588, 598 of trac.css (CSS
>          only allows /* winged comments */)
>    - Chromium's error console reports:
>        - A 404 error for "/trac/chrome/common/js/jquery.js"
>        - A 404 error for "/trac/chrome/common/js/trac.js"
>        - A 404 error for "/trac/chrome/common/js/search.js"
>        - A 404 error for "/builds/sphinx-html-15615/_static/dots.gif"

Ah! This explains why the search quit working.  Not sure why these files are
missing,  Probably something to do with the custom theme.

>    - The Twisted.Conch tutorial[1] has a number of things surrounded
>      with double-backticks like "``Deferred``" or
>      "``ClientTransport``". I'm not sure if that's deliberate or if
>      it's markup gone horribly wrong.

This is probably due to missing spaces before/after the markup in the rst
source.  One of those things that will need to be fixed manually.  I spent a
lot of time trying to fix this in lore2sphinx and when you fix it in one
place, it breaks somewhere else.  Right now the automated conversion is
about as good as it is likely to get with a sane amount of effort.

>    - On the same page, the text ":api:`
>      twisted.internet.interface.Transport
>      < twisted.internet.interface.Transport>`" appears, which also
>      looks wrong.


>    - The Conch code examples (as linked from the code examples page[2])
>      are sent with a Python mime-type (which Firefox tries to download)
>      while the .tac files are sent as text/html; this may be an
>      artifact of the docs being hosted on the buildbot machine rather
>      than Sphinx, but I think it would be nice if by default they were
>      displayed in the browser, syntax-highlighted, with an option to
>      download the original files (much like Trac does).

This is a web server configuration thing.  The files are actually .py and
.tac files (and probably need to remain so, if we ever want to get automated
example code tests going).

Maybe we can do some web server magic to get them nicely displayed in the
browser, but I see that as a secondary issue for the moment.  Anyone should
feel free to give a shout if they disagree, though.

>    - The Twisted Split FAQ[3] has more visible markup:
>      "twisted.protocols:superscript::ref:`[1]
>      <core-upgrades-2.0-split-protocols> `". The footnote target is
>      kind of messed up, too.

Another spacing issue.

>    - The Twisted Split FAQ[3] has a heading "Why are arr the packages
>      still named twisted.*subproject*?", but it's not obvious whether
>      the author was trying to use italics (in which case it's
>      a lore2sphinx bug) or globbing (in which case it's not).

Pretty sure it was supposed to be italics.  Spacing again.  There are ways
to make this display properly, but it'll need to be done manually.

Also, this is pretty outdated stuff...we might consider just removing some
of this.  Or perhaps moving some of it into a more obvious place.

>    - The Twisted Zope Interfaces FAQ[4] has inline Python source
>      examples that don't appear to be marked up properly (although this
>      is possibly a problem with the original Lore source). The answer
>      to question "How can I update my own code?" is one example.

The code bits are marked up this way because of the attributes used on the
elements in the Lore source.  We can change it easily

>    - The Banana Protocol Specifications[5], under the heading
>      "Examples" mentions "the type bytes are marked in bold" but
>      actually they're surrounded by double-asterisks.

Double asterisks are the markup for bold in rst.  Another spacing and/or
nested inline markup issue.

>    - The Twisted Coding Standard[6], mentions "the complete test
>      suite in trunk at HEAD" - and "trunk at HEAD" is converted into
>      a mailto: link.

Huh.  I guess that's Sphinx trying to be helpful.  Good catch.  That'll
definitely need fixing.

>    - Links followed by punctuation seem to leave a space between the
>      link text and the punctuation. See "test-driven ." and "Test
>      Standard ." in the Twisted Coding Standard[6], but I've observed
>      the same problem on multiple pages.

This is the spacing issue again, but in reverse.  It's relatively easy for
lore2sphinx to detect links and add a space at the end, so that's what it
does.  There's ways to get around this and get rid of the space by escaping
it, but it rapidly get mind-boggling, so this is another thing that will
need manual fixing.

>    - The Twisted Coding Standard[6] mentions "If you modify, or write
>      a new, HOWTO, please read the Lore documentation to learn how to
>      format the docs"; if there's not already a follow-up ticket for
>      the lore-sphinx conversion titled "Remove references to Lore from
>      Twisted's non-Lore-related documentation", there should be, and
>      this should be in it.

There isn't, but there will be.  See the transition plan here:

>    - The Twisted Coding Standard[6], under the heading "Modules" says
>      "Use this template:" and then
>      "../listings/new_module_template.py".. and *then* includes the
>      content of said template inline. Presumably it should either link
>      to it or include it, but not both.

It actually is including an external file there (i.e. the contents are not
in the rst source, but in the
"../listings/new_module_template.py" file.  What looks like a link there is
actually a sort of heading to
let the reader see where it's coming from.  I think this can probably be
made clearer with some theme/css changes.

>    - The HTML Documentation Standard for Twisted[7] has more visible
>      markup: in the list of allowable markup, most tags are rendered
>      properly except for "``<tr>``".

Missing space after preceding comma.  Another manual fix.

>    - It seems that the HTML Documentation Standard for Twisted[7] is
>      actually "how to write Lore documentation", despite the name.
>      Maybe this file should be scheduled for the post-transition purge,
>      or at least moved to the Lore documentation.

It pretty much is "how to write Lore documentation".  I think it should
probably be moved into the Lore docs, and replaced with the
(yet to be written, but again see the transition plan) planned Twisted
Documentation Guide.

>    - Working from Twisted's Subversion Repository[8] has visible markup
>      that looks like an attempt to link "the Subversion homepage" under
>      the heading "Checkout".

Missing preceding space again.

>    - Under the heading "Alternate tree names", the document uses
>      in 'directory other than”Twisted” .' Looks like a problem in the
>      original markup. Also note the space between the LEFT DOUBLE
>      QUOTATION MARK and the punctuation.

The missing preceding space actually is what's causing the quotes to be

>    - The same document has more broken links under the "Combinator"
>      heading.

Missing preceding spaces again.

>    - The same document has broken markup under "Running tests":
>      "``twisted/protocols/imap4.py``" and later
>      "``twisted.mail.test.test_imap``".
>    - The same document has broken markup under "Building docs":
>      "``doc/development/policy/svn-dev.xhtml`` ," (also trailing space
>      before punctuation) and
>      "``doc/development/policy/svn-dev.html`` :" (again).
>    - The same document has a broken link under "Committing and
>      Post-commit Hooks", attempting to link the text
>      "trac-post-commit-hook".
>    - The same document has broken markup under "Emacs":
>      "``emacs/twisted-dev.el``".
>    - The same document has a broken link under "Building Debian
>      packages", attempting to link th text "stdeb".

All bad spacing issues.

> That's probably enough feedback to be getting on with; most of the
> problems appear to be from normalising "\n" in Lore docs to "" instead
> of " ", and also from adding whitespace after things. It is generally
> looking pretty great, though!

Yeah, that's pretty much it.  As I said above though, if you "fix" it one
place, it breaks in another, so I tried to balance things in such a way that
the least broken markup appears in the output.

Almost all of the remaining problems will need to be fixed manually.

> Some extra thoughts:
>    - The ReviewingDocumentation wiki page has a section called "Editing
>      man pages" that describes how to turn the nicely-formatted
>      manpages into Lore input files. Would it be possible to do that as
>      part of the lore2sphinx run, have the manpages included in the
>      Sphinx documentation, and from then on generate the manpages from
>      the .rst files instead of the other way around?

Sphinx does have a man page builder now, but I don't think it existed when I
was writing lore2sphinx, so I haven't really considered this.

So you're suggesting convert the man pages to Lore format -> use lore2sphinx
to convert the resulting Lore docs to rst -> build as part of the Sphinx
process, yes?

I think this is a worthwhile idea, but I'd prefer to leave it until after
the main docs are converted (i.e. under a separate ticket).  lore2sphinx can
be used on just the man files later on if need be, though it would take a
little mucking around.

> [1]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    conch/howto/conch_client.html>
> [2]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    conch/examples/index.html>
> [3]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    core/upgrades/2.0/split.html>
> [4]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    core/upgrades/2.0/components.html>
> [5]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    core/specifications/banana.html>
> [6]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    core/development/policy/coding-standard.html>
> [7]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    core/development/policy/doc-standard.html>
> [8]: <http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/projects/
>    core/development/policy/svn-dev.html>
Thanks for the fantastic (and nicely detailed) feedback, Tim!

Please take a look at the transition plan.  In a few days (maybe sooner), I
should have the base docs in a branch, and the "chunk tickets" referenced in
the transition plan created.  This is pretty much _exactly_ what I'd like to
see in those "chunk tickets".  Hopefully you haven't already burned up your
brain staring at markup issues. :) We could really use this kind of help
throughout the process.

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20100710/19c735a6/attachment-0001.htm 

More information about the Twisted-Python mailing list