[Twisted-Python] Lore to Sphinx Conversion Progress Report 4

[Kevin Horn]

On Wed, Jan 20, 2010 at 9:57 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>wrote:

> On Jan 19, 2010, at 4:33 PM, Kevin Horn wrote:
>
> This time I think I'm gonna skip saying how I haven't gotten as much done
> as I would like...oh darn.
>
> Anyways, time for another gripping installment...
>
> Progress:
>   - tables are now handled (mostly) properly, thanks to Zeth at
> http://commandline.org.uk/
>   - blockquote tags handled
>   - much improved whitespace/indentation handling
>   - some nicer styling thanks to Michael Thompson
>   - I've managed to convert the docs for the 3 Divmod projects with Lore
>     docs, though I've yet to put them up anywhere.
>
>
> Yay!
>
>   - due to ReST's insistence on "inline markup" being surrounded by
>     whitespace or certain special characters, there are a lot of places
> where
>     such inline markup gets jacked up, by not including whitespace in front
> of
>     it.  If I put whitespace in front of everything though, my indentation
>     handling gets jacked up and about 400+ Sphinx build warning result.
>     Not sure if I should spend the time to make whitespace handling really
>     smart or if these should just be fixed manually post-conversion.
>
>
> I don't really understand this problem.  What do you mean about making
> whitespace handling really smart?  Isn't this the sort of detail that
> docutils is supposed to handle for you?
>
>
OK, lemme esplain...no, would take too long, lemme sum up.

In some of the Lore docs, you have stuff like:

    some<em>stuff</em>

which naively translated to ReST looks like:

    some*stuff*

but since ReST wants whitespace and/or special characters surrounding
"inline markup",  docutils/Sphinx doesn't recognize it properly as markup
and just sends it unmodified to the HTML (or whatever) output..

The obvious solution is to just surround all inline markup with spaces,
since we're mainly targeting HTML output at the moment and a few extra
spaces shouldn't hurt.  Which would look like this:

    some *stuff*

But this turns out to cause other problems, specifically in the case where
inline markup occurs on the beginning of a line, and the extra space jacks
up the indentation (which ReST considers significant).

So the whitespace handling I was referring to was the output of whitespace
from lore2sphinx.


>   - Themeing/styling: still mostly a TODO, though new styling looks a lot
>     better than the default to my eyes.  I'm starting to think that
>     eventually we might want to have 2 themes/styles...one to match the
>     trac-based website, and one for bundled docs (docs tarballs, CHM files,
> etc.)
>
>
> That would certainly be nice, but is in no way required for the initial
> migration.  Still, we should have a workable theme in order before we pull
> the trigger :).
>

Right, as I said "eventually".


>
>   - some of the Lore source files have nested "inline markup", which ReST
>     disallows.
>
>
> Ugh.  So ReST *can't do this?*  That's pretty lame.
>

It's a little bit lame, but I've found that it doesn't occur all that often
in practice.  Every markup language has it's limitations, and this is one I
can live with.


>       - just handle the outside level of nesting (what I'm doing now) and
>         fix any problems manually post-conversion.
>
>
> I'm assuming there are very few instances of this, so that sounds fine.
>

That's pretty much my plan.


>
>   - xhtml entities are not currently resolved...mostly because it makes
>     the build take a LOOOONG time.  They can be though.  This shouldn't
>     be a problem.
>
>
> Are you resolving them by downloading all the DTDs or something?
>

The lxml parser can download DTDs in order to resolve external entities, but
the way things are currently set up, it would end up doing this once for
each Lore source file, which ends up making my build process take much
longer.  Because I didn't want to deal with this during development, and
since there are only like 4 external entities in all of the Lore docs, I
turned this off and told lxml to ignore the errors. It may be that there is
a better way to do this (maybe cache the DTDs somehow), but I haven't really
bothered with it yet.  It's a simple matter to turn it back on when we're
ready, even if there isn't an easy/convenient way to avoid the repeated
downloads of the DTDs.


>
>   - Foolscap 0.5 was released today, which made me wonder what they use for
>
>     docs...and it's Lore.  I brought this up on IRC, and it was suggested
>     by many that Lore should stick around even after the conversion
> according
>     to the standard Twisted compatibility policy, to give anyone who still
>     uses it time to migrate.  This sounds like a fine idea to me.
>     Any thoughts?
>
>
> Since nobody really uses lore's API, the same compatibility policy doesn't
> really apply.  In lore's case, I would say that the policy should be that we
> include it with X more releases just for packaging convenience, but stop
> doing maintenance immediately.
>
>
>
Fine with me.

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100121/12501b1e/attachment-0001.html>