[Twisted-Python] Lore and generating reStructuredText (Lore2Sphinx)

[Kevin Horn]

On Tue, Mar 26, 2013 at 3:41 PM, Glyph <glyph at twistedmatrix.com> wrote:

> On Mar 26, 2013, at 7:03 AM, Kevin Horn <kevin.horn at gmail.com> wrote:
>
> - figure out a way to get Sphinx to take doctrees as input (I'm guessing
> you'd serialize them in whatever way Sphinx does when it does it's caching,
> stick them in the cache, and then get Sphinx to build from the cache?)
>
>
> This is the only part of the process I believe is actually necessary.  All
> the other stuff you wrote assumes that this can't be made to work :).  But
> as I understand it, this is specifically what JP asked Doug.
>

I don't think this (only needing to figure out the last part) is really the
case:

- figure out the node output of every directive you were trying to
replicate, with every type of (tedious in the best case, possibly very
tricky for some directives, but maybe not too bad)

You need this to know what nodes to create in your tree.  This doesn't seem
too bad, until you realize that a number of the Sphinx-specific directives
you *absolutely must have* (or at least the nodes they create, depend on
the Sphinx build environment.  So you need to either re-create the build
environment, or you need to re-create all of these directives in your own
code.

- figure out a system to "inject" those nodes at the right time (not sure
how difficult this would be)

This one is probably not too bad, since you could probably get away with a
bare minimum of just sticking your (for example) toctree nodes right after
your main heading or something.  And you could probably get away with
something similar for index entries or whatever.

- figure out how Sphinx modifies the doctrees to do all it's linking, index
generation, toctree handling, etc.

Probably not too much needs to be done here directly, as I *think* that
Sphinx does all this after it builds the doctrees, so if you can get the
doctrees into Sphinx you're probably fine.  Don't quote me on that, though.
 So I no longer think this is the hardest part.  But I think you'd still
have to have a decent understanding of how these bits work internally to
generate your nodes correctly.  So a learning curve, though probably no
actual code to write specifically for this.

- figure out a way to get Sphinx to take doctrees as input (I'm guessing
you'd serialize them in whatever way Sphinx does when it does it's caching,
stick them in the cache, and then get Sphinx to build from the cache?)

Then you have to do this bit. The "obvious" way to do this is to create
your doctrees and then pickle them, like Sphinx does when it caches parsed
documents.  Then make Sphinx build its output from these "cached" files
(which I don't think it will currently do, but it can probably be made to
do it).

The whole point is that we want to go straight from Lore->some docutils
> data structure.
>

Why?  What does this buy us?  To me it seems more complicated, requires
more work, depends on *internal* APIs of a separate project (actually 2
separate projects), and doesn't seem to gain very much if anything.  What's
the reasoning here?  Keep in mind that I'm without the benefit of whatever
discussion on this took place at PyCon, so maybe I'm just missing something.

If it's just an incremental transition, then I think we can get that
without resorting to relying on the guts of two fairly complicated systems.

If we have to emit intermediary ReST, it's almost as bad as having to do
> the whole source translation in the first place.
>

I don't see how emitting intermediary ReST, which at least has a spec
(granted the spec is ugly to look at, but it's pretty complete) is any
worse than emitting intermediary doctrees, which could change out from
under us.

Summing up a bit:

Generating ReST is a challenging problem, no doubt.  But it's the _only_
challenging problem if we go the source translation route.

If we go the doctrees route, I don't understand the advantage gained, and
I'm concerned about dealing with the internals of docutils and Sphinx.

(also it's more work, and I'm lazy :P )

--
Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20130326/9872effe/attachment.htm