[Twisted-Python] Lore and generating reStructuredText (Lore2Sphinx)

Kevin Horn kevin.horn at gmail.com
Mon Mar 25 12:16:08 EDT 2013

On Sat, Mar 23, 2013 at 9:57 PM, Glyph <glyph at twistedmatrix.com> wrote:

> On Mar 21, 2013, at 7:17 AM, Kevin Horn <kevin.horn at gmail.com> wrote:
> Notice that the node trees look exactly the same.  Now this is not quite
> true, as there's probably some attributes on the actual Python nodes that
> might be used to distinguish them when writing output which aren't
> displayed here...they certainly get rendered into HTML differently.  But
> the point is that the directive itself is GONE and you have no real way of
> recreating it from the node tree.
> The directive isn't "gone"; it turns into the attributes on the Python
> nodes that you're talking about.  Presumably that's what's used to render
> it into HTML.  I believe it was Doug Hellmann who indicated to Jean-Paul
> that this was possible.
While this is true for some built-in docutils directives, there is no
guarantee that this will be the case.  A directive basically says "call a
Python callable according to a certain interface, and put the returned
nodes here."  If the directive in question uses a callable that returns
nodes with attributes set in a certain way, then you have some breadcrumbs
to figure out how those nodes were created, but there's nothing that says
that the nodes will definitely be set up that way.

For example, you could have a directive that has a ReST list as content,
and changes the items in the list into some kind of link or something.
 maybe it looks something like this (not a real/valid nodetree...):


How can you tell that this was created by a directive?  You can't, because
it could just as easily have been a list full of links to begin with.

This is why rstgen has it's own node definitions, as it is focused on what
source constructs should be generated, rather than what the docutils output
should look like.

Of course it's possible that the docutils nodes that we would actually need
from the Twisted docs are all introspectible *enough* that you could maybe
just build a docutils doctree and make good enough guesses to create output
which included directives.  It might even be easy to make those guesses.
 But you'd still be guessing, and would fail in the general case.  Also,
you'd still need to write the code to render those nodes into valid ReST,
which is really the hard part of the process.  Also, you'd need to "parse"
everything inside the "directive node" (or whichever node you've decided
represents the directive) in order to turn it into directive arguments,
options and contents.

Reading exarkun's expansion of the notes Itamar posted [1]_ it looks like
another idea proposed was to generate Sphinx (or maybe Sphinx-looking)
output directly from Lore, which could maybe work, but I think would also
be a lot of work, for a lot less benefit.

However, if what you really want is to have a Lore plugin that generates
RestructuredText, then why not have a lore plugin that generates a rstgen
tree, which rstgen will already know how to render into ReST?  Other than
the obvious objection that rstgen isn't done yet, this seems the best
solution to me.  Of course I may be biased. :)

Even if you didn't want to use rstgen itself, though, I still think you're
better off creating some tree-like structure that is *not* a docutils
document tree, and then have that structure render itself into ReST.

BTW, Doug Hellman almost certainly knows more about the internals of
docutils than I do, so maybe he's right and there is a way to (relatively)
easily generate ReST from a docutils tree including the directives.  But I
don't think it is.

> Perhaps you mean "there's no public API for constructing the node tree
> representation of an arbitrary directive"?
This is certainly true, but I think it doesn't go far enough in describing
the issue

.. [1] https://twistedmatrix.com/trac/wiki/Fellowship2013/Priorities

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20130325/95d0bc2b/attachment.htm 

More information about the Twisted-Python mailing list