[Twisted-Python] Lore to Sphinx Conversion Progress Report

Kevin Horn kevin.horn at gmail.com
Thu Dec 3 12:27:12 EST 2009

On Thu, Dec 3, 2009 at 3:00 AM, Glyph Lefkowitz <glyph at twistedmatrix.com>wrote:

> On Dec 3, 2009, at 1:29 AM, Kevin Horn wrote:
> On Wed, Dec 2, 2009 at 8:38 PM, <exarkun at twistedmatrix.com> wrote:
>> As far as the (I'm going to call it) roadmap goes, the thought that's
>> pushed its way to the fore for me is that I'd like to try this with
>> something smaller and simpler than Twisted first.  It would be nice if
>> the Divmod projects would qualify here, but they may not be active
>> enough for any real experience to accumulate.
> I haven't run my lore2sphinx script against the Divmod stuff yet, but I
> could certainly try it.  The intention is to convert the Divmod stuff as
> well as the Twisted stuff eventually though.
> I've been concentrating on the Twisted docs, since they seemed the highest
> priority, but I could detour a bit if you like.  Obviously that would slow
> progress on the Twisted stuff a little.
> While converting pyOpenSSL's documentation would be a worthwhile goal in
> itself, I don't actually like this idea.
> Mainly I disagree with the premise that accumulation of experience is
> necessary for the conversion.  But this is not a strong disagreement, as I
> don't understand the motivation for saying so in the first place :).
> The beauty of this plan, as far as I see it, is that the only person who
> has to get any significant experience with Sphinx in order for the
> conversion to happen is Kevin.   Mostly what the rest of us will do is read
> the documentation and make sure it looks OK.  Obviously we all need to learn
> ReST *after* that conversion to write documentation, but there is so much
> documentation of ReST and sphinx available that I'm not really worried about
> that part.
> More importantly, as JP already noted, the pyOpenSSL documentation is in a
> different format and the conversion would use a different toolchain, so even
> if we do have to get some practice, it's not a particularly helpful place to
> start.  If we *do* need practice for some reason, I think Nevow would be the
> best place to start, but then, I don't see why the activity level of the
> project makes any difference.
> I have a few issues with the roadmap too, though:
> "blah, blah": I'm not sure what that's supposed to mean.

Doh!  Forgot to go back and change that...pretend it sayssomething like:
"(list of branches here)"

I'll fix it shortly.

"branches containing lore docs changes should be separated into two
> groups..." - no criteria are specified for deciding which go into which
> group.  Ideally we could just get all of those changes merged; if you are
> making progress on the conversion I'm sure we could organize a sprint to
> evaluate those changes and either abandon them or get them into review.

Well, I'm not entirely certain what criteria should be used until we at
least know what branches exist with lore docs in them.

If it's only say 2 or 3 branches that are relatively easy to do, we just do
all of them.  If it's a whole bunch, and some of them are those "everlasting
branches" which have been around for years with no sign of being merged any
time soon, well...that's group B. :)

Identifying tickets which propose lore functionality is pretty easy; just
> look at <
> http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&component=lore&order=priority>.
>  Any tickets which have not been classified properly and therefore don't
> show up in that list can be closed later, as we discover them.  I strongly
> feel that we do *not* need to conduct an exhaustive review of the entire
> ticket tracker and get everything perfectly in order in order to do this
> conversion, as long as it's clear to everyone what is supposed to happen to
> lore-related tickets in the future.

Yeah, I don't expect this to be too difficult.  Basically thought I'd run a
couple of searches (including the very one you mentioned above) and go from
there.  I'd like to get most of them though.  One of the problems I think
Twisted has is that there are just too dogone many open tickets.  It gets
tricky to figure out what really needs to be done unless you've been doing
it for years (as the core devs have) because of all the stuff getting in the
way.  Which is why I've been trying to find outstanding tickets that are
almost finished and "shepherd" them to completion.  But that's a discussion
for another day/thread.

> I see the biggest risk at the "docs freeze" step, that the doc conversion
> guy (or team, as the case may be by that point) will start work, then get
> distracted and walk away for 6 months, leaving a long period of time where
> nobody is supposed to write or edit documentation.
> The whole point of our branch-based review process is to avoid this sort of
> situation.  We can't always avoid it (for example, the immense outstanding
> Conch branches that made everyone afraid to edit those warning-filled tests
> for years) but I think it's best to follow the same plan as for any branch,
> and have no formal "freeze" duration, just a point where the conversion
> branch gets merged to trunk.  It's OK if a few stale doc branches get left
> out in the cold during the conversion; if they're still stale once the
> conversion is ready, they must not be terribly actively worked on anyway.
To be honest, I'm not even sure such a freeze would be necessary.  I just
thought I would throw it out there since the actual changeover would involve
a rather large "structural" change to that part of the Twisted source tree.
And I didn't expect it to last more than a few days.  Maybe even hours.
Perhaps I'm just used to having this kind of step.  If the consensus is that
it doesn't need to happen, I'm fine with that.

(FYI, I'm currently working in an environment where version control is
ACTIVELY DISCOURAGED, so it makes me a little paranoid that some kind of
disaster could happen.)

> Even in the worst case, where a branch is left stale well after the docs
> have been deleted from trunk, presumably the author of the change could run
> lore2sphinx against the conversion, copy the sphinxified doc to their trunk
> working copy, and make a new branch.  Assuming that the output of the
> conversion tool is deterministic, the diff should be small and readable.

Hmmm.   This kind of procedure could sort of work.  Pondering....

The only potential problem I see with this idea is that I think we'll need
to do some manual cleanups on the converted docs.  But I guess the author
could do as you suggested, then do their own (presumably small) manual
cleanup of their working copy after the conversion of their modified
branch.  Should be workable, if not _quite_ as simple as you've laid it out
(though almost as simple).

> I'm not saying that we shouldn't identify those changes... It would be good
> to identify the branches with outstanding doc changes so that we would at
> least *know* how many changes will be broken, and perhaps motivate their
> authors to fix them ahead of time.

I think we're on the same page here.

> I'm mainly interested in the "phase 0" outlined in the roadmap; I think
> that the stuff for phase 1 and 2 sounds good, but I don't think we need any
> special planning for it, since it fits into our normal development workflow
> pretty neatly.  People will be filing bugs for documentation typos pretty
> much forever ;-).  I'd like it if you could break down the "phase 0" a bit
> more clearly with regard to what happens when, since the review *before* the
> Big Switch gets thrown to put these changes in trunk is the most important
> part to get timely feedback from the community.

Looking at this after having been away from it for a day or so, I can see
that it's a bit chaotic :)  I guess I had been looking at it too long.

I'll try to update this soonish.  Anything specific that is unclear?

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20091203/26bf7e90/attachment.htm 

More information about the Twisted-Python mailing list