Tool to update / generate NEWS file

[radix]

It should take trunk log as input.

[exarkun]

It should probably find the trunk log itself, rather than require it as input. This is less work for whoever is invoking it, since part of this is figuring out how much of the log to look at (ie, from the last release to now).

Probably that's what you meant, but I wanted to make it explicit.

[therve]

(In [22028]) Branching to 'news-generator-2884'

[therve]

(In [22029]) This is a thing.

Refs #2884

[therve]

(In [22031]) Another brick *on* the wall.

Refs #2884

[therve]

(In [22032]) Retrieve log message from svn log, first try to build a NEWS file.

Refs #2884

[therve]

(In [22034]) Add the buildNewsFile method, some cleanups.

Refs #2884

[radix]

Thanks a lot for doing this work, therve! I have some comments (this isn't a real review).

• I don't think looking at the component of the ticket in trac is good enough to determine which project it's for. There are commits that go across project boundaries, and really those should be included in the NEWS for all projects affected. This should be easy enough to implement by looking at log -v and checking which paths changes were in.

• Do you have any ideas for how we can automate the retrieval of startRevision? Unfortunately our version control information only includes it in a very indirect way, which is not retrievable by just doing a log of trunk: it's the revision at which the release branch was created from trunk. I guess once we get going it'll be easy enough to find out which version of the software was the previous release, and then log *that* branch to find the copied revision...

• some of those methods could use some more thorough docstring, like fromData :-)

• Why does parse() catch ValueError from parseOneLog? That's not documented as something it can raise :-)

• I realize that this isn't yet the high-level tool, but at least some support must be added to allow a high-level tool to deal with the case where certain parts of a log aren't able to be parsed. That may just be as simple as storing the passed-in log as self.log in the SvnLogCommit, so that a tool can print it out when confronted with something unparseable.

• I don't think including the entire log message in the NEWS file is really that practical. At least, up until now each item in the NEWS file has been rather small, and I was assuming I'd continue to write the messages myself, based on reading the commit message and ticket associated with the fix. But I don't know; maybe with the increased release rate it won't be so overwhelming to include a thorough description of all the changes. I think that's what it comes down to; we want to be able to show a description of things that happened in a release short enough that people can get an idea of what it's about. What do you think? I'll talk to some other people and get their opinions.

Again, thanks for all the work you've done on this. This is going to greatly increase the speed of the release.

[therve]

(In [22044]) Keep track of modified files.

Refs #2884

[therve]

(In [22045]) Actually use the saved files.

Refs #2884

[therve]

Replying to radix:

- I don't think looking at the component of the ticket in trac is good enough to determine which project it's for. There are commits that go across project boundaries, and really those should be included in the NEWS for all projects affected. This should be easy enough to implement by looking at log -v and checking which paths changes were in.

Good idea, I've done it.

Do you have any ideas for how we can automate the retrieval of startRevision? Unfortunately our version control information only includes it in a very indirect way, which is not retrievable by just doing a log of trunk: it's the revision at which the release branch was created from trunk. I guess once we get going it'll be easy enough to find out which version of the software was the previous release, and then log *that* branch to find the copied revision...

Yeah that's not really easy. What I want to propose is doing it manually for this release, and make something in the process of the next release to make that easier (I'm not really sure about that, maybe using a tag in the log message, or storing the revision in a file).

some of those methods could use some more thorough docstring, like fromData :-)

Yep, I'll add some more :).

Why does parse() catch ValueError from parseOneLog? That's not documented as something it can raise :-)

Documented :).

I realize that this isn't yet the high-level tool, but at least some support must be added to allow a high-level tool to deal with the case where certain parts of a log aren't able to be parsed. That may just be as simple as storing the passed-in log as self.log in the SvnLogCommit, so that a tool can print it out when confronted with something unparseable.

I've added the log for now.

I don't think including the entire log message in the NEWS file is really that practical. At least, up until now each item in the NEWS file has been rather small, and I was assuming I'd continue to write the messages myself, based on reading the commit message and ticket associated with the fix. But I don't know; maybe with the increased release rate it won't be so overwhelming to include a thorough description of all the changes. I think that's what it comes down to; we want to be able to show a description of things that happened in a release short enough that people can get an idea of what it's about. What do you think? I'll talk to some other people and get their opinions.

Yes, the buildNewsFile/writeNewsFile was just a way to test the other parts of the process. I'll try to write a more high-level tool, if you have some more infos for this tool please tell me.

[radix]

1. many conflicts with trunk, but ultimately simple ones
2. might as well go ahead and add a script to admin/ that runs the command line

Many issues with reversion support. Each of the points below does actually indicate a different case that needs to be handled. I reran the script several times to avoid the revision that's blowing up to catch other issues.

3. It's barfing on revision 19272. It successfully detected it as a revert, but it thinks it's reverting revision 2. Here's what the log says:

------------------------------------------------------------------------
r19272 | glyph | 2007-01-09 00:03:52 -0500 (Tue, 09 Jan 2007) | 1 line

Revert that last one.  Apparently python 2.5 does not like the strategy employed
 in test_keepOriginalAttributes

This indicates that not only does the tool need better error recovery, it should also probably produce a report of things that it has tried to infer after a successful run; since the reversion parsing support is so flexible, I can see it misparsing something *successfully*, which we ought to be able to veto somehow.

4. Apparently it's detecting 19705 incorretly as a revert. I noticed this because 19706 reverts 19705, and that blows up because when a revision is a revert it's not added to the revisions mapping.

	  File "/home/radix/Projects/Twisted/trunk/twisted/python/_release.py", line 571, in parse
	    del revisions[logCommit.revisionReverted]
	exceptions.KeyError: 19705

19705's log message is:

------------------------------------------------------------------------
r19705 | ralphm | 2007-02-25 08:39:01 -0500 (Sun, 25 Feb 2007) | 2 lines

Rebranch for reverted r19669 re #1652 and #2476.

5. Continuing on, I find that 20183 is actually the first case of actually a messed up reversion for a revision that was not even in trunk. I think ralphm typo'd it for 20182. This needs to be dealt with in a better way.

6. 20348 reverts 20347. 20347 is a reversion. This failed because you only store the revision number if it's a non-revert commit.

7. Running with r20349 as the start revision, it blows up on that revision because the revision that it is reverting is before the start revision that I passed. I guess it is possible for a reversion to happen after a release for a revision that was before it, so this needs to be handled better somehow.

8. It's misparsing 20352, thinking that the revision it's reverting is 2604: Forgot an important commit. Reverting. Reopens #2604.

OK! with revno of 20354, it successfully finishes parsing. That's 9 months of data! congratulations! :-)

On the other hand, it's clear now that supporting reversions is going to be tough. It may be worth considering a simpler alternative involving slightly more human power.

9. Downloading these CSVs from trac takes a REALLY long time. I wonder if there's a more efficient way to get the data out in batch?

Alright, I'm going to let this run and go get some food. Thanks for the work.

[radix]

Hum. Ok, It ran to completion.

2008-02-23 12:47:47-0500 [HTTPPageGetter,client] Starting factory <HTTPClientFactory: http://twistedmatrix.com/trac/ticket/2928?format=csv>
2008-02-23 12:47:47-0500 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://twistedmatrix.com/trac/ticket/2770?format=csv>

2008-02-23 12:47:50-0500 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://twistedmatrix.com/trac/ticket/2928?format=csv>
2008-02-23 12:47:50-0500 [-] Main loop terminated.
radix@fuu ~/Projects/Twisted/trunk% 

radix@fuu ~/Projects/Twisted/trunk% cat NEWS.TEST 

Features
--------

Fixes
-----

Deprecations and Removals
-------------------------

Other
-----
radix@fuu ~/Projects/Twisted/trunk% 

I guess something is wrong :-)

[exarkun, therve]

(In [24934]) Branching to 'news-generator-2884-2'

[exarkun]

I propose an alternate approach.

• Changes to trunk must be accompanied by a NNNN.NEWS file (where NNNN is the ticket number being resolved).
• This file is added to the topfiles directory (or directories) of the subproject being modified.
• The contents of the file is a short description of the change, suitable for inclusion in a news file, or a marker (perhaps just an empty file) to indicate inclusion in the "misc" section (ie, no details provided).
• The release management tool bundles up all these files (in the release branch) into news files, and deletes them.

The advantages:

• This tool is much simpler to write than the svn changelog-based version.
  • No dealing with svn
  • No parsing commit messages
  • Reverts are tracked automatically in the data model
• Provides an easy way to determine if a ticket has been fixed since the last release

Disadvantages:

• If a change is reverted, if there was no branch, the news file kind of falls on the floor and has to be recovered somehow when the change is re-committed.

[exarkun]

(In [27609]) Branching to 'news-generator-2884-3'

[exarkun]

I've implemented what I described in my previous comment in the linked branch. I expect this will also resolve #3199.

[glyph]

There are a few minor cleanup issues, but I think it looks pretty good.

• build-news doesn't appear to have the svn:executable bit, so I can't run it out of the box.
• the API docs have a couple of weaknesses:
  • None of the class variables are documented @cvar or @ivars
  • parameter documentation is missing from all the new private methods and buildAll.

Please address these and merge.

[thijs]

In [27678]: Merge news-generator-2884-3 Author: exarkun Reviewer: glyph Fixes: #2884 Fixes: #3199 Add an administrative tool for generating news files for Twisted and Twisted subproject releases. This tool uses the files now required to be added to various topfiles directories when resolving tickets.