[Twisted-Python] Release blocker: Use latest pydoctor release ?

Maarten ter Huurne maarten at treewalker.org
Wed Mar 3 08:18:30 MST 2021


On Sunday, 28 February 2021 23:04:49 CET Craig Rodrigues wrote:

> With respect to API docs, I am not as familiar with the whole process,
> with how they are generated
> and what are doing with readthedocs vs. docs on twistedmatrix.com.
> API docs are generated and don't live in the source tree.
> 
> For example:
> https://twistedmatrix.com/documents/current/api/twisted.html
> 
> Is the long term direction to get rid of that, and point everything at
> readthedocs?

Yes, as far as I know the intention is to reduce the amount of 
infrastructure that has to be maintained by Twisted developers.

Adi is doing the actual work for the migration; I only contribute 
indirectly by reviewing PRs that make pydoctor integrate better with 
Sphinx.

> Since you have done a lot of work in this area, can you shed some
> light on what you think the future direction of all this stuff should
> be with respect to the API docs?

There are efforts to make the output of pydoctor more user friendly. 
This is mainly done by Tristan, but I occasionally work on it as well. 
In the next major release we should have a clearer presentation of 
parameter types, more navigation links and, if it's ready in time, a 
side bar containing a page outline.


The main thing I'm working on is improving accuracy. Currently pydoctor 
can draw conclusions based on incomplete information if import cycles 
are present and with type annotations being added, there are more import 
cycles than before. To solve this, I want to separate the parsing and 
local analysis from the analysis that involves multiple documented 
objects, and run the latter only after all of the former is finished.

On Twisted's side of improving accuracy, we currently have a lot of 
'@type' fields in the docstrings which aren't correct. In particular, 
there is still a lot of fallout from the Python 2 to 3 migration, where 
a documented type of 'str' can either mean 'bytes' or 'str'. I think 
that most '@type' fields should be replaced by type annotations, in 
which case mypy will verify that the documented type matches the type 
that the code is actually using.

Something that might help here is to generate type stubs from the 
information found in docstrings and then automatically apply those stubs 
to Twisted's code. A possible implementation would be for pydoctor to 
export the result of docstring parsing as JSON and then a separate tool 
could generate type stubs from that data.


Twisted's customizations to pydoctor are a bit of a pain, since changes 
in pydoctor break Twisted's API docs quite often. The customizations 
exist in two parts: code and templates.

For the code customizations, these are done using subclassing, so the 
tight coupling makes it hard to change pydoctor's design without 
breaking the customizations. If possible, I'd prefer to remove the need 
for these customizations. If that isn't possible, we'd have to design a 
new plugin interface that is a lot more shielded from pydoctor's 
internals.

For anyone interested, the details are discussed here:
https://github.com/twisted/pydoctor/issues/315

For the template customizations, we're splitting up the templates into 
smaller chunks. This will eliminate or at least greatly reduce the 
amount of copy-pasted template content, which should allow Twisted to 
switch to new major pydoctor reeleases without having to sync the 
template content changes almost every time.

The PR for the template rework is here:
https://github.com/twisted/pydoctor/issues/299

Bye,
		Maarten





More information about the Twisted-Python mailing list