[Twisted-Python] Release blocker: Use latest pydoctor release ?
Maarten ter Huurne
maarten at treewalker.org
Wed Mar 3 08:18:30 MST 2021
On Sunday, 28 February 2021 23:04:49 CET Craig Rodrigues wrote:
> With respect to API docs, I am not as familiar with the whole process,
> with how they are generated
> and what are doing with readthedocs vs. docs on twistedmatrix.com.
> API docs are generated and don't live in the source tree.
>
> For example:
> https://twistedmatrix.com/documents/current/api/twisted.html
>
> Is the long term direction to get rid of that, and point everything at
> readthedocs?
Yes, as far as I know the intention is to reduce the amount of
infrastructure that has to be maintained by Twisted developers.
Adi is doing the actual work for the migration; I only contribute
indirectly by reviewing PRs that make pydoctor integrate better with
Sphinx.
> Since you have done a lot of work in this area, can you shed some
> light on what you think the future direction of all this stuff should
> be with respect to the API docs?
There are efforts to make the output of pydoctor more user friendly.
This is mainly done by Tristan, but I occasionally work on it as well.
In the next major release we should have a clearer presentation of
parameter types, more navigation links and, if it's ready in time, a
side bar containing a page outline.
The main thing I'm working on is improving accuracy. Currently pydoctor
can draw conclusions based on incomplete information if import cycles
are present and with type annotations being added, there are more import
cycles than before. To solve this, I want to separate the parsing and
local analysis from the analysis that involves multiple documented
objects, and run the latter only after all of the former is finished.
On Twisted's side of improving accuracy, we currently have a lot of
'@type' fields in the docstrings which aren't correct. In particular,
there is still a lot of fallout from the Python 2 to 3 migration, where
a documented type of 'str' can either mean 'bytes' or 'str'. I think
that most '@type' fields should be replaced by type annotations, in
which case mypy will verify that the documented type matches the type
that the code is actually using.
Something that might help here is to generate type stubs from the
information found in docstrings and then automatically apply those stubs
to Twisted's code. A possible implementation would be for pydoctor to
export the result of docstring parsing as JSON and then a separate tool
could generate type stubs from that data.
Twisted's customizations to pydoctor are a bit of a pain, since changes
in pydoctor break Twisted's API docs quite often. The customizations
exist in two parts: code and templates.
For the code customizations, these are done using subclassing, so the
tight coupling makes it hard to change pydoctor's design without
breaking the customizations. If possible, I'd prefer to remove the need
for these customizations. If that isn't possible, we'd have to design a
new plugin interface that is a lot more shielded from pydoctor's
internals.
For anyone interested, the details are discussed here:
https://github.com/twisted/pydoctor/issues/315
For the template customizations, we're splitting up the templates into
smaller chunks. This will eliminate or at least greatly reduce the
amount of copy-pasted template content, which should allow Twisted to
switch to new major pydoctor reeleases without having to sync the
template content changes almost every time.
The PR for the template rework is here:
https://github.com/twisted/pydoctor/issues/299
Bye,
Maarten
More information about the Twisted-Python
mailing list