Opened 11 years ago

Last modified 10 years ago

#2789 defect new

Windows Process Pipes are polled with 100% cpu

Reported by: Peaker Owned by: Peaker
Priority: lowest Milestone:
Component: core Keywords: windows, iocpreactor
Cc: Branch:
Author:

Description

The Windows process implementation _dumbwin32proc.py uses _pollingfile.py as a read/write transport for its pipes to the subprocess.

_pollingfile is called every rediculously-small amount of time (which probably amounts to an actual 0 due to timer granularity) to attempt writing to the subprocess. If the subprocess is not fast enough to consume this, Twisted takes 100% cpu and tries to write 0 bytes to the subprocess a whole lot of times.

I believe there are several problems:

  1. The obvious lack of real asynchronous pipe support - hopefully someone will add this to the iocp reactor.
  2. The MIN_TIMEOUT constant is too small in _pollingfile, 1e-5 is probably small enough.
  3. It seems to be using MIN_TIMEOUT even though 0 bytes are written each time. It would probably make more sense to wait longer if few bytes are written, especially if 0 are written, and not MIN_TIMEOUT.

Change History (4)

comment:1 Changed 11 years ago by Glyph

Owner: changed from Glyph to Peaker
Priority: normallowest

This is an inherent problem in Windows, and the correct answer is to have a working, fully functional IOCP reactor. I am 90% sure that you can't do any better with select().

If you'd like to submit some tweaks to the constants involved, please feel free, but it would be nice if we had some description of a purpose for the change involving descriptions of latency and so on. Personally I could care less :).

comment:2 Changed 11 years ago by jknight

glyph: Just how much less could you care? :)

I can't say I've fully absorbed _pollingfile.py, but I was under the impression that it was supposed to back off and starting waiting at MAX_TIMEOUT (0.1s) if data wasn't being sent. Each time, it looks like it doubles the delay, so after 27 retries, it's waiting 0.1s between tries.

If that isn't happening, it's probably a trivial bug.

comment:3 in reply to:  2 Changed 11 years ago by Glyph

Replying to jknight:

glyph: Just how much less could you care? :)

I'm not sure which units my reply should be in.

I can't say I've fully absorbed _pollingfile.py, but I was under the impression that it was supposed to back off and starting waiting at MAX_TIMEOUT (0.1s) if data wasn't being sent. Each time, it looks like it doubles the delay, so after 27 retries, it's waiting 0.1s between tries.

If that isn't happening, it's probably a trivial bug.

You're correct. I believe I have observed this happening in practice. I understood the report to mean that the pessimal behavior could be triggered by a pessimal reader/writer, but a closer reading suggest that this always, or almost always doesn't happen as expected. I'd definitely appreciate it if that were fixed.

comment:4 Changed 10 years ago by PenguinOfDoom

Keywords: windows iocpreactor added
Note: See TracTickets for help on using tickets.