Memory Consumption Issue with Twisted+pyOpenSSL
|Reported by:||wirepair||Owned by:|
Description (last modified by thijs)
While running some code in production using twisted web Agent I noticed memory climbing excessively high (4GB) when making many SSL based requests. I've reduced the issue down to being reproduceable even in the sample twisted tls echo server and a modified tls client (using endpoints). I've tried the following operating environments:
CentOS 6.0 (x86_64) (Linux 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux) Python 2.6.6, Twisted 11.1 (and 12.0) pyopenssl 0.13. Python 2.7.2 (from src), Twisted 12.0 and pyopenssl 0.13.
Amazon Linux (x86_64) 18.104.22.168-95.38.amzn1.x86_64 #1 SMP Thu Aug 25 17:11:23 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Python 2.6.7, Twisted 11.1, pyopenssl 0.13.
I tried a very simple blocking pyopenssl *only* script which didn't appear to consume memory even though the request was basically the same (sending/receiving 1100 bytes). (See attached pyossltest.py.)
It should be noted that Python 2.7 with Twisted 11.0 and pyopenssl 0.13 in win32 (x86 python) would properly release memory and never appeared to consume more than ~40mb using the attached test code. The systems which are showing problems consumed around 300mb during the same time period, and never appeared to release any memory (always went up, never down).
Also if you only establish the connection with the client, then immediately close (call self.transport.loseConnection prior calling self.transport.write()) the memory footprint does not increase.
I also tried epoll and select reactors and didn't notice any change. Let me know if you need any more information, i've brought this issue up in #twisted on IRC and other people appear to be able to reproduce the issue.