Changes between and of Version 2Version 6Ticket #3450

Timestamp:

12/19/2008 11:17:36 AM (4 years ago)

Author:

exarkun

Comment:

Adjusting summary and description to reflect the feature request.

The old description was:

Hello,

sorry for my bad english. I hope someone understand what I mean.

The short description: In twisted.web.client must be a bug. If I have a URL list with many bad URLs I get a lot (50 - 100%) of strange user timeout exceptions. But only if I have to many parallel requests (more than 10).

For example:

I have a list of 20.000 URLs (with many bad URLs in it). I have a script, that has always 10 parallel requests -> 2.65 % of the requests get the timeout exception. I start the same script with the same URLs list and 50 parallel requests -> 5,88 % of the requests get the timeout exception. I start the same script with the same URLs list and 100 parallel requests -> 64,1 % of the requests get the timeout exception. 250 parallel requests -> 99,38 % of the requests get the timeout exception.

But if I use the script with a list of 20.000 good URLs I get this result:

100 parallel requests -> 0,34 % of the requests get the timeout exception. 250 parallel requests -> 0,48 % of the requests get the timeout exception.

Therefore, I think there must be a bug.

Why are there 100 % timeouts with 250 parallel request, but with 10 parallel request only 2.65 % with the bad URL list?

The timeout exception is:

Traceback (most recent call last): Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure.

---------------------

I will attach my test script, a spreadsheet with many statistic datas and other things (so you can better look at the bug) in this bug report:

Content of the zip file:

------------------ The URL lists: ------------------

• the good urls: miscurls.py
• the bad urls: webproxys.py

-------------------------------- The script (WebProxyChecker.py): --------------------------------

Input (over comand line):

• first parameter: quantity of parallel requests
• second parameter: the url list "mode":
  • "miscurls" for the good url list (20.000 urls)
  • "webproxy" for the bad url list (20.000 urls)
• third parameter: quantity of requests
  • (the script selects the urls per random.sample() from the url list)

example: python WebProxyChecker.py 20000 webproxy 50

the script prints statistic data and all exceptions to stdout.

--------------- bash_script.sh --------------- calls the test script with many different parameters. (runs on my machine ca. 30 hours, with 526.100 requests)

usage example: bash bash_script.sh | tee WebProxyCheckStatRaw.nfo

---------- parser.py ----------

parser for the output from the bash script, witch generates a) a shelve db for easyly usage of the datas

(incl. all received exceptions) and

b) a spreadsheet for a better overlook.

---------------- The spreadsheet ----------------

1. column: number of parallel requests (first comand line parameter from WebProxyChecker.py)
2. column: url list (second comand line parameter from WebProxyChecker.py)
3. column: number of requests (third comand line parameter from WebProxyChecker.py)
4. column: number of successful requests
5. column: errors (number of received exceptions)
6. column: user timeouts (number of the strange user timeouts)
7. column: % requests with user timeouts -> column_6 / (column_3 / 100)
8. column: duration of all requests (in seconds)
9. column: duration per requests (in seconds)
10. column: the key in in the shelve db for this record (for example to look at the exceptions)

--------------- The shelve db ---------------

db key = the comand on the comand line (10. column in the spreadsheet)

for example: python WebProxyChecker.py 20000 webproxy 50

    >>> import shelve
    >>> db = shelve.open('result_db.slv')
    >>> record = [r for r in db.values()][0]
    >>> record['parallel_requests'] # 1. spreadsheet column
    '250'
    >>> record['URL_mode'] # 2. spreadsheet column
    'miscurls'
    >>> record['checked_URLs'] # 3. spreadsheet column
    '100'
    >>> record['successful_requests'] # 4. spreadsheet column
    '99'
    >>> record['errors'] # 5. spreadsheet column
    '1'
    >>> record['seconds_altogether'] # 8. spreadsheet column
    '10.0048542023'
    >>> record['seconds_per_url'] # 9. spreadsheet column
    '0.100048542023'
    >>> exceptions = record['exceptions'] # all exceptions
    >>> for exception, data in exceptions.iteritems():
    ...     print exception # the original exception string
    ...     data['cnt'] # occurrence of this exception
    ...     data['urls'] # a set of the urls that caused this exception
    ...     data['vars'] # a dict of lists, with removed data (all strings between '', "" and <>)
    ...     break
    ... 
    Traceback (most recent call last):
    Failure: twisted.web.error.Error: 400 Bad Request
    1
    set(['http://www.cruise.ch'])
    {}
    >>> 

---------- The rest: ----------

• WebProxyCheckStatRaw.nfo: output from the bash script.

Unessential; all data are in the shelve db or in the spreadsheet for simple usage.

• this text

-------------------------------------------------------

My System:

• Ubuntu 8.04
• Python 2.5.2
• Twisted 8.1.0

Do you need additional information?

I am searching this bug since over a week, but I don't know twisted internals and can't find it.

Sorry for my terrible, poorly english.