Opened 3 years ago

Closed 3 years ago

#7871 defect closed duplicate (duplicate)

Lack of useful information when callback isn't callable

Reported by: jessamynsmith Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Branch:
Author:

Description

I was trying to scrape a website with scrapy, and encountered the following issue:

Code highlighting:

2015-04-25 19:09:51-0400 [wow] ERROR: Spider error processing <GET https://www2.woolworthsonline.com.au/Shop/Browse/confectionery?_mode=ajax&_ajaxsource=navigation-panel&_referrer=%2F&_bannerViews=627,6064&_=1429927496128>
        Traceback (most recent call last):
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/core/scraper.py", line 111, in _scrape_next
            self._scrape(response, request, spider).chainDeferred(deferred)
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/core/scraper.py", line 118, in _scrape
            dfd = self._scrape2(response, request, spider) # returns spiders processed output
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/core/scraper.py", line 128, in _scrape2
            request_result, request, spider)
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/core/spidermw.py", line 74, in scrape_response
            dfd = mustbe_deferred(process_spider_input, response)
        --- <exception caught here> ---
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/utils/defer.py", line 39, in mustbe_deferred
            result = f(*args, **kw)
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/core/spidermw.py", line 50, in process_spider_input
            value = scrape_func(response, request, spider)
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/scrapy/core/scraper.py", line 138, in call_spider
            dfd.addCallbacks(request.callback or spider.parse, request.errback)
          File "/Users/jessamyn/.virtualenvs/grant/lib/python2.7/site-packages/twisted/internet/defer.py", line 289, in addCallbacks
            assert callable(callback)
        exceptions.AssertionError: 

I eventually tracked this down to a typo in the callable name I was passing in. This would have been much simpler to debug if the AssertionError had contained a useful message. I tracked the code down to defer.py, in the addCallbacks method. I suggest you modify the assertions to contain useful information, perhaps similar to the following:

Code highlighting:

  def addCallbacks(self, callback, errback=None,
                   callbackArgs=None, callbackKeywords=None,
                   errbackArgs=None, errbackKeywords=None):
      """
      Add a pair of callbacks (success and error) to this L{Deferred}.

      These will be executed when the 'master' callback is run.

      @return: C{self}.
      @rtype: a L{Deferred}
      """
      assert callable(callback), "callback '%s' was not callable" % callback
      assert errback == None or callable(errback), "errback '%s' was not callable" % errback
      cbs = ((callback, callbackArgs, callbackKeywords),
             (errback or (passthru), errbackArgs, errbackKeywords))
      self.callbacks.append(cbs)

      if self.called:
          self._runCallbacks()
      return self

Change History (2)

comment:1 Changed 3 years ago by jessamynsmith

Sorry, there was no feedback when I clicked the submit button, and I wasn't sure if I'd actually clicked it, so I clicked again and created a duplicate ticket. The other is #7870

comment:2 Changed 3 years ago by Glyph

Resolution: duplicate
Status: newclosed

No worries, sometimes the site is a little slow. Closing this one as a duplicate.

Note: See TracTickets for help on using tickets.