Stats Collection

Scrapy provides a convenient facility for collecting stats in the form of key/values, where values are often counters. The facility is called the Stats Collector, and can be accessed through the stats attribute of the Crawler API, as illustrated by the examples in the Common Stats Collector uses section below.

The Stats Collector API is always available, so you can always use it (to increment or set new stat keys), regardless of whether the stats collection is enabled or not. If it’s disabled, the API will still work but it won’t collect anything. This is aimed at simplifying the stats collector usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you’re using the Stats Collector from.

Another feature of the Stats Collector is that it’s very efficient (when enabled) and extremely efficient (almost unnoticeable) when disabled.

See Built-in stats reference below for the stats that Scrapy sets.

Common Stats Collector uses

Access the stats collector through the stats attribute. Here is an example of an extension that access stats:

class ExtensionThatAccessStats:
    def __init__(self, stats):
        self.stats = stats

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.stats)

Set stat value:

stats.set_value("hostname", socket.gethostname())

Increment stat value:

stats.inc_value("custom_count")

Set stat value only if greater than previous:

stats.max_value("max_items_scraped", value)

Set stat value only if lower than previous:

stats.min_value("min_free_memory_percent", value)

Get stat value:

>>> stats.get_value("custom_count")
1

Get all stats:

>>> stats.get_stats()
{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}

Available Stats Collectors

Besides the basic StatsCollector there are other Stats Collectors available in Scrapy which extend the basic Stats Collector. You can select which Stats Collector to use through the STATS_CLASS setting. The default Stats Collector used is the MemoryStatsCollector.

MemoryStatsCollector

class scrapy.statscollectors.MemoryStatsCollector(crawler: Crawler)[source]

A simple stats collector that keeps the stats of the last scraping run (for each spider) in memory, after they’re closed. The stats can be accessed through the spider_stats attribute, which is a dict keyed by spider name.

This is the default stats collector used in Scrapy.

spider_stats: dict[str, StatsT]: A dict of dicts (keyed by spider name) containing the stats of the last scraping run for each spider.

DummyStatsCollector

class scrapy.statscollectors.DummyStatsCollector(crawler: Crawler)[source]: A stats collector which does nothing but is very efficient (because it does nothing). This stats collector can be set via the STATS_CLASS setting, to disable stats collection in order to improve performance. However, the performance penalty of stats collection is usually marginal compared to other Scrapy workload like parsing pages.

Built-in stats reference

Scrapy sets the following stats. Components other than those built into Scrapy may set additional stats; see their documentation.

Stat keys that contain a {placeholder} below stand for a family of stats, one per actual value of the placeholder.

Note

Most stats are set by a specific component, and are only present if that component is enabled and its code path is reached. A stat that is missing from get_stats() output is equivalent to a counter of 0.

depth/request_ignored_count

Number of requests dropped for exceeding DEPTH_LIMIT.

Set by DepthMiddleware.

downloader/exception_count

Number of exceptions raised while downloading requests.

Set by DownloaderStats.

downloader/exception_type_count/{exception_type}

Number of exceptions raised while downloading requests, per exception type, where {exception_type} is the import path of the exception class, e.g. twisted.internet.error.DNSLookupError.

Set by DownloaderStats.

downloader/request_bytes

Total size, in bytes, of the requests sent, counting the request line, the headers and the body. As with downloader/request_count, requests served from the cache are also counted.

It is an approximation, reconstructed from each Request object instead of measured on the wire, so it does not account for the actual bytes that the download handler sends, e.g. transport-level overhead.

Set by DownloaderStats.

downloader/request_count

Number of requests sent.

Requests that HttpCacheMiddleware serves from the cache are also counted, even though they are never sent, because it handles requests after DownloaderStats.

Set by DownloaderStats.

downloader/request_method_count/{method}

Number of requests sent, per HTTP method, e.g. GET or POST. As with downloader/request_count, requests served from the cache are also counted.

Set by DownloaderStats.

downloader/response_bytes

Total size, in bytes, of the responses received, counting the status line, the headers and the body. It covers the same responses as downloader/response_count.

The body is counted as received, i.e. still compressed for responses that used Content-Encoding, because DownloaderStats handles responses before HttpCompressionMiddleware decompresses them. See httpcompression/response_bytes for decompressed sizes.

Set by DownloaderStats.

downloader/response_count

Number of responses received.

It counts responses that HttpCacheMiddleware serves from the cache, even though they do not come from the network, and responses that a downloader middleware consumes before they reach your spider, e.g. redirect responses that RedirectMiddleware turns into new requests. Compare with response_received_count.

Set by DownloaderStats.

downloader/response_status_count/{status_code}

Number of responses received, per HTTP status code, e.g. 200 or 404. It covers the same responses as downloader/response_count.

Set by DownloaderStats.

dupefilter/filtered

Number of requests dropped as duplicates.

Set by RFPDupeFilter.

elapsed_time_seconds

Time, as a float, in seconds, between the spider_opened and the spider_closed signals.

Set by CoreStats.

feedexport/failed_count/{storage}: Number of feeds that could not be stored, per storage backend, where {storage} is the class name of the storage backend, e.g. FileFeedStorage.

feedexport/success_count/{storage}: Number of feeds stored successfully, per storage backend, where {storage} is the class name of the storage backend, e.g. FileFeedStorage.

file_count: Number of files handled by the media pipelines.

file_status_count/{status}

Number of files handled by the media pipelines, per status, where {status} is one of:

downloaded: the file was downloaded.
cached: the file came from the HttpCacheMiddleware cache.
uptodate: the file was already in the storage backend and had not expired, so it was not downloaded again.

finish_reason

String indicating why the crawl finished. It matches the reason argument of the spider_closed signal.

Scrapy uses the following reasons:

cancelled: the spider was closed without a more specific reason, e.g. because CloseSpider was raised without one.
closespider_errorcount: see CLOSESPIDER_ERRORCOUNT.
closespider_itemcount: see CLOSESPIDER_ITEMCOUNT.
closespider_pagecount: see CLOSESPIDER_PAGECOUNT.
closespider_pagecount_no_item: see CLOSESPIDER_PAGECOUNT_NO_ITEM.
closespider_timeout: see CLOSESPIDER_TIMEOUT.
closespider_timeout_no_item: see CLOSESPIDER_TIMEOUT_NO_ITEM.
finished: the spider became idle with no pending requests, i.e. it finished normally.
memusage_exceeded: see MEMUSAGE_LIMIT_MB.
shutdown: the crawl was interrupted, e.g. by a system signal such as SIGINT (Ctrl-C).
start_error: start() raised an exception, so some start requests may never have been sent, see Handling start errors.

Third-party components and your own code may use any other reason, e.g. by raising CloseSpider with it.

Set by CoreStats.

finish_time

Timezone-aware datetime object, in UTC, indicating when the spider_closed signal was sent.

Set by CoreStats.

httpcache/errorrecovery

Number of times that a stale cached response was used because downloading a fresh response raised an exception.

Set by HttpCacheMiddleware.

httpcache/firsthand

Number of responses that were downloaded without a matching cache entry to validate against, i.e. responses for requests counted in httpcache/miss.

It is lower than httpcache/miss when some of those requests yield no response, either because they are dropped (see httpcache/ignore) or because their download fails.

Set by HttpCacheMiddleware.

httpcache/hit

Number of requests served from the cache.

Set by HttpCacheMiddleware.

httpcache/ignore

Number of requests dropped because they were not in the cache and HTTPCACHE_IGNORE_MISSING is True.

Set by HttpCacheMiddleware.

httpcache/invalidate

Number of times that a cached response failed validation and was replaced with a freshly downloaded response.

Set by HttpCacheMiddleware.

httpcache/miss

Number of requests for which no cache entry could be read, either because there was none or because reading it failed, in which case the request is also counted in httpcache/retrieve_error. Those requests are downloaded (see httpcache/firsthand), or dropped if HTTPCACHE_IGNORE_MISSING is True (see httpcache/ignore).

Requests with a stale cache entry are not counted here; see httpcache/revalidate and httpcache/invalidate.

Set by HttpCacheMiddleware.

httpcache/retrieve_error

Number of cache entries that could not be read, and hence were treated as cache misses. Those requests are also counted in httpcache/miss.

Set by HttpCacheMiddleware.

httpcache/revalidate

Number of times that a cached response was successfully validated against the target server, and hence used instead of the fresh response.

Set by HttpCacheMiddleware.

httpcache/store

Number of responses stored in the cache.

Set by HttpCacheMiddleware.

httpcache/uncacheable

Number of responses not stored in the cache because the HTTPCACHE_POLICY did not allow it.

Every response considered for caching is counted either here or in httpcache/store, so httpcache/store + httpcache/uncacheable equals httpcache/firsthand + httpcache/invalidate.

Set by HttpCacheMiddleware.

httpcompression/response_bytes

Total size, in bytes, of decompressed response bodies, counting only the body and only responses that were actually decompressed. Compare with downloader/response_bytes.

Set by HttpCompressionMiddleware.

httpcompression/response_count

Number of decompressed responses.

Set by HttpCompressionMiddleware.

httperror/response_ignored_count

Number of responses dropped because of their HTTP status code.

Set by HttpErrorMiddleware.

httperror/response_ignored_status_count/{status_code}

Number of responses dropped because of their HTTP status code, per HTTP status code, e.g. 404.

Set by HttpErrorMiddleware.

item_dropped_count

Number of items dropped by an item pipeline, i.e. number of times that the item_dropped signal was sent.

Set by CoreStats.

item_dropped_reasons_count/{exception}

Number of items dropped, per exception, where {exception} is the class name of the exception that caused the item to be dropped.

Only DropItem and its subclasses drop items, and each one is counted under its own class name, e.g. item_dropped_reasons_count/DropItem for DropItem itself and item_dropped_reasons_count/MyDropItem for a MyDropItem subclass of it. Any other exception raised by an item pipeline triggers the item_error signal instead of item_dropped, and is not counted here or in item_dropped_count.

Set by CoreStats.

item_scraped_count

Number of items that passed all item pipelines, i.e. number of times that the item_scraped signal was sent.

Set by CoreStats.

items_per_minute

Average number of items scraped per minute during the crawl.

It is None if the crawl took less than a minute.

Set by LogStats.

log_count/{level}

Number of log messages, per logging level name, e.g. INFO or WARNING.

Only messages that the LOG_LEVEL setting allows are counted.

Set by LogCount.

memdebug/gc_garbage_count

Number of objects in gc.garbage when the spider is closed.

Set by MemoryDebugger, which requires MEMDEBUG_ENABLED to be True.

memdebug/live_refs/{cls}

Number of live objects of class {cls} when the spider is closed, as reported by trackref, e.g. memdebug/live_refs/HtmlResponse.

Only set for classes with at least 1 live object.

Set by MemoryDebugger, which requires MEMDEBUG_ENABLED to be True.

memusage/limit_reached

1 if memory usage exceeded MEMUSAGE_LIMIT_MB, which also stops the crawl.

Set by MemoryUsage.

memusage/max

Maximum peak memory usage, in bytes, observed during the crawl.

Set by MemoryUsage.

memusage/startup

Peak memory usage, in bytes, when the engine started.

Set by MemoryUsage.

memusage/warning_reached

1 if memory usage exceeded MEMUSAGE_WARNING_MB.

Set by MemoryUsage.

offsite/domains

Number of distinct domains for which at least 1 request was dropped for being offsite.

Set by OffsiteMiddleware.

offsite/filtered

Number of requests dropped for being offsite.

Set by OffsiteMiddleware.

request_depth_count/{depth}

Number of requests scheduled at depth {depth}, e.g. request_depth_count/2.

Set by DepthMiddleware, which requires DEPTH_STATS_VERBOSE to be True for this stat.

request_depth_max

Maximum depth reached.

Set by DepthMiddleware.

response_received_count

Number of responses received, i.e. number of times that the response_received signal was sent.

Unlike downloader/response_count, it does not count responses that a downloader middleware consumes before they reach the engine, e.g. redirect responses that RedirectMiddleware turns into new requests. Both count responses that HttpCacheMiddleware serves from the cache.

Set by CoreStats.

responses_per_minute

Average number of responses received per minute during the crawl.

It is None if the crawl took less than a minute.

Set by LogStats.

retry/count

Number of requests retried.

Set by get_retry_request(), which RetryMiddleware uses.

retry/max_reached

Number of requests that were not retried because they had already been retried RETRY_TIMES times.

Set by get_retry_request(), which RetryMiddleware uses.

retry/reason_count/{reason}

Number of requests retried, per reason, e.g. retry/reason_count/twisted.internet.error.TimeoutError or retry/reason_count/504 Gateway Time-out.

Set by get_retry_request(), which RetryMiddleware uses.

Note

Code calling get_retry_request() may pass a custom stats_base_key, in which case retry is replaced with that key in the 3 stats above.

robotstxt/exception_count/{exception_type}

Number of exceptions raised while downloading robots.txt files, per exception type, where {exception_type} is the string representation of the exception class, e.g. <class 'twisted.internet.error.DNSLookupError'>.

Set by RobotsTxtMiddleware.

robotstxt/forbidden

Number of requests dropped for being disallowed by robots.txt.

Set by RobotsTxtMiddleware.

robotstxt/request_count

Number of robots.txt files requested, i.e. 1 per network location for which at least 1 request was sent.

Set by RobotsTxtMiddleware.

robotstxt/response_count

Number of robots.txt responses received.

Set by RobotsTxtMiddleware.

robotstxt/response_status_count/{status_code}

Number of robots.txt responses received, per HTTP status code, e.g. 404.

Set by RobotsTxtMiddleware.

scheduler/dequeued: Number of requests read from the scheduler.

scheduler/dequeued/disk: Number of requests read from the disk queue of the scheduler.

scheduler/dequeued/memory: Number of requests read from the memory queue of the scheduler.

scheduler/enqueued: Number of requests stored into the scheduler.

scheduler/enqueued/disk: Number of requests stored into the disk queue of the scheduler.

scheduler/enqueued/memory: Number of requests stored into the memory queue of the scheduler.

scheduler/unserializable: Number of requests that could not be stored into the disk queue of the scheduler because they could not be serialized, and hence were stored into the memory queue instead.

spider_exceptions/count

Number of unhandled exceptions raised by spider callbacks or by start().

Set by the engine and the scraper.

spider_exceptions/{exception}

Same as spider_exceptions/count, per exception, where {exception} is the class name of the exception, e.g. spider_exceptions/ValueError.

Set by the engine and the scraper.

start_time

Timezone-aware datetime object, in UTC, indicating when the spider_opened signal was sent.

Set by CoreStats.

urllength/request_ignored_count

Number of requests dropped for having a URL longer than URLLENGTH_LIMIT.

Set by UrlLengthMiddleware.