settings组件分析

• COOKIES_ENABLED

  进入 scrapy 包 -> downloadermiddlewares 子包 -> cookies模块中

  class CookiesMiddleware:
      """This middleware enables working with sites that need cookies"""
      # 使用该中间件可以处理需要Cookie的网站
  
      def __init__(self, debug=False):
          self.jars = defaultdict(CookieJar)
          self.debug = debug
  
      @classmethod
      def from_crawler(cls, crawler):
          # 这是一个组件，作用于所有的scrapy Request
          # 通过这个组件，提取前一个Request中的cookie，并加入下一个Request cookie中去
          if not crawler.settings.getbool('COOKIES_ENABLED'):
              raise NotConfigured
          return cls(crawler.settings.getbool('COOKIES_DEBUG'))
  
      def process_request(self, request, spider):
          if request.meta.get('dont_merge_cookies', False):
              return

---

---

• USER_AGENT

  设置全局USER_AGENT，与DOWNLOADERMIDDLEWARES 配合

  USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

• DOWNLOADER_MIDDLEWARES

  赋予每个Scrapy Request USER-AGENT

  DOWNLOADER_MIDDLEWARES = {
  #    'ArticleSpider.middlewares.ArticlespiderDownloaderMiddleware': 543,
      'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 2
  }