标签：user web exce spl 首页 查找 head 下载 新闻内容
位置：middlewares.py文件中
一、爬虫中间件
二、下载中间件
1、位置：引擎和下载器之间
2、作用：批量拦截到整个工程中所有的请求和响应
3、拦截请求：
UA伪装：写在process_request方法中
# UA池 user_agent_list = [ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 " "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1", "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 " "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 " "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 " "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 " "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 " "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5", "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 " "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 " "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 " "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 " "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24" ]
# 拦截请求 def process_request(self, request, spider): # UA伪装 request.headers[‘User-Agent‘] = random.choice(self.user_agent_list) return None
代理IP ：写在process_exception方法中，最后需要return request
# 可被选用的代理IP PROXY_http = [ ‘126.120.102.124:80‘, ‘101.208.131.169:56055‘, ] PROXY_https = [ ‘20.83.49.90:900‘, ‘91.182.12.212:3508‘, ]
# 拦截发生异常的请求 def process_exception(self, request, exception, spider): if request.url.split(‘:‘)[0] == ‘http‘: # 代理ip request.meta[‘proxy‘] = ‘http://‘ + random.choice(self.PROXY_http) else: request.meta[‘proxy‘] = ‘https://‘ + random.choice(self.PROXY_https) return request # 将修正之后的请求对象进行重新的请求发送
4、拦截响应：
篡改响应数据，响应对象
需求：爬取网易新闻中的新闻数据（新闻标题、内容）
--如何判断是否动态加载，通过抓包工具查看请求url的response，搜索相关内容，如果查找得到则不是动态加载，反之是动态加载
标签：user web exce spl 首页 查找 head 下载 新闻内容
原文地址：https://www.cnblogs.com/nanjo4373977/p/13024536.html
