码迷,mamicode.com
首页 > 编程语言 > 详细

python scrapy 重复执行

时间:2019-12-27 13:22:03      阅读:116      评论:0      收藏:0      [点我收藏+]

标签:runner   configure   info   ogg   style   def   span   tor   imp   

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
import time
import logging
from scrapy.utils.project import get_project_settings


#在控制台打印日志
configure_logging()
#CrawlerRunner获取settings.py里的设置信息
runner = CrawlerRunner(get_project_settings())

@defer.inlineCallbacks
def crawl():
    while True:
        logging.info("new cycle starting")
        yield runner.crawl("xxxxx")
        #1s跑一次
        time.sleep(1)
    reactor.stop()

crawl()
reactor.run()

  

python scrapy 重复执行

标签:runner   configure   info   ogg   style   def   span   tor   imp   

原文地址:https://www.cnblogs.com/winstonsias/p/12106667.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!