码迷,mamicode.com
首页 > 编程语言 > 详细

python 第二周(第十一天) 我的python成长记 一个月搞定python数据挖掘!(19) -scrapy + mongo

时间:2017-08-04 00:29:47      阅读:234      评论:0      收藏:0      [点我收藏+]

标签:start   text   not   selectors   question   storage   select   setting   生成   

mongoDB 3.2之后默认是使用wireTiger引擎

在启动时更改存储引擎: 

  mongod --storageEngine mmapv1 --dbpath d:\data\db

这样就可以解决mongvue不能查看文档的问题啦!

 

项目流程(步骤):

前去准备(安装scrapy pymongo mongodb )

 1. 生成项目目录: scrapy startproject  stack

 2.itmes   

from scrapy import Item,Field


class StackItem(Item):
title = Field()
url = Field()

 3. 创建爬虫

from scrapy import Spider
from scrapy.selector import Selector
from stack.items import StackItem

class StackSpider(Spider):
name = "stack"
allowed_domains = ["stackoverflow.com"]
start_urls = [
"http://stackoverflow.com/questions?pagesize=50&sort=newest",
]

def parse(self, response):
questions = response.xpath(‘//div[@class="summary"]/h3‘)

for question in questions:
item = StackItem()
item[‘title‘] = question.xpath(
‘a[@class="question-hyperlink"]/text()‘).extract()[0]
item[‘url‘] = question.xpath(
‘a[@class="question-hyperlink"]/@href‘).extract()[0]
yield item

 4.学会使用xpath selectors 进行数据的提取

 5.存储数据到mongo中

  5.1 setting.py

ITEM_PIPELINES = {
‘stack.pipelines.MongoDBPipeline‘: 300,
}

MONGODB_SERVER = "localhost"
MONGODB_PORT = 27017
MONGODB_DB = "stackoverflow"
MONGODB_COLLECTION = "questions"

  5.2 pipelines.py

import pymongo

from scrapy.conf import settings
from scrapy.exceptions import DropItem
from scrapy import log

class MongoDBPipeline(object):
def __init__(self):
connection = pymongo.MongoClient(
settings[‘MONGODB_SERVER‘],
settings[‘MONGODB_PORT‘]
)
db = connection[settings[‘MONGODB_DB‘]]
self.collection = db[settings[‘MONGODB_COLLECTION‘]]

def process_item(self, item, spider):
valid = True
for data in item:
if not data:
valid = False
raise DropItem("Missing {0}!".format(data))
if valid:
self.collection.insert(dict(item))
log.msg("Question added to MongoDB database!",
level=log.DEBUG, spider=spider)

return item

 6. 启动爬虫 main.py

from scrapy import cmdline

cmdline.execute(‘scrapy crawl stack‘.split())

效果图

技术分享

 


 

    

python 第二周(第十一天) 我的python成长记 一个月搞定python数据挖掘!(19) -scrapy + mongo

标签:start   text   not   selectors   question   storage   select   setting   生成   

原文地址:http://www.cnblogs.com/yugengde/p/7282699.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!