码迷,mamicode.com
首页 > 其他好文 > 详细

爬虫-BS4

时间:2020-06-25 19:49:45      阅读:40      评论:0      收藏:0      [点我收藏+]

标签:输入   css   div   port   处理   his   lang   不用   ring   

简介
BeautifulSoup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱,通过解析文档为用户提供需要抓取的数据,因为简单,所以不需要多少代码就可以写出一个完整的应用程序。BeautifulSoup自动将输入文档转换为Unicode编码,输出文档转换为utf-8编码。你不需要考虑编码方式,除非文档没有指定一个编码方式,这时你仅仅需要说明一下原始编码方式就可以了。BeautifulSoup已成为和lxml、html6lib一样出色的python解释器,为用户灵活地提供不同的解析策略或强劲的速度

from bs4 import BeautifulSoup
html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.prettify())
print(soup.title.string)
<html>
 <head>
  <title>
   The Dormouse‘s story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse‘s story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
The Dormouse‘s story

标签选择器

# 选择元素
soup.p # 标签加其内部的所有内容,且只获取第一个匹配的标签

type(soup.p) # bs4.element.Tag
bs4.element.Tag
soup.p
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
data = []
for i in soup.find(‘body‘).children:
    data.append(i)
data
[‘\n‘,
 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>,
 ‘\n‘,
 <p class="story">Once upon a time there were three little sisters; and their names were
 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
 and they lived at the bottom of a well.</p>,
 ‘\n‘,
 <p class="story">...</p>,
 ‘\n‘]
print(soup.body.prettify())
<body>
 <p class="title" name="dromouse">
  <b>
   The Dormouse‘s story
  </b>
 </p>
 <p class="story">
  Once upon a time there were three little sisters; and their names were
  <a class="sister" href="http://example.com/elsie" id="link1">
   <!-- Elsie -->
  </a>
  ,
  <a class="sister" href="http://example.com/lacie" id="link2">
   Lacie
  </a>
  and
  <a class="sister" href="http://example.com/tillie" id="link3">
   Tillie
  </a>
  ;
and they lived at the bottom of a well.
 </p>
 <p class="story">
  ...
 </p>
</body>
# 提取信息
soup.p.string
# 获取名称
soup.p.name 
soup.p.parent.name
# 获取属性
print(soup.p.attrs) # 返回字典
print(soup.p.attrs[‘name‘])
print(soup.p[‘name‘])
# 获取内容
print(soup.p.string)
soup.title.string

type(soup.title.string)

type(soup.title.contents)

soup.head

soup.p

soup.p.string
"The Dormouse‘s story"

嵌套选择

# 每一返回结果是bs4.element.Tag
print(type(soup.p))
print(type(soup.p.b))

print(soup.p.string)
print(soup.p.b.string)
<class ‘bs4.element.Tag‘>
<class ‘bs4.element.Tag‘>
The Dormouse‘s story
The Dormouse‘s story

关联选择

# 选取到了一个节点元素之后,如果想要获取它的直接子节点可以调用contents属性
print(soup.body.contents)  # contents返回的是列表类型
[‘\n‘, <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>, ‘\n‘, <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>, ‘\n‘, <p class="story">...</p>, ‘\n‘]
print(soup.title.contents)
["The Dormouse‘s story"]
list_ = soup.body.contents
len(list_)
7

子节点和子孙节点

# print(soup.body.children)
for i,child in enumerate(soup.body.children):
    print(i,child)
0 

1 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
2 

3 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
4 

5 <p class="story">...</p>
6 

?

print(soup.body.descendants) # 生成器,递归查询所有子孙节点
for i,childs in enumerate(soup.body.descendants):
    print(i,childs)
<generator object Tag.descendants at 0x000002435F075ED0>
0 

1 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
2 <b>The Dormouse‘s story</b>
3 The Dormouse‘s story
4 

5 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
6 Once upon a time there were three little sisters; and their names were

7 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
8  Elsie 
9 ,

10 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
11 Lacie
12  and

13 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
14 Tillie
15 ;
and they lived at the bottom of a well.
16 

17 <p class="story">...</p>
18 ...
19 

?

父节点和祖先节点

print(soup.a.parent) # 返回其直接父节点即其下内容
type(soup.a.parent) # bs4.element.Tag
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>





bs4.element.Tag
print(soup.a.parents) # 
for i,des in enumerate(soup.a.parents):
    print(i,des)
<generator object Tag.descendants at 0x000002435F075ED0>
0 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
1 <body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body>
2 <html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>
3 <html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>

兄弟节点

html = """
<html>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            Hello
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(‘Next Sibling‘, soup.a.next_sibling)
print(‘Prev Sibling‘, soup.a.previous_sibling)
print(‘Next Siblings‘, list(enumerate(soup.a.next_siblings)))
print(‘Prev Siblings‘, list(enumerate(soup.a.previous_siblings)))
Next Sibling 
            Hello
            
Prev Sibling 
            Once upon a time there were three little sisters; and their names were
            
Next Siblings [(0, ‘\n            Hello\n            ‘), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ‘ \n            and\n            ‘), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, ‘\n            and they lived at the bottom of a well.\n        ‘)]
Prev Siblings [(0, ‘\n            Once upon a time there were three little sisters; and their names were\n            ‘)]

方法选择器

# find_all()、find()等方法,我们可以调用方法然后传入相应等参数就可以灵活地进行查询了
find_all(name , attrs , recursive , text , **kwargs)
find_all,顾名思义,就是查询所有符合条件的元素,可以给它传入一些属性或文本来得到符合条件的元素
soup.find_all(True) # 返回所有标签
html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
soup = BeautifulSoup(html, ‘lxml‘)

name属性

print(soup.find_all(name=‘ul‘)) # 查询所有ul标签,返回list
print(‘#‘*50)
print(type(soup.find_all(name=‘ul‘)))
print(type(soup.find_all(name=‘ul‘)[0]))
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
##################################################
<class ‘bs4.element.ResultSet‘>
<class ‘bs4.element.Tag‘>
for ul in soup.find_all(name=‘ul‘):
    print(ul.find_all(name=‘li‘)) # 结果返回列表类型,列表种的每个元素依然是Tagl类型
    for li in ul.find_all(name=‘li‘):
        print(li.string)
[]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
Foo
Bar
Jay
[]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
Foo
Bar

attrs属性

print(soup.find_all(attrs={‘class‘:‘element‘})) # 传入的是attrs参数,参数的类型是字典类型

print(soup.find_all(‘ul‘,attrs={‘id‘:‘list-1‘}))
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

常用属性如id,class可以不用attrs来传递,而直接用id和class_

print(soup.find_all(class_=‘element‘))

print(soup.find_all(id=‘list-1‘))
[<li class="element">Foo</li>,
 <li class="element">Bar</li>,
 <li class="element">Jay</li>,
 <li class="element">Foo</li>,
 <li class="element">Bar</li>]

text属性

# text参数可以用来匹配节点的文本,传入的形式可以是字符串,可以是正则表达式
import re
html=‘‘‘
<div class="panel">
    <div class="panel-body">
        <a>Hello, this is a link</a>
        <a>Hello, this is a link, too</a>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.find_all(text=re.compile(‘link‘))) # 返回所有匹配正则表达式的节点文本组成的列表
[‘Hello, this is a link‘, ‘Hello, this is a link, too‘]

find(name,attrs,recursive,text,**kwargs)
find()方法返回的是单个元素,也就是第一个匹配的元素

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,‘lxml‘)
print(soup.find(neme=‘div‘)) # Return only the first child of this Tag matching the given
print(‘#‘*10)
print(soup.find(name=‘ul‘))
print(‘#‘*10)
print(soup.find(attrs={‘panel-heading‘}))
print(‘#‘*10)
print(soup.find(class_=‘element‘))
None
##########
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
##########
<div class="panel-heading">
<h4>Hello</h4>
</div>
##########
<li class="element">Foo</li>
soup.find_parent() # 返回直接父节点
soup.find_parents() # 返回所有祖先节点
soup.find_next_sibling()# 返回后面第一个兄弟节点
soup.find_next_siblings() # 返回后面所有兄弟节点
soup.find_next() # 返回第一个符合条件的节点
soup.find_all_next() # 返回节点后所有符合条件的节点

CSS选择器

使用CSS选择器,只需要调用select()方法,传入相应的CSS选择器即可

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.select(‘.panel .panel-heading‘)) # 符合CSS选择器的节点组成的列表

print(soup.select(‘ul li‘)) # 符合CSS选择器的节点组成的列表

print(soup.select(‘#list-2 .element‘)) # 符合CSS选择器的节点组成的列表
print(type(soup.select(‘ul‘)[0])) # print(type(soup.select(‘ul‘)[0]))
[<div class="panel-heading">
<h4>Hello</h4>
</div>]

嵌套选择

soup.select(‘ul‘)
[<ul class="list" id="list-1">
 <li class="element">Foo</li>
 <li class="element">Bar</li>
 <li class="element">Jay</li>
 </ul>, <ul class="list list-small" id="list-2">
 <li class="element">Foo</li>
 <li class="element">Bar</li>
 </ul>]
for ul in soup.select(‘ul‘):
    print(ul.select(‘li‘))
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
# 获取属性
for ul in soup.select(‘ul‘):
    print(ul[‘id‘])
    print(ul.attrs[‘id‘])
    print(ul.get_text()) # 获取文本
list-1
list-1

Foo
Bar
Jay

list-2
list-2

Foo
Bar

?

print(soup.find_all(‘div‘)[1])
[out]
<div class="panel-heading">
<h4>Hello</h4>
</div>
soup.find_all(‘div‘)[1].get_text()
[out]
‘\nHello\n‘
print(soup.find_all(‘div‘)[1].h4.string)
[out]
Hello
print(soup.find_all(‘div‘)[1].string)
[out]
None

爬虫-BS4

标签:输入   css   div   port   处理   his   lang   不用   ring   

原文地址:https://www.cnblogs.com/aspiration/p/13192445.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!