python-爬虫初识-采集汽车资讯信息案例(一)

系统 158 0

目录

一,什么是爬虫

二,初识爬虫-采集汽车资讯信息

三,requests和BeautifulSoup模块基本使用

requests: import requests

BeautifulSoup:from bs4 import BeautifulSoup

四,初识爬虫-自动登录购酒网http://order.gjw.com/login/login

五,requests模块详细介绍

六,一大波"自动登陆"示例


一,什么是爬虫

     很久很久以前,还没有"百度","谷歌",有的还是传说中的"大黄页",如果想要上网查找一些东西....需要记住这些东西的"域名".....越来越多....想要查找这些"域名",在"大黄页"上太费劲。然后就出现了做 "搜索引擎" 的一批人.... 这就出现了像 "百度"...等等这些...在网络上抓取所有的''网页"。 

     这些"网页",等网络上的信息,这么收录到百度,这就是 爬虫, 爬虫:自动化的应用程序。分为:定向和非定向的。像"百度" 就是非定向的,什么都要爬取。

二,初识爬虫-采集汽车资讯信息

    采集汽车资讯信息- 实际生活中有某些"公司"也在使用,比如"某公司"是做汽车业务的,包括很多,比如 汽车报价...等等, 这些公司的首页有大量的“资讯文章”,由于这些公司没有编辑,文章都是通过 爬虫 到各大网站爬取到这些资讯文章,然后稍加修改,通过运营后台去把这些文章展示为自己的文章.... 

代码步骤如下-爬取汽车资讯图片(第一版):

            
              import requests
from bs4 import BeautifulSoup

# 1,拉取页面
response = requests.get('https://www.autohome.com.cn/news/')
# 2,编码
response.encoding = response.apparent_encoding
# 3,获取文本
# print(response.text)
# 4,将文本转换为html文本对象
soup = BeautifulSoup(response.text, features='html.parser')
# 5,获取id为"XXX"的html文本
target = soup.find(id='auto-channel-lazyload-article')
# 6,获取target中所有的 
              
  • 标签列表 li_list = target.findAll('li') # print(li_list) # 7,获取
  • 中所有的 for i in li_list: a = i.find('a') if a: # 获取所有 标签href属性值 print(a.attrs.get('href')) # 获取文章标题 对象类型- txt = a.find('h3') txt = a.find('h3').text print(txt) # 获取 标签src属性 img = a.find('img') img_url = img.attrs.get('src').strip("http://") print(img_url) img_url = 'http://' + img_url # 获取图片 img_response = requests.get(url=img_url) import uuid # 将图片写入本地 # file_name = str(uuid.uuid4()) + '.jpg' path = 'C:\\Users\\xxxxx\\Desktop\\skin\\%s.gif' % (str(uuid.uuid4())) with open(path, 'wb') as f: f.write(img_response.content)
  • 三,requests和BeautifulSoup模块基本使用

    requests: import requests

    response = requests.get('URL') # 发生get请求

    response.text #文本

    response.content #内容

    response.enocding #编码

    response.aparent_encoding #编码

    response.status_code #状态码

    BeautifulSoup:from bs4 import BeautifulSoup

    BeautifulSoup是一个模块,该模块用于接收一个HTML或XML字符串,然后将其进行格式化,之后遍可以使用他提供的方法进行快速查找指定元素,从而使得在HTML或XML中查找指定元素变得简单。

    suop=BeautifulSoup(response.text, features='html.parser') #将文本转换为html对象,处理引擎 features 默认为 html.parser

    suop.find('div') # 获取suop子元素中第一个div元素

    v2 = suop.findAll() # 获取所有子元素, 返回列表

    obj = v2[0] #或者for in 循环获取里面的元素

    obj.text

    obj.attrs

    四,初识爬虫-自动登录购酒网http://order.gjw.com/login/login

                
                  import requests
    from bs4 import BeautifulSoup
    
    
    # 1,发送登录请求 http://order.gjw.com/login/login
    post_dict = {
        "txtPassword": '112233aassdd',
        "txtUserName": '13393406705',
    }
    response = requests.post('http://order.gjw.com/login/login', post_dict)
    print(response.text)
    
    get_dict = response.cookies.get_dict()  # 获取cookies
    print(get_dict)
    
    get = requests.get('http://order.gjw.com/UserCenter/MyOrder.html', cookies=get_dict)
    print(get.text)
                
              

    五,requests模块详细介绍

    文档地址:

    https://2.python-requests.org//zh_CN/latest/user/quickstart.html

    request.get(...)

    request.post(...)

    request.put(....)

    request.delete(...)

    request.request("post/get",...)

    - 常用参数

    -method: 请求方式

    -url:提交地址

    -params:在url上传递的参数,get方式参数

    -data: 请求体传递参数body,post方式参数

    -json :请求体传递参数

    -headers : 请求头

    -cookies : Cookies

    -高级参数

    -files : 字典形式,文件上传参数

                
                  requests.post(
        url='xxx',
        files={
            'f1': open('s1.py', 'rb'),
            'f2': ('上传后的文件名', open('s1.py', 'rb'))
        }
    )
                
              

    -session :用于保存客户端历史访问信息.

    -更多参数

                
                  def request(method, url, **kwargs):
        """Constructs and sends a :class:`Request 
                  
                    `.
    
        :param method: method for the new :class:`Request` object.
        :param url: URL for the new :class:`Request` object.
        :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
        :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
        :param json: (optional) json data to send in the body of the :class:`Request`.
        :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
        :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
        :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
            ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
            or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string
            defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
            to add for the file.
        :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
        :param timeout: (optional) How long to wait for the server to send data
            before giving up, as a float, or a :ref:`(connect timeout, read
            timeout) 
                    
                      ` tuple.
        :type timeout: float or tuple
        :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
        :type allow_redirects: bool
        :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
        :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
        :param stream: (optional) if ``False``, the response content will be immediately downloaded.
        :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
        :return: :class:`Response 
                      
                        ` object
        :rtype: requests.Response
    
        Usage::
    
          >>> import requests
          >>> req = requests.request('GET', 'http://httpbin.org/get')
          
                        
                          
        """
                        
                      
                    
                  
                
              
                
                  def param_method_url():
        # requests.request(method='get', url='http://127.0.0.1:8000/test/')
        # requests.request(method='post', url='http://127.0.0.1:8000/test/')
        pass
    
    
    def param_param():
        # - 可以是字典
        # - 可以是字符串
        # - 可以是字节(ascii编码以内)
    
        # requests.request(method='get',
        # url='http://127.0.0.1:8000/test/',
        # params={'k1': 'v1', 'k2': '水电费'})
    
        # requests.request(method='get',
        # url='http://127.0.0.1:8000/test/',
        # params="k1=v1&k2=水电费&k3=v3&k3=vv3")
    
        # requests.request(method='get',
        # url='http://127.0.0.1:8000/test/',
        # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8'))
    
        # 错误
        # requests.request(method='get',
        # url='http://127.0.0.1:8000/test/',
        # params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding='utf8'))
        pass
    
    
    def param_data():
        # 可以是字典
        # 可以是字符串
        # 可以是字节
        # 可以是文件对象
    
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # data={'k1': 'v1', 'k2': '水电费'})
    
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # data="k1=v1; k2=v2; k3=v3; k3=v4"
        # )
    
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # data="k1=v1;k2=v2;k3=v3;k3=v4",
        # headers={'Content-Type': 'application/x-www-form-urlencoded'}
        # )
    
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件内容是:k1=v1;k2=v2;k3=v3;k3=v4
        # headers={'Content-Type': 'application/x-www-form-urlencoded'}
        # )
        pass
    
    
    def param_json():
        # 将json中对应的数据进行序列化成一个字符串,json.dumps(...)
        # 然后发送到服务器端的body中,并且Content-Type是 {'Content-Type': 'application/json'}
        requests.request(method='POST',
                         url='http://127.0.0.1:8000/test/',
                         json={'k1': 'v1', 'k2': '水电费'})
    
    
    def param_headers():
        # 发送请求头到服务器端
        requests.request(method='POST',
                         url='http://127.0.0.1:8000/test/',
                         json={'k1': 'v1', 'k2': '水电费'},
                         headers={'Content-Type': 'application/x-www-form-urlencoded'}
                         )
    
    
    def param_cookies():
        # 发送Cookie到服务器端
        requests.request(method='POST',
                         url='http://127.0.0.1:8000/test/',
                         data={'k1': 'v1', 'k2': 'v2'},
                         cookies={'cook1': 'value1'},
                         )
        # 也可以使用CookieJar(字典形式就是在此基础上封装)
        from http.cookiejar import CookieJar
        from http.cookiejar import Cookie
    
        obj = CookieJar()
        obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None,
                              discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False,
                              port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False)
                       )
        requests.request(method='POST',
                         url='http://127.0.0.1:8000/test/',
                         data={'k1': 'v1', 'k2': 'v2'},
                         cookies=obj)
    
    
    def param_files():
        # 发送文件
        # file_dict = {
        # 'f1': open('readme', 'rb')
        # }
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # files=file_dict)
    
        # 发送文件,定制文件名
        # file_dict = {
        # 'f1': ('test.txt', open('readme', 'rb'))
        # }
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # files=file_dict)
    
        # 发送文件,定制文件名
        # file_dict = {
        # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf")
        # }
        # requests.request(method='POST',
        # url='http://127.0.0.1:8000/test/',
        # files=file_dict)
    
        # 发送文件,定制文件名
        # file_dict = {
        #     'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'})
        # }
        # requests.request(method='POST',
        #                  url='http://127.0.0.1:8000/test/',
        #                  files=file_dict)
    
        pass
    
    
    def param_auth():
        from requests.auth import HTTPBasicAuth, HTTPDigestAuth
    
        ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))
        print(ret.text)
    
        # ret = requests.get('http://192.168.1.1',
        # auth=HTTPBasicAuth('admin', 'admin'))
        # ret.encoding = 'gbk'
        # print(ret.text)
    
        # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass'))
        # print(ret)
        #
    
    
    def param_timeout():
        # ret = requests.get('http://google.com/', timeout=1)
        # print(ret)
    
        # ret = requests.get('http://google.com/', timeout=(5, 1))
        # print(ret)
        pass
    
    
    def param_allow_redirects():
        ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False)
        print(ret.text)
    
    
    def param_proxies():
        # proxies = {
        # "http": "61.172.249.96:80",
        # "https": "http://61.185.219.126:3128",
        # }
    
        # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}
    
        # ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies)
        # print(ret.headers)
    
    
        # from requests.auth import HTTPProxyAuth
        #
        # proxyDict = {
        # 'http': '77.75.105.165',
        # 'https': '77.75.105.165'
        # }
        # auth = HTTPProxyAuth('username', 'mypassword')
        #
        # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth)
        # print(r.text)
    
        pass
    
    
    def param_stream():
        ret = requests.get('http://127.0.0.1:8000/test/', stream=True)
        print(ret.content)
        ret.close()
    
        # from contextlib import closing
        # with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
        # # 在此处理响应。
        # for i in r.iter_content():
        # print(i)
    
    
    def requests_session():
        import requests
    
        session = requests.Session()
    
        ### 1、首先登陆任何页面,获取cookie
    
        i1 = session.get(url="http://dig.chouti.com/help/service")
    
        ### 2、用户登陆,携带上一次的cookie,后台对cookie中的 gpsd 进行授权
        i2 = session.post(
            url="http://dig.chouti.com/login",
            data={
                'phone': "8615131255089",
                'password': "xxxxxx",
                'oneMonth': ""
            }
        )
    
        i3 = session.post(
            url="http://dig.chouti.com/link/vote?linksId=8589623",
        )
        print(i3.text)
    
    参数示例
                
              

    六,一大波"自动登陆"示例

    1.github

                
                  #!/usr/bin/env python
    # -*- coding:utf-8 -*-
    
    import requests
    from bs4 import BeautifulSoup
    
    # ############## 方式一 ##############
    #
    # # 1. 访问登陆页面,获取 authenticity_token
    # i1 = requests.get('https://github.com/login')
    # soup1 = BeautifulSoup(i1.text, features='lxml')
    # tag = soup1.find(name='input', attrs={'name': 'authenticity_token'})
    # authenticity_token = tag.get('value')
    # c1 = i1.cookies.get_dict()
    # i1.close()
    #
    # # 1. 携带authenticity_token和用户名密码等信息,发送用户验证
    # form_data = {
    # "authenticity_token": authenticity_token,
    #     "utf8": "",
    #     "commit": "Sign in",
    #     "login": "xxxx@live.com",
    #     'password': 'xxoo'
    # }
    #
    # i2 = requests.post('https://github.com/session', data=form_data, cookies=c1)
    # c2 = i2.cookies.get_dict()
    # c1.update(c2)
    # i3 = requests.get('https://github.com/settings/repositories', cookies=c1)
    #
    # soup3 = BeautifulSoup(i3.text, features='lxml')
    # list_group = soup3.find(name='div', class_='listgroup')
    #
    # from bs4.element import Tag
    #
    # for child in list_group.children:
    #     if isinstance(child, Tag):
    #         project_tag = child.find(name='a', class_='mr-1')
    #         size_tag = child.find(name='small')
    #         temp = "项目:%s(%s); 项目路径:%s" % (project_tag.get('href'), size_tag.string, project_tag.string, )
    #         print(temp)
    
    
    
    # ############## 方式二 ##############
    # session = requests.Session()
    # # 1. 访问登陆页面,获取 authenticity_token
    # i1 = session.get('https://github.com/login')
    # soup1 = BeautifulSoup(i1.text, features='lxml')
    # tag = soup1.find(name='input', attrs={'name': 'authenticity_token'})
    # authenticity_token = tag.get('value')
    # c1 = i1.cookies.get_dict()
    # i1.close()
    #
    # # 1. 携带authenticity_token和用户名密码等信息,发送用户验证
    # form_data = {
    #     "authenticity_token": authenticity_token,
    #     "utf8": "",
    #     "commit": "Sign in",
    #     "login": "xxxx@live.com",
    #     'password': 'xxoo'
    # }
    #
    # i2 = session.post('https://github.com/session', data=form_data)
    # c2 = i2.cookies.get_dict()
    # c1.update(c2)
    # i3 = session.get('https://github.com/settings/repositories')
    #
    # soup3 = BeautifulSoup(i3.text, features='lxml')
    # list_group = soup3.find(name='div', class_='listgroup')
    #
    # from bs4.element import Tag
    #
    # for child in list_group.children:
    #     if isinstance(child, Tag):
    #         project_tag = child.find(name='a', class_='mr-1')
    #         size_tag = child.find(name='small')
    #         temp = "项目:%s(%s); 项目路径:%s" % (project_tag.get('href'), size_tag.string, project_tag.string, )
    #         print(temp)
    
    github
                
              

    2,知乎

                
                  #!/usr/bin/env python
    # -*- coding:utf-8 -*-
    import time
    
    import requests
    from bs4 import BeautifulSoup
    
    session = requests.Session()
    
    i1 = session.get(
        url='https://www.zhihu.com/#signin',
        headers={
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
        }
    )
    
    soup1 = BeautifulSoup(i1.text, 'lxml')
    xsrf_tag = soup1.find(name='input', attrs={'name': '_xsrf'})
    xsrf = xsrf_tag.get('value')
    
    current_time = time.time()
    i2 = session.get(
        url='https://www.zhihu.com/captcha.gif',
        params={'r': current_time, 'type': 'login'},
        headers={
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
        })
    
    with open('zhihu.gif', 'wb') as f:
        f.write(i2.content)
    
    captcha = input('请打开zhihu.gif文件,查看并输入验证码:')
    form_data = {
        "_xsrf": xsrf,
        'password': 'xxooxxoo',
        "captcha": 'captcha',
        'email': '424662508@qq.com'
    }
    i3 = session.post(
        url='https://www.zhihu.com/login/email',
        data=form_data,
        headers={
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
        }
    )
    
    i4 = session.get(
        url='https://www.zhihu.com/settings/profile',
        headers={
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
        }
    )
    
    soup4 = BeautifulSoup(i4.text, 'lxml')
    tag = soup4.find(id='rename-section')
    nick_name = tag.find('span',class_='name').string
    print(nick_name)
    
    知乎
                
              

    3,博客园

                
                  #!/usr/bin/env python
    # -*- coding:utf-8 -*-
    import re
    import json
    import base64
    
    import rsa
    import requests
    
    
    def js_encrypt(text):
        b64der = 'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCp0wHYbg/NOPO3nzMD3dndwS0MccuMeXCHgVlGOoYyFwLdS24Im2e7YyhB0wrUsyYf0/nhzCzBK8ZC9eCWqd0aHbdgOQT6CuFQBMjbyGYvlVYU2ZP7kG9Ft6YV6oc9ambuO7nPZh+bvXH0zDKfi02prknrScAKC0XhadTHT3Al0QIDAQAB'
        der = base64.standard_b64decode(b64der)
    
        pk = rsa.PublicKey.load_pkcs1_openssl_der(der)
        v1 = rsa.encrypt(bytes(text, 'utf8'), pk)
        value = base64.encodebytes(v1).replace(b'\n', b'')
        value = value.decode('utf8')
    
        return value
    
    
    session = requests.Session()
    
    i1 = session.get('https://passport.cnblogs.com/user/signin')
    rep = re.compile("'VerificationToken': '(.*)'")
    v = re.search(rep, i1.text)
    verification_token = v.group(1)
    
    form_data = {
        'input1': js_encrypt('wptawy'),
        'input2': js_encrypt('asdfasdf'),
        'remember': False
    }
    
    i2 = session.post(url='https://passport.cnblogs.com/user/signin',
                      data=json.dumps(form_data),
                      headers={
                          'Content-Type': 'application/json; charset=UTF-8',
                          'X-Requested-With': 'XMLHttpRequest',
                          'VerificationToken': verification_token}
                      )
    
    i3 = session.get(url='https://i.cnblogs.com/EditDiary.aspx')
    
    print(i3.text)
    
    博客园
                
              

    4,拉个网

                
                  #!/usr/bin/env python
    # -*- coding:utf-8 -*-
    
    import requests
    
    
    # 第一步:访问登陆页,拿到X_Anti_Forge_Token,X_Anti_Forge_Code
    # 1、请求url:https://passport.lagou.com/login/login.html
    # 2、请求方法:GET
    # 3、请求头:
    #    User-agent
    r1 = requests.get('https://passport.lagou.com/login/login.html',
                     headers={
                         'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
                     },
                     )
    
    X_Anti_Forge_Token = re.findall("X_Anti_Forge_Token = '(.*?)'", r1.text, re.S)[0]
    X_Anti_Forge_Code = re.findall("X_Anti_Forge_Code = '(.*?)'", r1.text, re.S)[0]
    print(X_Anti_Forge_Token, X_Anti_Forge_Code)
    # print(r1.cookies.get_dict())
    # 第二步:登陆
    # 1、请求url:https://passport.lagou.com/login/login.json
    # 2、请求方法:POST
    # 3、请求头:
    #    cookie
    #    User-agent
    #    Referer:https://passport.lagou.com/login/login.html
    #    X-Anit-Forge-Code:53165984
    #    X-Anit-Forge-Token:3b6a2f62-80f0-428b-8efb-ef72fc100d78
    #    X-Requested-With:XMLHttpRequest
    # 4、请求体:
    # isValidate:true
    # username:15131252215
    # password:ab18d270d7126ea65915c50288c22c0d
    # request_form_verifyCode:''
    # submit:''
    r2 = requests.post(
        'https://passport.lagou.com/login/login.json',
        headers={
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
            'Referer': 'https://passport.lagou.com/login/login.html',
            'X-Anit-Forge-Code': X_Anti_Forge_Code,
            'X-Anit-Forge-Token': X_Anti_Forge_Token,
            'X-Requested-With': 'XMLHttpRequest'
        },
        data={
            "isValidate": True,
            'username': '15131255089',
            'password': 'ab18d270d7126ea65915c50288c22c0d',
            'request_form_verifyCode': '',
            'submit': ''
        },
        cookies=r1.cookies.get_dict()
    )
    print(r2.text)
    
    拉勾网
                
              

     


    更多文章、技术交流、商务合作、联系博主

    微信扫码或搜索:z360901061

    微信扫一扫加我为好友

    QQ号联系: 360901061

    您的支持是博主写作最大的动力,如果您喜欢我的文章,感觉我的文章对您有帮助,请请扫描上面二维码支持博主1元、2元、5元等您想捐的金额吧,狠狠点击下面给点支持吧

    发表我的评论
    最新评论 总共0条评论