user-agent与cookie查询与配置
用户代理（user-agent）是浏览器客户端与服务器交互时的重要信息之一，用于帮助网站识别请求用户的浏览器类别，以便于网站发送相应的网页数据。用户代理数据包括：操作系统标识、加密等级标识和浏览器语言。如今越来越多的网络自动搜索工具出现，尤其是网络爬虫，为了做到更好地、流畅地网络搜索，用户代理信息是不可或缺的。
开发人员工具 ——> network ——> 大小排序
![[Python爬虫] user-agent与cookie查询与配置_第1张图片](https://img.it610.com/image/info8/b2e3cb70f1554aa39407a29ad23cf13c.jpg)
![[Python爬虫] user-agent与cookie查询与配置_第2张图片](https://img.it610.com/image/info8/2249803c9544483a9bcfe63d5fb2e027.jpg)
常用user-agent
class
UA
(
)
:
USER_AGENTS
=
[
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60'
,
'Opera/8.0 (Windows NT 5.1; U; en)'
,
'Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50'
,
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0'
,
'Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36'
,
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'
,
'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER'
,
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)'
,
'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0'
,
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)'
,
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36'
]
def
get_proxy
(
self
)
:
self
.
page
+=
1
headers
=
{
"User-Agent"
:
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
}
request
=
urllib2
.
Request
(
"https://www.kuaidaili.com/free/inha/"
+
str
(
self
.
page
)
,
headers
=
headers
)
html
=
urllib2
.
urlopen
(
request
)
.
read
(
)
爬虫headers格式：
headers
=
{
"Host"
:
"######"
,
"Connection"
:
"keep-alive"
,
"Cache-Control"
:
"max-age=0"
,
"Upgrade-Insecure-Requests"
:
"1"
,
"User-Agent"
:
user_agent
,
"Accept-Encoding"
:
"gzip, deflate, br"
,
"Accept-Language"
:
"zh-CN,zh;q=0.9"
,
"Cookie"
:
"######"
}
