Python爬取百度指数搜索结果，查看你想了解的热点信息吧-python黑洞网

本站消息

站长简介/公众号

出租广告位,需要合作请联系站长

听爸爸的话

1123

文章

885197

访问

+关注

分类

暂无分类

日期归档

2023-05(2)

2023-06(3)

Python爬取百度指数搜索结果，查看你想了解的热点信息吧

发布于2019-12-08 11:17 阅读(570) 评论(0) 点赞(29) 收藏(1)

今天无意间搜索问题的时候跳转到了百度指数这里，索性就打开来看看，下面是首页截图：

这里你可以自己输入自己想要查询的人物、事件等等，anything，只要是你感兴趣的都可以，有一种感觉就是你认为是热点的就是热点。。。。

闲话不多说了，这里直接进入实践，先看代码，完整的实现如下：


#!usr/bin/env python
#encoding:utf-8
 
 
'''
__Author__:沂水寒城
功能： 爬取百度指数结果
       搜索入口地址：
                    http://index.baidu.com/v2/index.html#/
'''
 
 
 
import os
import sys
import json
import time
import random
import requests
import datetime
 
 
if sys.version_info < (3, 0):
    reload(sys)
    sys.setdefaultencoding("utf-8")
 
 
month_day_dict = {
                    "01": 31,
                    "02": 28,
                    "03": 31,
                    "04": 30,
                    "05": 31,
                    "06": 30,
                    "07": 31,
                    "08": 31,
                    "09": 30,
                    "10": 31,
                    "11": 30,
                    "12": 31,
                }
 
 
def generateMonthDays(month_day_dict,year='2017',month='03'):
    '''
    生成指定年份、月份中的所有日期
    '''
    day_num=month_day_dict[month]
    day_date_list=[]
    for i in range(1,day_num+1):
        one=str(i)
        if len(one)==1:
            one='0'+one 
        day_date_list.append(year+'-'+month+'-'+one)
    return day_date_list
 
 
def generatePeriodDays(start="2019-11-01", end="2019-11-31"):
    """
    生成指定时间段内所有的 天 日期
    """
    start_year, start_mon = start.split("-")[0].strip(), start.split("-")[1].strip()
    all_day_list = generateMonthDays(month_day_dict, year=start_year, month=start_mon)
    end_year, end_mon = start.split("-")[0].strip(), start.split("-")[1].strip()
    all_day_list += generateMonthDays(month_day_dict, year=end_year, month=end_mon)
    day_list = [one for one in all_day_list if one >= start and one <= end]
    return sorted(list(set(day_list)))
 
 
def reqData(start="2019-11-01", end="2019-12-03", kw="海康威视"):
    '''
    网络请求数据
    '''
    date_list = generatePeriodDays(start = start, end = end)
    date_str = ','.join(date_list)
    search_url = urlTemplate.format(date_str, kw)
    data = requests.get(search_url, timeout=5)
    content = data.json()
    return content
 
 
def dataParser(content):
    '''
    数据解析
    '''
    data_list= content['data']["追我吧"]
    result = []
    key_list = ['date', 'title', 'url', 'source', 'same_news']
    for one_dict in data_list:
        T = one_dict['date']
        news_list = one_dict['news']
        for one_son_dict in news_list:
            one_tmp_list = []
            for i in range(len(key_list)):
                try:
                    one_tmp_list.append(one_son_dict[key_list[i]])
                except:
                    one_tmp_list.append(None)
            result.append(one_tmp_list)
    result = sorted(result, key = lambda e:e[0])
    result.insert(0, key_list)
    for one in result:
        print one
 
 
 
 
 
if __name__ == '__main__':
    content = reqData(start="2019-11-01", end="2019-12-03", kw="追我吧")
    dataParser(content)

最近《追我吧》这个节目组还是上了好久的热点头条的，这里我就搜索“追我吧”，结果如下：

{"status":0,"data":{"追我吧":[{"date":"2019-11-24","news":[{"date":"2019-11-24","title":"《追我吧<\/em>》这期大咖云集,跑男家族相聚一起,阵容再变引起热议","url":"https:\/\/new.qq.com\/omn\/20191124\/20191124A00PJP.html?pc","source":"腾讯新闻","same_news":2}]},{"date":"2019-11-25","news":[{"date":"2019-11-25","title":"《追我吧<\/em>》VS跑男家族团魂"开战" 喜提收视三连冠","url":"http:\/\/news.gxnews.com.cn\/staticpages\/20191125\/newgx5ddb716d-19060415.shtml","source":"广西新闻网","same_news":5},{"date":"2019-11-25","title":"《追我吧<\/em>》VS跑男家族团魂“开战”引爆多平台喜提收视三连冠","url":"http:\/\/www.dzwww.com\/yule\/zy\/201911\/t20191125_19393325.htm","source":"大众网","same_news":3},{"date":"2019-11-25","title":"《追我吧<\/em>》下期放大招!请来两大流量男团,粉丝都开心坏了","url":"https:\/\/new.qq.com\/omn\/20191125\/20191125A057NX.html?pc","source":"腾讯新闻","same_news":1}]},{"date":"2019-11-26","news":[{"date":"2019-11-26","title":"跑男团神武3战队双双踢馆追我吧<\/em> “踢”出周末最高收视","url":"http:\/\/www.cet.com.cn\/xwsd\/2426554.shtml","source":"中国经济新闻网","same_news":6},{"date":"2019-11-26","title":"《追我吧<\/em>》VS跑男家族团魂“开战” 引爆多平台喜提收视三连冠","url":"http:\/\/ent.ynet.com\/2019\/11\/26\/2226817t1254.html","source":"北青网","same_news":2},{"date":"2019-11-26","title":"《追我吧<\/em>》190长腿追跑者 中国山地车攀爬第一人张京坤","url":"http:\/\/www.cnr.cn\/rdzx\/cxxhl\/zxxx\/20191126\/t20191126_524873496.shtml","source":"中国广播网","same_news":1}]},{"date":"2019-11-27","news":[{"date":"2019-11-27","title":"高以翔去世是什么原因?追我吧<\/em>凌晨录制极限项目被质疑安全性","url":"http:\/\/www.mnw.cn\/news\/ent\/2224569.html","source":"闽南网","same_news":5},{"date":"2019-11-27","title":"钟楚曦回应追我吧<\/em>难度和强度:连吃3天速效救心丸","url":"http:\/\/ent.sina.com.cn\/tv\/zy\/2019-11-27\/doc-iihnzahi3716875.shtml","source":"新浪","same_news":4},{"date":"2019-11-27","title":"高以翔出事的《追我吧<\/em>》,李小鹏喊累邹市明要吸氧 高强度节目如此...","url":"http:\/\/www.dzwww.com\/yule\/nd\/201911\/t20191127_19400574.htm","source":"大众网","same_news":4}]},{"date":"2019-11-28","news":[{"date":"2019-11-28","title":"追我吧<\/em>结束录制 陈伟霆张继科离开剧组","url":"https:\/\/news.china.com\/socialgd\/10000169\/20191128\/37471028.html","source":"中华网","same_news":10},{"date":"2019-11-28","title":"探访高以翔节目录制地:《追我吧<\/em>》停录,部分设施被拆除","url":"http:\/\/news.sina.com.cn\/c\/2019-11-28\/doc-iihnzhfz2240506.shtml","source":"新浪新闻","same_news":8},{"date":"2019-11-28","title":"浙江卫视终于发声:为高以翔猝死负责!徐峥痛斥!《追我吧<\/em>》节目组有...","url":"http:\/\/finance.ifeng.com\/c\/7ryAnVtT1A7","source":"凤凰网","same_news":6}]},{"date":"2019-11-29","news":[{"date":"2019-11-29","title":"网曝《追我吧<\/em>》负责人因高以翔事件被全部开除","url":"http:\/\/ent.sina.com.cn\/tv\/zy\/2019-11-29\/doc-iihnzhfz2423924.shtml","source":"新浪","same_news":14},{"date":"2019-11-29","title":"浙江卫视宣布高以翔参录综艺《追我吧<\/em>》今晚停播","url":"http:\/\/ent.163.com\/19\/1129\/15\/EV5K41ET00038FO9.html","source":"网易","same_news":6},{"date":"2019-11-29","title":"《追我吧<\/em>》正常播出?知情人士:非新一期节目","url":"http:\/\/ent.sina.com.cn\/tv\/zy\/2019-11-29\/doc-iihnzahi4134718.shtml","source":"新浪","same_news":4}]},{"date":"2019-11-30","news":[{"date":"2019-11-30","title":"疑《追我吧<\/em>》工作人员发声:带节奏的请消停一点","url":"http:\/\/ent.163.com\/19\/1130\/07\/EV7CIOE600038FO9.html","source":"网易","same_news":3},{"date":"2019-11-30","title":"邹市明参加追我吧<\/em>腿部失去知觉,粉丝:轩轩爸爸命真大","url":"http:\/\/m.sohu.com\/a\/357415896_403336","source":"搜狐","same_news":2},{"date":"2019-11-30","title":"凭借《奔跑吧》走红的宋雨琦人气不再,参加《追我吧<\/em>》如今遭停播","url":"https:\/\/new.qq.com\/omn\/20191130\/20191130A0JQF900","source":"腾讯新闻","same_news":1}]}]},"message":""}

接下来搜索一下“海康威视”，结果如下：


{
	"status": 0,
	"data": {
		"海康威视": [
			{
				"date": "2019-11-20",
				"news": [
					{
						"date": "2019-11-20",
						"title": "海康威视</em>高级副总经理蒋海青辞职 年薪为248.62万元",
						"url": "http://finance.eastmoney.com/a/201911201296670767.html",
						"source": "东方财富网",
						"same_news": 3
					},
					{
						"date": "2019-11-20",
						"title": "...高新技术企业创新能力百强榜发布,士兰微、新华三、海康威视</em>等...",
						"url": "https://laoyaoba.com/html/share/news?source",
						"source": "集微网",
						"same_news": 1
					},
					{
						"date": "2019-11-20",
						"title": "海康威视</em>与国网杭州供电公司签订战略合作协议",
						"url": "http://www.afzhan.com/news/detail/78811.html",
						"source": "中国安防展览网",
						"same_news": 1
					}
				]
			},
			{
				"date": "2019-11-22",
				"news": [
					{
						"date": "2019-11-22",
						"title": "海康威视</em>:融资净偿还8784.07万元,两市排名第三(11-21)",
						"url": "http://stock.eastmoney.com/a/201911221299836442.html",
						"source": "东方财富网",
						"same_news": 2
					},
					{
						"date": "2019-11-22",
						"title": "后安防时代,海康威视</em>的信心与底气在哪? ",
						"url": "http://www.techsir.com/a/201911/59129.html",
						"source": "Techsir",
						"same_news": 2
					}
				]
			},
			{
				"date": "2019-11-25",
				"news": [
					{
						"date": "2019-11-25",
						"title": "海康威视</em>闪耀2019首届四川教育装备博览会 ",
						"url": "http://www.ceiea.com/html/201911/201911251324011299.shtml",
						"source": "中国教育..",
						"same_news": 1
					},
					{
						"date": "2019-11-25",
						"title": "湖南广电、网易、海康威视</em>都来了!浙传这场招聘会让2000多人达成...",
						"url": "http://edu.zjol.com.cn/jyjsb/gx/201911/t20191125_11397473.shtml",
						"source": "浙江在线",
						"same_news": 1
					},
					{
						"date": "2019-11-25",
						"title": "海康威视</em>龚虹嘉的财富密码:十年暴赚两万倍后,为何被查?",
						"url": "http://finance.sina.com.cn/stock/relnews/cn/2019-11-25/doc-iihnzahi3264395.shtml",
						"source": "新浪",
						"same_news": 1
					}
				]
			},
			{
				"date": "2019-11-29",
				"news": [
					{
						"date": "2019-11-29",
						"title": "龚虹嘉退胡扬忠进 海康威视</em>玩“对敲”?",
						"url": "http://finance.sina.com.cn/stock/s/2019-11-29/doc-iihnzhfz2363840.shtml",
						"source": "新浪",
						"same_news": 3
					},
					{
						"date": "2019-11-29",
						"title": "海康威视</em>:融资净偿还110.94万元,融资余额16.31亿元(11-28)",
						"url": "http://stock.eastmoney.com/a/201911291307749335.html",
						"source": "东方财富网",
						"same_news": 1
					},
					{
						"date": "2019-11-29",
						"title": "投资者提问:标普:将海康威视</em>移出观察名单,因供应链恢复。海康是被...",
						"url": "http://finance.sina.com.cn/stock/relnews/dongmiqa/2019-11-29/doc-iihnzhfz2562007.shtml?source",
						"source": "新浪",
						"same_news": 1
					}
				]
			},
			{
				"date": "2019-12-02",
				"news": [
					{
						"date": "2019-12-02",
						"title": "海康威视</em>:关于部分国有股权无偿划转完成的公告",
						"url": "http://finance.sina.com.cn/roll/2019-12-02/doc-iihnzhfz3131565.shtml",
						"source": "新浪",
						"same_news": 3
					},
					{
						"date": "2019-12-02",
						"title": "海康威视</em>如何构建智慧农业的基石?",
						"url": "http://www.afzhan.com/news/detail/79064.html",
						"source": "中国安防展览网",
						"same_news": 1
					},
					{
						"date": "2019-12-02",
						"title": "[公司]海康威视</em>:实控人完成划转公司0.22%国有股权",
						"url": "http://www.p5w.net/kuaixun/201912/t20191202_2358966.htm",
						"source": "全景网",
						"same_news": 1
					}
				]
			}
		]
	},
	"message": ""
}

格式化解析结果如下：


['date', 'title', 'url', 'source', 'same_news']
[u'2019-11-20', u'海康威视</em>高级副总经理蒋海青辞职 年薪为248.62万元', u'http://finance.eastmoney.com/a/201911201296670767.html', u'东方财富网', 3]
[u'2019-11-20', u'...高新技术企业创新能力百强榜发布,士兰微、新华三、海康威视</em>等...', u'https://laoyaoba.com/html/share/news?source', u'集微网', 1]
[u'2019-11-20', u'海康威视</em>与国网杭州供电公司签订战略合作协议', u'http://www.afzhan.com/news/detail/78811.html', u'中国安防展览网', 1]
[u'2019-11-22', u'海康威视</em>:融资净偿还8784.07万元,两市排名第三(11-21)', u'http://stock.eastmoney.com/a/201911221299836442.html', u'东方财富网', 2]
[u'2019-11-22', u'后安防时代,海康威视</em>的信心与底气在哪? ', u'http://www.techsir.com/a/201911/59129.html', u'techsir', 2]
[u'2019-11-25', u'海康威视</em>闪耀2019首届四川教育装备博览会 ', u'http://www.ceiea.com/html/201911/201911251324011299.shtml', u'中国教育..', 1]
[u'2019-11-25', u'湖南广电、网易、海康威视</em>都来了!浙传这场招聘会让2000多人达成...', u'http://edu.zjol.com.cn/jyjsb/gx/201911/t20191125_11397473.shtml', u'浙江在线', 1]
[u'2019-11-25', u'海康威视</em>龚虹嘉的财富密码:十年暴赚两万倍后,为何被查?', u'http://finance.sina.com.cn/stock/relnews/cn/2019-11-25/doc-iihnzahi3264395.shtml', u'新浪', 1]
[u'2019-11-29', u'龚虹嘉退胡扬忠进 海康威视</em>玩“对敲”?', u'http://finance.sina.com.cn/stock/s/2019-11-29/doc-iihnzhfz2363840.shtml', u'新浪', 3]
[u'2019-11-29', u'海康威视</em>:融资净偿还110.94万元,融资余额16.31亿元(11-28)', u'http://stock.eastmoney.com/a/201911291307749335.html', u'东方财富网', 1]
[u'2019-11-29', u'投资者提问:标普:将海康威视</em>移出观察名单,因供应链恢复。海康是被...', u'http://finance.sina.com.cn/stock/relnews/dongmiqa/2019-11-29/doc-iihnzhfz2562007.shtml?source', u'新浪', 1]
[u'2019-12-02', u'海康威视</em>:关于部分国有股权无偿划转完成的公告', u'http://finance.sina.com.cn/roll/2019-12-02/doc-iihnzhfz3131565.shtml', u'新浪', 3]
[u'2019-12-02', u'海康威视</em>如何构建智慧农业的基石?', u'http://www.afzhan.com/news/detail/79064.html', u'中国安防展览网', 1]
[u'2019-12-02', u'[公司]海康威视</em>:实控人完成划转公司0.22%国有股权', u'http://www.p5w.net/kuaixun/201912/t20191202_2358966.htm', u'全景网', 1]