我选取的网页爬取的是ppt nba版
需要的工具:pycharm,浏览器
爬虫需要观察它的网页信息,然后开始首先爬取它的html,可以看到有人气,标题,日期,咨询
可以看到用get方法
import requests
url="https://img-home.csdnimg.cn/images/20230724024159.png?origin_url=https%3A%2F%2Fwww.ptt.cc%2Fbbs%2FNBA%2Findex.html&pos_id=lJ5YB9Tu"
respone=requests.get(url)
print(respone.text)接下来把它下载下来
with open('output.html','w',encoding='utf-8')as f:
f.write(respone.text)反爬虫使用header方法,观察User-Agent: AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
放入header
import requests
url="https://img-home.csdnimg.cn/images/20230724024159.png?origin_url=https%3A%2F%2Fwww.ptt.cc%2Fbbs%2FNBA%2Findex.html&pos_id=lJ5YB9Tu"
headers={'User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'}
respone=requests.get(headers=headers)
# print(respone.text)
with open('output.html','w',encoding='utf-8')as f:
f.write(respone.text)接下来正式开始爬取自己想要的资料
去终端机安一下beautifulsoup
pip install beautifulsoup4解析获得文章列表parser
soup beautifulSoup(respone.text,"html.parser" )接下来把我们需要的值一个一个取出来
标题: div a title.text
人气:div class=nrec span
日期:div date
