https://plumephp.com/tags/beautifulsoup/Recent content in BeautifulSoup on PlumePHPHugozh-CNMon, 28 Sep 2026 14:00:00 +0800https://plumephp.com/python-web-scraping-automation/Mon, 28 Sep 2026 14:00:00 +0800https://plumephp.com/python-web-scraping-automation/<blockquote> <p>网络爬虫是从公开网页采集数据的自动化技术。本文覆盖从单页请求到分布式爬虫、从静态页面到浏览器自动化的完整工具箱，并强调<strong>合规边界</strong>。</p> </blockquote> <hr> <h2 id="目录">目录</h2> <ol> <li><a href="#1-%E7%88%AC%E8%99%AB%E7%9A%84%E5%90%88%E8%A7%84%E8%BE%B9%E7%95%8C">爬虫的合规边界</a></li> <li><a href="#2-http-%E8%AF%B7%E6%B1%82requests-%E4%B8%8E-httpx">HTTP 请求：requests 与 httpx</a></li> <li><a href="#3-%E8%A7%A3%E6%9E%90-htmlbeautifulsoup-%E4%B8%8E-lxml">解析 HTML：BeautifulSoup 与 lxml</a></li> <li><a href="#4-xpath-%E4%B8%8E-css-%E9%80%89%E6%8B%A9%E5%99%A8">XPath 与 CSS 选择器</a></li> <li><a href="#5-json-api-%E4%B8%8E%E5%BC%82%E6%AD%A5%E6%8A%93%E5%8F%96">JSON API 与异步抓取</a></li> <li><a href="#6-scrapy-%E6%A1%86%E6%9E%B6">Scrapy 框架</a></li> <li><a href="#7-playwright-%E6%B5%8F%E8%A7%88%E5%99%A8%E8%87%AA%E5%8A%A8%E5%8C%96">Playwright 浏览器自动化</a></li> <li><a href="#8-%E5%8F%8D%E7%88%AC%E5%BA%94%E5%AF%B9%E4%B8%8E%E9%99%90%E9%80%9F">反爬应对与限速</a></li> <li><a href="#9-%E6%95%B0%E6%8D%AE%E6%B8%85%E6%B4%97%E4%B8%8E%E5%AD%98%E5%82%A8">数据清洗与存储</a></li> <li><a href="#10-%E9%80%9F%E6%9F%A5%E8%A1%A8%E4%B8%8E%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5">速查表与最佳实践</a></li> </ol> <hr> <h2 id="1-爬虫的合规边界">1. 爬虫的合规边界</h2> <p>在动手之前，先确认边界：</p>