ai. python Beautiful Soup tutorial. 2026-08-03

By Xah Lee. Date: .

write a tutorial of python HTML parser beautiful soup

write a tutorial of python HTML parser, beautiful soup

(ai answer)

Beautiful Soup Tutorial: Parsing HTML with Python

Beautiful Soup is Python’s most popular library for parsing HTML (and XML). It turns messy, real-world HTML into a navigable tree that you can search and extract data from easily. It is forgiving with broken markup and pairs perfectly with requests (or httpx) for web scraping.

Latest version (as of 2026): Beautiful Soup 4.15.x
Package name: beautifulsoup4
Import: from bs4 import BeautifulSoup

1. Installation

pip install beautifulsoup4 lxml requests

Parser options:

Parser Install needed? Speed Notes
lxml Yes Fastest Best default choice
html.parser No (built-in) Medium Zero dependencies
html5lib Yes Slowest Most browser-like

Always specify the parser explicitly:

soup = BeautifulSoup(html, "lxml")          # recommended
soup = BeautifulSoup(html, "html.parser")   # fallback

2. Creating a Soup Object

from bs4 import BeautifulSoup

html_doc = """
<html>
<head><title>The Dormouse's story</title></head>
<body>
  <p class="title"><b>The Dormouse's story</b></p>
  <p class="story">Once upon a time there were three little sisters; and their names were
  <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
  <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
  <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
  and they lived at the bottom of a well.</p>
  <p class="story">...</p>
</body>
</html>
"""

soup = BeautifulSoup(html_doc, "lxml")

You can also parse from a file:

with open("page.html", encoding="utf-8") as f:
    soup = BeautifulSoup(f, "lxml")

3. Core Objects

Object Description Example
BeautifulSoup The whole document soup
Tag An HTML/XML tag soup.title, soup.a
NavigableString Text inside a tag soup.title.string
ResultSet List returned by find_all() / select() soup.find_all("a") 
print(soup.title)           # <title>The Dormouse's story</title>
print(soup.title.name)      # title
print(soup.title.string)    # The Dormouse's story
print(type(soup.title))     # <class 'bs4.element.Tag'>

4. Navigating the Tree

# Direct access (first matching tag)
soup.title
soup.a
soup.p

# Parent
soup.title.parent          # <head>
soup.a.parent              # <p class="story">

# Children & descendants
list(soup.head.children)   # direct children
list(soup.head.descendants)# all levels

# Siblings
soup.a.next_sibling
soup.a.next_sibling.next_sibling   # next <a> tag

# Going up the tree
for parent in soup.a.parents:
    print(parent.name)

5. Searching the Tree (Most Important Part)

find() – returns the first match (or None)

soup.find("a")
soup.find("a", id="link2")
soup.find("p", class_="title")          # note the underscore!
soup.find(href="http://example.com/elsie")

find_all() – returns all matches (as a ResultSet)

soup.find_all("a")
soup.find_all("a", class_="sister")
soup.find_all("a", limit=2)             # only first 2 results

# Multiple classes or attributes
soup.find_all("a", attrs={"class": "sister", "id": "link1"})

CSS Selectors (very powerful)

soup.select("a.sister")                 # all <a class="sister">
soup.select("#link1")                   # by ID
soup.select("p.story > a")              # direct children
soup.select("a[href*='example']")       # attribute contains
soup.select_one("p.title")              # first match only

Tip: Prefer select() / select_one() when you know CSS. Prefer find() / find_all() when you need more Python power (functions, regex, etc.).

6. Extracting Data

# Text content
print(soup.title.string)                # single string
print(soup.get_text())                  # all text in document
print(soup.get_text(separator=" | ", strip=True))

# Attributes
link = soup.find("a")
print(link["href"])                     # http://example.com/elsie
print(link.get("href"))                 # safer (returns None if missing)
print(link.attrs)                       # {'href': '...', 'class': ['sister'], 'id': 'link1'}

# Looping over results
for a in soup.find_all("a"):
    print(a.get_text(), "→", a.get("href"))

7. Practical Example – Scrape a Real Page

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com/"
response = requests.get(url)
response.raise_for_status()             # check for HTTP errors

soup = BeautifulSoup(response.text, "lxml")

# Extract all quotes
for quote in soup.select("div.quote"):
    text = quote.select_one("span.text").get_text(strip=True)
    author = quote.select_one("small.author").get_text(strip=True)
    tags = [tag.get_text() for tag in quote.select("div.tags a.tag")]

    print(f"{text} — {author}")
    print(f"Tags: {', '.join(tags)}\n")

8. Useful Tips & Common Pitfalls

  1. Always use class_
    class is a reserved word in Python → use class_="sister".

  2. Check for None

    tag = soup.find("h1")
if tag:
    print(tag.get_text())

  3. Different parsers → different trees
    Always specify the parser so results stay consistent across machines.

  4. Get clean text

    text = tag.get_text(strip=True)

  5. Limit results on large pages

    soup.find_all("div", class_="item", limit=50)

  6. Pretty print the HTML

    print(soup.prettify())

  7. Beautiful Soup does not execute JavaScript
    If the content is loaded by JS, you need Selenium, Playwright, or a scraping API.

9. Quick Cheat Sheet

soup = BeautifulSoup(html, "lxml")

soup.find("tag")                    # first match
soup.find_all("tag")                # all matches
soup.select("css.selector")         # CSS
soup.select_one("css.selector")     # first CSS match

tag.name                            # tag name
tag.string / tag.get_text()         # text
tag["attr"] / tag.get("attr")       # attribute
tag.attrs                           # all attributes
tag.parent / tag.children           # navigation

Next Steps

Would you like a more advanced tutorial (pagination, tables, error handling, saving to CSV, concurrent scraping, etc.)?