CyberCode Academy

CyberCode Academy

By CyberCode AcademyTechnologyEducationCourses
Download on the App Store

CyberCode Academy episodes

  • Course 40 - Web Scraping with Python | Episode 18: Mastering HTML Parse Tree Navigation and Element Extraction with Beautiful Soup
    In this lesson, you’ll learn about: how Beautiful Soup builds a navigable HTML tree, how to search and filter elements, and how to move through the structure to extract clean, structured data1. Parsing HTML with Beautiful Soup🔹 From Raw HTML → Structured Tree🔹 Basic Workflowimport requests from bs4 import BeautifulSoup html = requests.get("https://example.com").text soup = BeautifulSoup(html, "lxml") 🔹 Visualizing the Structureprint(soup.prettify()) 👉 Key Insight
    Beautiful Soup turns messy HTML into a clean tree structure2. Core Elements of the Parse Tree🔹 The 4 Building Blocks🔹 Key Components
    • Tags → HTML elements (, )
    • Attributes → stored as dictionaries
    NavigableString → text inside tagsComments → hidden HTML notes🔹 Exampletag = soup.a tag.attrs tag.string 👉 Key Insight
    Everything in HTML becomes an object you can navigate3. Searching & Filtering Elements🔹 Finding Data Efficiently🔹 Common Methodssoup.title soup.find("div") soup.find_all("a") 🔹 Using Regeximport re soup.find_all("a", href=re.compile("example")) 👉 Key Insight
    find_all() is your main tool for scalable extraction4. Navigating the HTML Tree🔹 Directional Navigation5. Moving Down the Treesoup.body.contents    Access childrenIterate through nested elements6. Moving Up the Treetag.parentMove to parentAccess ancestors7. Moving Sidewaystag.next_sibling tag.previous_siblingAccess elements at same level👉 Key Insight
    Scraping = navigating the tree in the right direction8. Extracting Clean Data🔹 Practical Extraction🔹 Example: Extract Table Datafor row in soup.find_all("tr"): cols = row.find_all("td") data = [col.text.strip() for col in cols] 👉 Key Insight
    .text + .strip() = clean usable data9. Mental ModelThink of BeautifulSoup as:    🌳 A tree🔍 find() = search tool🧭 navigation = movement (up/down/sideways)Final TakeawayBeautiful Soup transforms web scraping from:❌ guessing text patterns
    ➡️ into
    ✅ navigating structured dataOnce you understand:    Tree structureSearch methodsNavigation directions👉 You gain full control over extracting any data from any HTML page

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    23 min
  • Course 40 - Web Scraping with Python | Episode 17: Mastering Requests, Regex, and Beautiful Soup
    In this lesson, you’ll learn about: how Python retrieves web pages, how regex is used for pattern-based extraction, and how BeautifulSoup improves scraping by understanding HTML structure instead of treating it as plain text1. Fetching Web Content in Python🔹 HTTP Request FlowWeb scraping always starts with getting the page content.🔹 Libraries Used
    • urllib → built-in, basic control
    • httplib2 → low-level control
    • requests → easiest and most popular
    🔹 Requests Exampleimport requests response = requests.get("https://example.com") html = response.text 🔹 User-Agent HandlingSome sites block bots, so you can:headers = {"User-Agent": "Mozilla/5.0"} requests.get(url, headers=headers) 👉 Key Insight
    Without proper headers, many sites will reject your scraper2. Regular Expressions (Regex Basics)🔹 Pattern Matching ConceptRegex treats web data as raw text patterns.3. Core Regex FunctionsFunctionBehaviormatch()checks start onlysearch()finds first match anywherefindall()returns all matches🔹 Special SymbolsSymbolMeaning\ddigits\wletters + numbers\swhitespace🔹 Example Patternimport re re.findall(r"\d+", "Price is 123 dollars") 👉 Key Insight
    Regex is powerful but fragile for HTML4. Advanced Regex Techniques🔹 Ranges & Groups
    • [A-Z] → uppercase letters
    • {3} → exact repetition
    • ( ) → capture groups
    🔹 Example: Extract Namesre.search(r"(\w+) (\w+)", "John Smith") 5. Real Web Scraping Use Cases🔹 Inspecting HTMLUsing browser tools, you can locate:
    • items
    , headerscontact detailslocation data🔹 Example Targets
    • Phone numbers
    • Zip codes
    • City/state data
    6. BeautifulSoup (Structured Parsing)🔹 DOM-Based ApproachBeautifulSoup understands HTML as a tree structure, not text.🔹 Basic Usagefrom bs4 import BeautifulSoup soup = BeautifulSoup(html, "lxml") print(soup.title.string) 🔹 Key Advantage
    • Navigates tags easily
    • Handles broken HTML
    • Cleaner extraction than regex
    7. Parsers (LXML vs HTML5lib)ParserStrengthlxmlfasthtml5libvery forgiving👉 Key Insight
    Parser choice affects speed vs accuracy8. Regex vs BeautifulSoupFeatureRegexBeautifulSoupStructure aware❌✔️Speed✔️MediumReliability❌✔️9. Mental ModelThink of scraping like:
    • 📥 Requests → download page
    • 🔍 Regex → pattern hunting
    • 🌳 BeautifulSoup → structured navigation
    Final TakeawayWeb scraping becomes powerful when you stop treating HTML as text and start treating it as a structured tree of data.👉 Use:
    • Requests → fetch
    • Regex → quick patterns
    • BeautifulSoup → real extraction
    That combination covers most real-world scraping tasks.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    22 min
  • Course 40 - Web Scraping with Python | Episode 16: Mastering Data Extraction with Beautiful Soup
    In this lesson, you’ll learn about: how web scraping works end-to-end, why fetching and parsing are the two core stages, and how different tools like Regex, BeautifulSoup, and Scrapy compare in real-world data extraction1. What is Web Scraping?🔹 Core IdeaWeb scraping = automated data extraction from websitesInstead of manually copying data, a program:
    • Visits a page
    • Reads the HTML
    • Extracts structured information
    2. Two-Phase Scraping Workflow🔹 Overall PipelinePhase 1: Fetching Content
    • Send HTTP request (GET)
    • Receive HTML response
    • Store raw page content
    Tools:
    • Requests
    • urllib
    • httplib2
    Phase 2: Parsing & Extraction
    • Analyze HTML structure
    • Extract required data
    • Clean results
    3. Regex vs Structured Parsers🔹 Regular ExpressionsRegex:
    • Works on text patterns
    • Fast but fragile
    • Breaks easily on messy HTML
    👉 Key Insight
    HTML is not flat text—it’s structured data4. BeautifulSoup (Structure-Aware Parsing)🔹 Why It Works BetterBeautifulSoup:
    • Understands HTML tree structure
    • Fixes broken markup
    • Lets you navigate elements easily
    🔹 Key AdvantageInstead of guessing text patterns:👉 you navigate the DOM like a tree5. HTML vs DOM ParsingTypeDescriptionHTML parsingRaw server outputDOM parsingRendered browser structure🔹 Important Difference
    • HTML = static snapshot
    • DOM = live, updated by JavaScript
    6. Static vs Dynamic Content🔹 Static Pages
    • Easy to scrape
    • No JavaScript required
    • BeautifulSoup works well
    🔹 Dynamic Pages
    • Content generated by JavaScript
    • Requires browser rendering
    Tools:
    • Selenium
    • Scrapy
    • Headless browsers
    👉 Key Insight
    If data appears after page load → you need a browser engine7. Advanced Tools Overview🔹 Scrapy (Industrial Tool)
    • Built for scale
    • Handles crawling + pipelines
    • Used for production systems
    🔹 Selenium
    • Controls real browser
    • Handles JavaScript
    • Slower but powerful
    🔹 Computer Vision Scraping (Sikuli)
    • Reads screen pixels
    • Works without HTML
    • Used when UI has no accessible structure
    8. Mental ModelThink of scraping as:
    • 📥 Fetch → download the page
    • 🧠 Parse → understand structure
    • 🎯 Extract → get useful data
    Final TakeawayWeb scraping is not just “copying data”—it’s a structured pipeline:👉 fetch → parse → extract → transformAnd the tool you choose depends on one question:Is the data static HTML or dynamically generated?That single decision determines everything else.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    20 min
  • Course 40 - Web Scraping with Python | Episode 15: Mastering Items, Loaders, and Processing Pipelines
    In this lesson, you’ll learn about: how Scrapy structures scraped data using Items, how Item Loaders simplify extraction and cleaning, and how Pipelines transform raw scraped output into usable datasets1. Scrapy Items (Structured Data Containers)🔹 What Are Items?Scrapy Items are structured containers for scraped data.Think of them as:a strongly-typed dictionary for scraped content🔹 Example Structureclass StockItem(scrapy.Item): name = scrapy.Field() symbol = scrapy.Field() price = scrapy.Field() 👉 Key Insight
    Items force structure into messy web data2. Using Items in Scrapy Shell🔹 Manual Assignment FlowYou can:
    • Test XPath selectors
    • Extract values manually
    • Assign them into Items
    🔹 Exampleitem["name"] = response.xpath("//h1/text()").get() item["price"] = response.xpath("//fin-streamer/text()").get() 👉 Key Insight
    Scrapy Shell helps you validate structure before automation3. Project-Based Item Integration🔹 Moving into Real SpidersItems are defined in:items.py Then used inside spiders:yield StockItem( name=name, symbol=symbol, price=price ) 👉 Key Insight
    Items enforce consistency across your whole scraping system4. Exporting Data (CSV / JSON)🔹 Built-in Export Systemscrapy crawl stocks -o data.csv 🔹 Output Formats
    • CSV → analytics
    • JSON → APIs
    • XML → legacy systems
    👉 Key Insight
    Scrapy can export structured data without extra libraries5. Item Loaders (Automation Layer)🔹 Why They ExistItem Loaders reduce repetitive code and handle transformation automatically.🔹 Example Usageloader.add_xpath("price", "//span/text()") 6. Input & Output Processors🔹 MapCompose (Input Cleaning)from scrapy.loader.processors import MapCompose Used to:
    • Clean URLs
    • Format strings
    • Convert data types
    🔹 TakeFirst (Output Simplification)from scrapy.loader.processors import TakeFirst Used to:
    • Convert lists → single values
    👉 Key Insight
    Processors turn raw extraction into clean structured data automatically7. Pipelines (Post-Processing System)🔹 What Happens After ScrapingPipelines run after data extraction🔹 Example Pipelineclass PriceFilterPipeline: def process_item(self, item, spider): if float(item["price"]) > 100: item["high_value"] = True return item 👉 Key Insight
    Pipelines are where business logic lives8. Enabling PipelinesIn settings.py:ITEM_PIPELINES = { "myproject.pipelines.PriceFilterPipeline": 300, } Lower number = higher priority9. Full Data Flow Model
    1. Spider extracts data
    2. Items structure it
    3. Item Loaders clean it
    4. Pipelines transform it
    5. Export stores it
    10. Mental ModelThink of Scrapy like a factory:
    • 🕷️ Spider → collector
    • 📦 Items → containers
    • 🧼 Loaders → cleaning station
    • 🏭 Pipelines → production line
    Final TakeawayScrapy is not just about scraping—it’s about turning raw web data into structured, validated datasets automatically.Once you master Items → Loaders → Pipelines:👉 you stop “extracting data”
    👉 and start engineering data systems

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    25 min
  • Course 40 - Web Scraping with Python | Episode 14: Building and Automating Custom Spiders with the Scrapy Framework
    In this lesson, you’ll learn about: Scrapy’s full architecture, how to build real spiders from scratch, and how to move from simple extraction to production-ready crawling with structured data pipelines1. Scrapy Architecture (How Everything Works)🔹 Core System FlowScrapy is built around a central engine that coordinates everything.🔹 Main ComponentsComponentRoleEngineControls flowSchedulerQueues URLsDownloaderFetches pagesSpiderExtracts dataPipelineProcesses & stores data👉 Key Insight
    You don’t control HTTP manually—Scrapy does it for you2. Project Setup & Spider Creation🔹 Initialize a Projectscrapy startproject myproject 🔹 Generate a Spiderscrapy genspider stocks yahoo.com 🔹 Project Structuremyproject/ ├── spiders/ ├── items.py ├── pipelines.py ├── settings.py 👉 Key Insight
    Each file has a strict responsibility → clean separation of logic3. Extracting Real Data (Yahoo Finance Example)🔹 Target Use CaseWe extract:
    • Company name
    • Stock price
    • Market data
    🔹 XPath in Spiderdef parse(self, response): yield { "name": response.xpath("//h1/text()").get(), "price": response.xpath("//fin-streamer[@data-field='regularMarketPrice']/text()").get() } 👉 Key Insight
    Spiders are just Python classes with extraction rules4. Running the Spider🔹 Execution Commandscrapy crawl stocks 🔹 Output Options
    • Console print
    • JSON export
    • CSV export
    • File writing
    🔹 Save to Filescrapy crawl stocks -o data.json 👉 Key Insight
    Scrapy supports structured output without extra code5. Item Loaders (Cleaner Code)🔹 Why They MatterItem Loaders help:
    • Clean data
    • Normalize values
    • Reduce repeated logic
    🔹 Examplefrom scrapy.loader import ItemLoader loader = ItemLoader(item=StockItem(), response=response) loader.add_xpath("price", "//span/text()") return loader.load_item() 👉 Key Insight
    You separate extraction from transformation6. Pipelines (Final Processing Layer)🔹 What Pipelines Do
    • Clean data
    • Validate data
    • Save to database/files
    🔹 Example Pipelineclass CleanPipeline: def process_item(self, item, spider): item["price"] = float(item["price"]) return item 👉 Key Insight
    Pipelines act like a data factory assembly line7. Full Data Flow
    1. Scheduler queues URL
    2. Downloader fetches page
    3. Spider extracts data
    4. Pipeline cleans it
    5. Output stored
    8. Mental ModelThink of Scrapy as:
    • 🧠 Brain → Engine
    • 📦 Factory line → Pipelines
    • 🕷️ Workers → Spiders
    • 🚚 Delivery system → Downloader
    Final TakeawayScrapy turns scraping into a fully automated data engineering system.Once you combine:
    • Spiders (logic)
    • Selectors (extraction)
    • Pipelines (processing)
    👉 You don’t just collect data anymore—you build production-grade data pipelines.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    23 min
  • Course 40 - Web Scraping with Python | Episode 13: Mastering Scrapy Shell, CSS, and XPath Selectors
    In this lesson, you’ll learn about: how to use Scrapy Shell for interactive crawling, how CSS selectors work for fast extraction, and how XPath enables advanced and flexible data targeting1. What is Scrapy Shell?🔹 Interactive Prototyping ToolScrapy Shell is a live testing environment where you can:
    • Test selectors before writing spiders
    • Inspect HTML responses instantly
    • Experiment with scraping logic
    🔹 Key Objects Inside Shell
    • response → HTML content of the page
    • request → HTTP request details
    • spider → scraper context
    👉 Key Insight
    You can test everything before writing real crawling logic2. Working with Live URLs and FilesScrapy Shell supports:
    • 🌐 Live websites
    • 📄 Local HTML files
    👉 This makes it ideal for debugging broken or complex pages3. CSS Selectors (Fast & Simple)🔹 Basic Extraction🔹 Common SyntaxSelectorMeaning#idSelect by ID.classSelect by classtagSelect by tag🔹 Scrapy Shell Methodsresponse.css("title").get() response.css("p").get_all() 🔹 Extract Attributesresponse.css("img::attr(src)").get() 👉 Key Insight
    CSS is perfect for quick, readable extraction4. Important Behavior: Cached Responses🔹 One Hidden DetailScrapy Shell:
    • Works on cached HTML
    • Won’t reflect live changes unless restarted
    👉 Key Insight
    Always restart shell when debugging updated pages5. XPath Selectors (Advanced Power)🔹 Full DOM NavigationXPath lets you navigate HTML like a tree structure6. Absolute vs Relative XPath🔹 Absolute Path/html/body/div/p
    • Starts from root
    • Very strict
    🔹 Relative Path//div/p
    • Searches anywhere
    • More flexible
    7. Attribute Matching in XPath🔹 Using @response.xpath("//img[@id='logo']").get() 🔹 Using Wildcardsresponse.xpath("//*[@id='main']").get() 8. Using contains()🔹 Pattern Matchingresponse.xpath("//p[contains(text(), 'news')]").get() 👉 Key Insight
    XPath is powerful for uncertain or messy HTML structures9. CSS vs XPathFeatureCSSXPathSimplicity✔️ Easy❌ More complexPower❌ Limited✔️ Very powerfulFlexibilityMediumVery high10. Mental ModelThink of Scrapy Shell as:
    • A laboratory
    • CSS = quick filters
    • XPath = surgical precision tools
    Final TakeawayScrapy Shell bridges the gap between:👉 “guessing selectors”
    and
    👉 “engineered extraction logic”Once you master CSS + XPath inside the shell, you can confidently build spiders that work on even the most complex websites.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    21 min
  • Course 40 - Web Scraping with Python | Episode 12: From Parsing Foundations to Scrapy Essentials
    In this lesson, you’ll learn about: how Scrapy turns simple scraping into large-scale crawling systems, the difference between scraping and crawling, and how to use a framework-driven approach for industrial web data extraction1. From Parsing to Real-World Crawling🔹 HTML vs DOM Parsing🔹 Key DifferenceTypeWhat it seesHTML parsingRaw server responseDOM parsingFinal rendered page👉 Key Insight
    JavaScript can completely change what your scraper sees after load2. Scraping vs Crawling🔹 Two Levels of Data CollectionConceptScopeScrapingSpecific pages/dataCrawlingEntire websites🔹 Real-World Analogy
    • Scraping → reading one article
    • Crawling → reading the entire library
    3. Why Scrapy Exists🔹 The Framework AdvantageScrapy is not just a tool—it is a framework.👉 It controls execution and calls your code🔹 Inversion of ControlInstead of:you controlling everythingScrapy:controls the flow and executes your logic4. Core Scrapy Concepts🔹 Spider System
    • Defines what to crawl
    • Defines how to parse data
    • Sends requests automatically
    🔹 Engine Flow
    1. Scheduler queues URLs
    2. Engine sends requests
    3. Spider processes responses
    4. Pipeline stores data
    5. Getting Started Tools🔹 Installationpip install scrapy 🔹 Useful Commands
    • scrapy bench → performance test
    • scrapy fetch URL → download raw HTML
    • scrapy view URL → see rendered page
    👉 Key Insight
    These tools let you inspect how Scrapy “sees” the web6. Scrapy Shell (Prototyping Tool)🔹 Interactive TestingUse it to:
    • Test selectors
    • Debug parsing logic
    • Inspect live responses
    7. CSS vs XPath Selectors🔹 Two Ways to Target DataMethodStrengthCSSSimple & readableXPathPowerful & flexible🔹 Exampleresponse.css("div.title").get() response.xpath("//div[@class='title']").get() 👉 Key Insight
    XPath can navigate complex structures CSS cannot8. Performance Thinking🔹 Why Scrapy is Fast
    • Asynchronous requests
    • Built-in scheduler
    • Efficient pipelines
    🔹 Metrics You Monitor
    • Pages per minute
    • Response size
    • Crawl depth
    9. Mental ModelThink of Scrapy as:
    • A robot army
    • A data factory
    • A controlled pipeline system
    You only define:
    👉 what to collect
    👉 how to parse itFinal TakeawayScrapy is where web scraping becomes engineering instead of scripting.Once you understand its structure:
    • Scraping becomes scalable
    • Crawling becomes automated
    • Data collection becomes production-grade
    And you stop writing scripts… and start building systems.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    20 min
  • Course 40 - Web Scraping with Python | Episode 11: Advanced Filtering and Efficient Extraction with BeautifulSoup
    In this lesson, you’ll learn about: advanced BeautifulSoup filtering techniques, custom extraction logic, real-world link scraping, and performance optimization using SoupStrainer1. Core Extraction Tools: find vs find_all🔹 The Basic Building Blocks🔹 What They DoMethodPurposefind()Returns first matchfind_all()Returns all matches🔹 Basic Examplefrom bs4 import BeautifulSoup import requests html = requests.get("https://example.com").text soup = BeautifulSoup(html, "lxml") soup.find("p") soup.find_all("a") 2. Filtering Beyond Tags🔹 Attribute-Based Selectionsoup.find_all("img", src=True) soup.find_all("a", id="main-link") 👉 Key Insight
    You’re no longer just finding tags—you’re filtering structured conditions3. Using Regular Expressions for Precision🔹 Regex in BeautifulSoupUse Regular Expressions for advanced filtering:import re soup.find_all("a", href=re.compile("wiki")) 🔹 What This Enables
    • Match patterns in URLs
    • Filter partial text
    • Detect structured formats
    4. Custom Filtering Functions (Advanced Logic)🔹 When Built-ins Aren’t Enoughdef custom_filter(tag): return tag.has_attr("src") and not tag.has_attr("href") soup.find_all(custom_filter) 🔹 Real Use Cases
    • Images without links
    • Links pointing to specific domains
    • Complex multi-condition filtering
    👉 Key Insight
    You can encode any logic you want in Python5. Real-World Project: Scraping Links🔹 Target Site Workflow🔹 Step 1: Fetch Pageimport requests from bs4 import BeautifulSoup url = "https://mashable.com" html = requests.get(url).text soup = BeautifulSoup(html, "lxml") 🔹 Step 2: Extract Linkslinks = soup.find_all("a") 6. Absolute vs Relative URLs🔹 The ProblemTypeExampleAbsolutehttps://site.com/pageRelative/page🔹 Fixing Relative Linksfrom urllib.parse import urljoin full_url = urljoin(url, "/about") 👉 Key Insight
    Scrapers must normalize URLs for reliability7. Performance Optimization with SoupStrainer🔹 The IdeaInstead of parsing everything…👉 parse only what you need🔹 Implementationfrom bs4 import SoupStrainer, BeautifulSoup only_links = SoupStrainer("a") soup = BeautifulSoup(html, "lxml", parse_only=only_links) 🔹 Benefits
    • Faster parsing
    • Lower memory usage
    • Cleaner output
    8. Mental Model🔹 Think Like This:
    • HTML = giant dataset
    • find/find_all = SQL queries
    • regex = advanced filtering conditions
    • SoupStrainer = pre-filter at ingestion
    Final TakeawayMastering Beautiful Soup is not just about extracting data—it’s about building a filtering system.Once you combine:
    • Structural selection
    • Regex logic
    • Custom filters
    • Performance optimization
    👉 You can extract exactly what you want from any webpage efficiently and at scale.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    18 min
  • Course 40 - Web Scraping with Python | Episode 10: Navigating and Extracting Web Data with Beautiful Soup
    In this lesson, you’ll learn about: how HTML is structured as a tree, how to turn raw pages into navigable data using Beautiful Soup, and how to extract specific elements efficiently1. Understanding the HTML Parse Tree🔹 The Structure of a Web PageEvery web page is a hierarchical tree made of nodes:
    • Root →
    • Children → and
    • Siblings → elements at the same level
    🔹 Key Sections
    • → metadata (title, scripts, styles)
    • → visible content
    👉 Key Insight
    Scraping is really about navigating this tree intelligently2. Turning HTML into Data (Beautiful Soup)🔹 The Core ToolUse Beautiful Soup
    • Converts raw HTML → structured Python object
    • Makes navigation simple and readable
    🔹 Why It’s Powerful
    • Handles messy HTML
    • Supports multiple parsers
    • Easy to search and extract
    3. Choosing the Right Parser🔹 Available ParsersParserStrengthlxmlFast and efficienthtml5libHandles broken HTML🔹 When to Use Each
    • Use lxml → performance
    • Use html5lib → unreliable or malformed pages
    👉 Pro Insight
    Real-world pages are often messy → parser choice matters4. From Request to Parsed Tree🔹 Workflow Overview
    1. Send HTTP request
    2. Receive HTML
    3. Parse with Beautiful Soup
    4. Navigate and extract
    🔹 Example Setupimport requests from bs4 import BeautifulSoup r = requests.get("https://example.com") soup = BeautifulSoup(r.text, "lxml") 5. Extracting Text Content🔹 Headers & Paragraphstitle = soup.h1.string paragraph = soup.p.string 👉 Use Case
    • Blog titles
    • Article content
    • Product descriptions
    6. Extracting Attributes (Links & Images)🔹 Accessing Attributeslink = soup.a["href"] image = soup.img["src"] 👉 What You Can Extract
    • URLs
    • Image sources
    • Metadata
    7. Working with CSS Classes🔹 Finding Elements by Classitems = soup.find_all("div", class_="product") 🔹 Important Note
    • Classes can be multi-valued
    👉 Beautiful Soup handles this intelligently8. Navigating the Tree🔹 Moving Through Nodes
    • .parent
    • .children
    • .next_sibling
    🔹 Examplefor child in soup.body.children: print(child) 👉 Key Skill
    Understanding relationships = better extraction9. Real Extraction Strategy🔹 Step-by-Step Thinking
    1. Inspect HTML
    2. Identify target element
    3. Choose selector
    4. Extract data
    5. Clean output
    10. Common Pitfalls🔹 Things to Watch Out For
    • Missing tags
    • Nested complexity
    • Dynamic content (JavaScript)
    👉 Solution
    • Always verify structure first
    • Use browser DevTools
    11. Mental ModelHTML Page = Tree
    Beautiful Soup = Navigator👉 You are not scraping randomly
    You are walking a structured mapFinal TakeawayMastering Beautiful Soup means mastering how the web is structured.Once you understand the tree, extraction becomes predictable, scalable, and precise—turning messy HTML into clean, usable data.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    19 min
  • Course 40 - Web Scraping with Python | Episode 9: Navigating Requests, Redirects, and Timeouts
    In this lesson, you’ll learn about: how to handle HTTP requests in Python, compare different libraries, manage redirects and errors, and use modern tools like Requests effectively1. The Big Picture: Talking to the Web🔹 What You’re Really DoingWhen working with HTTP in Python, you're:
    • Sending requests
    • Receiving responses
    • Handling edge cases (errors, redirects, timeouts)
    👉 This is the foundation of:
    • Web scraping
    • API integration
    • Automation
    2. HTTP Methods Beyond the Basics🔹 Core Methods RecapMethodPurposeGETRetrieve dataPOSTSend dataPUTUpdate (idempotent)DELETERemove🔹 Advanced MethodsMethodUse CaseHEADGet headers only (no body)OPTIONSDiscover server capabilities👉 Pro Insight
    • HEAD is great for checking if a resource exists without downloading it
    • OPTIONS helps when working with APIs and permissions
    3. Redirect Handling (Critical in Real-World Scraping)🔹 What is a Redirect?A redirect happens when:
    • Server tells you → “Go to another URL”
    🔹 Types of Redirects
    • Safe Redirects
      • GET, HEAD
      • Automatically followed
    • Unsafe Redirects
      • POST, PUT
      • May require confirmation
    🔹 Why It Matters
    • Prevent infinite loops
    • Track where data actually comes from
    • Debug login flows or APIs
    4. URL Anatomy (Using urllib)🔹 Breaking Down a URLExample:https://example.com/products?id=10#reviews PartMeaningSchemehttpsLocationexample.comPath/productsQueryid=10Fragmentreviews🔹 Tool for ThisUse urllibfrom urllib.parse import urlparse parsed = urlparse("https://example.com/products?id=10") print(parsed.scheme, parsed.netloc) 👉 Why It’s Important
    • Helps build clean scrapers
    • Useful for filtering and routing URLs
    5. Error Handling (Making Your Code Bulletproof)🔹 Common ErrorsErrorMeaning403Forbidden (blocked)404Not foundTimeoutServer too slow🔹 Best Practiceimport requests try: r = requests.get("https://example.com", timeout=5) r.raise_for_status() except requests.exceptions.RequestException as e: print("Error:", e) 👉 Key Insight
    Good scrapers don’t just work…
    they fail gracefully6. Comparing Python HTTP Libraries🔹 The Three Main Tools1. Low-Level ControlUse httplib2
    • Fine-grained control
    • More verbose
    2. Built-in OptionUse urllib
    • No installation
    • متوسط التعقيد
    3. Modern Standard ⭐Use Requests
    • Clean syntax
    • Developer-friendly
    • الأكثر استخدامًا
    7. Why Requests is the Go-To Tool🔹 Key Features
    • Automatic POST encoding
    • Easy JSON parsing
    • Built-in timeout support
    🔹 Example: GET Requestimport requests r = requests.get("https://api.example.com/data", timeout=5) data = r.json() print(data) 🔹 Example: POST Requestpayload = {"username": "test", "password": "1234"} r = requests.post("https://api.example.com/login", data=payload) print(r.status_code) 👉 Why Developers Love It
    • Less code
    • More readability
    • Handles complexity internally
    8. Redirect Tracking in Requestsr = requests.get("http://example.com") print(r.url) # Final URL print(r.history) # Redirect chain 👉 Use Case
    • Detect hidden redirects
    • Analyze tracking URLs
    9. Timeouts (Avoid Hanging Programs)🔹 The ProblemWithout timeout:
    • Your script may freeze forever
    🔹 The Solutionrequests.get("https://example.com", timeout=3) 👉 Always set a timeout in production10. Mental ModelHTTP Request Handling =
    Send → Wait → Handle → RecoverFinal TakeawayMastering HTTP in Python isn’t about memorizing libraries—it’s about understanding how to control communication with servers.Once you combine:
    • Proper method usage
    • Smart redirect handling
    • Strong error management
    • And the power of Requests
    👉 You move from basic scripts to production-level data systems.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    22 min

About CyberCode Academy

From the publisher's feed

Welcome to CyberCode Academy — your audio classroom for Programming and Cybersecurity.
🎧 Each course is divided into a series of short, focused episodes that take you from beginner to…