CyberCode Academy

CyberCode Academy

By CyberCode AcademyTechnologyEducationCourses
Download on the App Store

CyberCode Academy episodes

  • Course 40 - Web Scraping with Python | Episode 38: Scraping Dynamic Premier League Stats and News with Selenium and BeautifulSoup
    This episode is a practical end-to-end example of the Selenium + Beautiful Soup hybrid scraping pattern, applied to a real sports data use case (Premier League player pages).⚽ Goal of the ProjectScrape structured data about Wayne Rooney from a dynamic football website, including:
    • News headlines
    • Career statistics
    • Player profile information
    This is a classic case where:
    • Content is JavaScript-rendered (dynamic)
    • Page structure changes after interaction
    • Static scraping alone would fail
    🧭 1. Phase One — Selenium (Browser Automation)Selenium is used here as a real user simulator.What it does:
    • Opens the Premier League website
    • Navigates to the player section
    • Uses search to find Wayne Rooney
    • Clicks through profile tabs (news, stats, etc.)
    Why Selenium is required:Because the site:
    • Loads content dynamically via JavaScript
    • Requires user interaction (clicks, navigation)
    • Doesn’t expose all data in initial HTML
    ⏳ Critical Concept: WaitsThe episode emphasizes two types of synchronization:🔹 Implicit Wait
    • Global delay applied to all element searches
    • Selenium keeps retrying until element appears
    🔹 Explicit Wait
    • Waits for specific conditions:
      • element becomes clickable
      • element is visible
      • DOM finishes loading
    👉 This is essential because dynamic pages load unpredictably.📥 2. Capture the Final Rendered PageAfter navigation:
    • Selenium grabs the final DOM using page_source
    At this point:You have the fully rendered browser state, including JavaScript-generated content.🧪 3. Phase Two — Beautiful Soup (Fast Parsing)Now Selenium steps out, and Beautiful Soup takes over.Why switch tools?Because:
    • Selenium is slow for repeated extraction
    • Beautiful Soup works on local HTML memory
    • Parsing becomes significantly faster
    🧠 Extraction ProcessOnce HTML is passed into BS4:📰 Headlines extraction
    • Locate or structured containers
    • Extract text cleanly from tags
    📊 Stats extraction
    • Target stat containers
    • Read:
      • labels from attributes
      • numeric values from text nodes
    🔄 Key Design InsightThis architecture is:Selenium = navigation engine
    Beautiful Soup = data extraction engineThey are not competing tools — they are complementary.📌 Why this approach scalesThe episode highlights a key idea:Player-agnostic designOnce built, the same script can:
    • scrape any player profile
    • reuse the same selectors
    • scale across hundreds of pages
    🚀 Extension Path (Important)The workflow naturally evolves into:1. Data structuring
    • Convert scraped data into tables using Pandas
    2. Analytics
    • Compare players statistically
    • Track performance over time
    3. ML applications
    • performance prediction
    • sentiment analysis on news articles
    • scouting models
    🧠 Core TakeawayThis is a real production scraping pattern:
    1. Selenium → reach the data (dynamic navigation)
    2. page_source → freeze the state
    3. Beautiful Soup → extract efficiently
    4. Pandas/ML → analyze downstream


    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    21 min
  • Course 40 - Web Scraping with Python | Episode 37: Integrating Selenium and Beautiful Soup
    This episode is basically about building a hybrid scraping pipeline where each tool does what it’s best at instead of forcing one tool to do everything.🧩 Core Idea: Split the Problem in TwoModern scraping usually has two phases:
    1. Browser simulation (Selenium)
    2. HTML parsing (Beautiful Soup)
    The key insight:Selenium is for interacting with the page, not for extracting data at scale.🧠 1. Beautiful Soup — the fast “data reader”Beautiful Soup is introduced as the lightweight parsing engine.What it does well:
    • Parses HTML / XML into a structured tree
    • Handles broken or messy markup automatically
    • Works with different parsers (especially LXML for speed)
    Core object types:
    • Tag → HTML elements like ,
    • NavigableString → text inside tags
    Comment → HTML commentsBeautifulSoup object → full document containerWhy it matters:It turns raw HTML into something you can query like Python objects instead of scraping strings manually.⚡ 2. Why not just use Selenium for everything?This is the key performance argument:Selenium drawbacks:Every action goes through HTTP (JSON Wire Protocol)Each .find_element() is relatively slowRepeated DOM queries become expensiveSo:Selenium is great for interaction, but inefficient for extraction.🔁 3. The Hybrid Strategy (Best Practice)This is the actual workflow the episode teaches:Step 1 — Use Selenium for dynamic actionsYou use Selenium to:open the pageclick buttonsscrollfill formswait for JS-rendered contentStep 2 — Capture final HTMLOnce the page is fully loaded:grab page_source from SeleniumStep 3 — Switch to Beautiful Souppass HTML into Beautiful Soupparse locally in memory (fast)🚀 Why this works so wellBecause it separates responsibilities:ToolRoleSeleniumbrowser control (slow, interactive)Beautiful Soupdata extraction (fast, local parsing)🧩 Mental ModelThink of it like this:Selenium = a human controlling a browserBeautiful Soup = a machine reading the saved pageSo instead of repeatedly asking the browser for data, you:load once → extract locally at high speed🔥 Key TakeawayThe real optimization is not “use better selectors” — it’s:“stop scraping live DOM repeatedly and instead parse a snapshot of it”

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    19 min
  • Course 40 - Web Scraping with Python | Episode 36: Comprehensive Element Locating and Advanced Webpage Navigation
    This tutorial series is basically showing how Selenium moves from “clicking elements” into real-world browser automation, where pages are messy, slow, and full of UI traps.🧭 1. Core Setup + Basic NavigationEverything starts with controlling the browser:
    • ChromeDriver setup
      • Acts as the bridge between Python and Chrome
    • driver.get(url)
      • Opens a webpage inside the automated browser session
    Once the page loads, the first interactions usually target simple inputs like search bars.✍️ Basic interaction flowTypical steps:
    • locate input field
    • clear existing text
    • send new text using keyboard input
    • submit or trigger search
    This is the foundation of all automation flows.🎯 2. Element Location (the real core skill)The series reinforces multiple ways to find elements depending on page structure:🆔 ID (best case)
    • fastest and most stable
    🏷️ Name
    • common in forms (login, search, signup)
    🎨 CSS Selectors
    • uses class-based targeting
    • flexible and widely used in real projects
    🧭 XPath
    • most powerful option
    • works even when HTML is messy or missing IDs/classes
    🔗 Links
    • exact link text
    • partial link text
      Useful for navigation between pages.
    ⚙️ 3. Handling Real Web Behavior (Dynamic Pages)This is where Selenium becomes “real automation” instead of simple scripting.⏳ WebDriverWait (critical concept)Modern websites load content asynchronously, so elements might not exist immediately.Instead of failing instantly, Selenium can:
    • wait until element appears
    • wait until element becomes clickable
    • pause execution until condition is met
    This prevents most “element not found” errors.🧾 4. Complex Form HandlingForms are not just text inputs — they include dropdowns, validations, and dynamic fields.📋 Dropdown strategyInstead of selecting blindly:
    • collect all elements
    • loop through them
    • match desired value
    • click selection
    This makes automation resilient when UI order changes.🧱 5. Handling Real UI Complexity🪟 iframes
    • embedded pages inside pages
    • Selenium cannot access them directly
    • must switch context before interacting
    ⚠️ Pop-ups / Alerts / PromptsYou can:
    • accept (OK)
    • dismiss (Cancel)
    • read alert text
    These often block automation flows if not handled.🧠 Key InsightThis module is really about this transition:from “clicking elements” → to “controlling unpredictable browser behavior”Because real websites are not static:
    • they load slowly
    • they restructure DOM dynamically
    • they interrupt workflows with modals and alerts
    ⚡ Summary Mental ModelThink of Selenium automation like this:
    1. Open browser
    2. Wait for page stability
    3. Find elements reliably (ID/CSS/XPath)
    4. Interact carefully (click/type/select)
    5. Handle interruptions (alerts, iframes, delays)
    6. Repeat across navigation flows


    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    24 min
  • Course 40 - Web Scraping with Python | Episode 35: Locating Dynamic Elements with Selenium and Python
    This module is basically about the core skill in Selenium automation: reliably finding the right element on a page that keeps changing.🧩 What “locating elements” really meansIn Selenium, everything you interact with is a web element, such as:
    • buttons
    • input fields
    • links
    • images
    • hidden UI components used by JavaScript
    Modern web apps are often dynamic, meaning:
    • IDs change on every refresh
    • classes are generated randomly
    • elements appear/disappear after AJAX calls
    So the real challenge is not clicking elements — it’s finding them consistently.⚠️ The Dynamic Web ProblemUnlike static HTML pages, modern JavaScript-heavy sites:
    • regenerate DOM elements constantly
    • load content asynchronously
    • modify attributes at runtime
    That’s why a locator that works once may fail on the next page load.🔍 The 8 Ways to Locate ElementsSelenium gives multiple strategies. Each has a different “strength level”.1. 🆔 ID (Best option)
    • Fastest and most reliable
    • Must be unique
    • Breaks only if developers change structure
    2. 🏷️ Name
    • Works when ID is missing
    • Common in forms
    3. 🔗 Link Text
    • Matches full hyperlink text
    • Example: “Login”
    Partial Link Text
    • Matches part of a link
    • More flexible but less precise
    4. 🎯 CSS Selectors
    • Very powerful and widely used
    • Uses patterns like:
      • classes
      • hierarchy
      • attributes
    Example idea:“div.container button.primary”5. 🧱 Tag Name
    • Finds elements like , ,
    • Usually returns many results
    6. 🎨 Class NameUses CSS class attributeRisk: many elements share same class7. 🧭 XPath (Most powerful)Can navigate DOM like a treeWorks even when structure is messyKey advantage:supports relative pathscan search based on text, attributes, hierarchy⚙️ Two Core Retrieval Methods🔹 find_elementreturns single elementthrows error if not foundbest when you expect exactly one match🔹 find_elementsreturns list of elementssafe (no exception if empty)you can loop or index results🧠 Practical InsightThe real decision rule is:Use ID firstIf not available → CSS SelectorIf structure is complex → XPathIf multiple results → find_elements⚡ Key TakeawayThis module is really teaching one idea:Selenium automation fails not because of actions, but because of bad element selection strategiesSo robust scraping depends on:choosing stable attributesavoiding fragile selectorshandling dynamic DOM changes

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    21 min
  • Course 40 - Web Scraping with Python | Episode 34: Architecture, Setup, and Basic Web Automation
    This episode focuses on how Selenium WebDriver actually works under the hood, and then walks into the practical setup and first automation steps.🧠 Selenium WebDriver ArchitectureSelenium WebDriver is designed to control browsers as realistically as possible, which is why it uses a multi-layer architecture instead of direct code-to-browser control.🧩 1. Language BindingsThese are client libraries that let you write automation scripts in different languages:
    • Python
    • Java
    • JavaScript
    • C#
    They translate your code into commands WebDriver can understand.🌐 2. JSON Wire Protocol (or W3C WebDriver Protocol)This is the communication layer.
    • Your script sends HTTP requests
    • Commands are encoded as JSON payloads
    • These requests are sent to the browser driver
    Think of it as:“Selenium speaking HTTP to the browser”🧭 3. Browser DriversEach browser has its own driver:
    • Chrome → ChromeDriver
    • Firefox → GeckoDriver
    Their job is to:
    • receive commands
    • translate them into browser-native actions
    🖥️ 4. Real BrowserFinally, the driver controls the actual browser:
    • opens pages
    • clicks elements
    • executes JavaScript
    • renders content
    ⚙️ How Execution FlowsA Selenium action follows this chain:Your Python code → Selenium library → HTTP request → Browser Driver → BrowserThis layered design is what allows cross-browser automation.🛠️ Environment Setup OverviewThe episode walks through setting up a working Selenium environment:📦 Install core libraries
    • Selenium (automation engine)
    • BeautifulSoup (optional parsing tool)
    🌐 Install browser driver
    • Must match your browser version exactly
    • Example: Chrome version ↔ ChromeDriver version
    📓 Optional tools
    • Jupyter Notebook for interactive testing
    • Useful for debugging selectors step-by-step
    🚀 Basic WebDriver UsageOnce setup is complete, the workflow becomes:1. Start browser instance
    • Launch Chrome/Firefox via WebDriver
    2. Navigate to a page
    • Open a URL like a normal user
    3. Perform actions
    • click
    • scroll
    • input text
    • extract elements
    4. Close browser
    • clean shutdown of session
    🧊 Headless BrowsingA key optimization introduced is headless mode.What it means:
    • Browser runs without UI
    • No visible window opens
    Why it matters:
    • faster execution
    • lower memory usage
    • ideal for servers and automation pipelines
    🧠 Key InsightThe main idea of this episode is:Selenium is not just a scraping tool — it's a remote control system for real browsersThat’s why it can handle:
    • JavaScript-rendered content
    • user interactions
    • dynamic page updates


    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    20 min
  • Course 40 - Web Scraping with Python | Episode 33: Foundations of Scraping Dynamic Webpages with Python and Selenium
    This episode is essentially a setup guide for moving from simple HTTP-based scraping to full browser automation using Selenium, especially for websites where content is rendered or modified by JavaScript.🌐 Web Scraping vs Dynamic Web Pages🧾 What “web scraping” means hereWeb scraping is framed as:Converting web page content into structured data for analysisBut the key challenge is that not all content is immediately visible in HTML.🧱 Static vs Dynamic Content📄 Static content
    • Same HTML for every user
    • Can be scraped with tools like Requests or BeautifulSoup
    • No JavaScript dependency
    ⚡ Dynamic content
    • Changes based on:
      • user interaction
      • time
      • location
      • JavaScript execution
    • Often not present in raw HTML
    • Requires browser simulation to access
    🤖 Why Selenium is NeededTraditional scrapers only download HTML.But modern websites:
    • render content with JavaScript
    • load data after page load
    • require clicks/scrolling to reveal content
    👉 Selenium solves this by controlling a real browser.🧰 Selenium OverviewSelenium is described as an automation framework for browsers, not just a scraping tool.It allows you to:
    • open web pages
    • click buttons
    • scroll pages
    • fill forms
    • simulate real users
    🧩 Core Selenium Components1. 🧪 Selenium IDE
    • Record & playback tool
    • Used for quick prototyping
    • No coding required
    2. 🧬 Selenium RC (Legacy)
    • First generation framework
    • Allowed multi-language test scripts
    • Now largely obsolete
    3. 🧭 Selenium WebDriver (Main tool)This is the core engine used in real projectsIt:
    • directly controls the browser
    • executes user-like actions
    • interacts with page elements
    👉 This is the most important part for scraping dynamic sites4. 🌐 Selenium Grid
    • Enables parallel execution
    • Runs tests across multiple machines/browsers
    • Used for scaling automation
    ⚙️ Prerequisites for Using SeleniumBefore practical usage, you need:
    • Python basics
    • HTML/CSS understanding
    • Browser driver setup (ChromeDriver / GeckoDriver conceptually)
    • Ability to inspect web elements
    🚀 What Selenium Enables in ScrapingWith Selenium WebDriver, you can:
    • load JavaScript-heavy pages
    • wait for content to appear
    • interact with UI elements
    • extract final rendered DOM
    This is crucial for modern websites like:
    • dashboards
    • social media pages
    • e-commerce filters
    • infinite scroll pages
    🧠 Key InsightThe main takeaway is:Traditional scrapers read HTML. Selenium scrapes the rendered browser state.That difference is what makes it powerful for dynamic content.

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    21 min
  • Course 40 - Web Scraping with Python | Episode 32: Native Data Storage and Implementation
    This episode is about removing custom storage code from your Scrapy project and replacing it with Scrapy’s built-in Feed Export system, which turns scraping into a fully configurable data export pipeline.📤 Scrapy Feed Exporters (Automated Data Storage)🧠 Core IdeaInstead of manually writing data to files or databases, Scrapy can automatically export scraped items using:Feed Exporters = built-in serialization + storage systemThey handle:
    • formatting
    • writing
    • destination management
    📊 1. Supported Output FormatsScrapy can serialize scraped data into multiple formats:🧾 File formats
    • JSON → full structured export
    • JSON Lines (JSONL) → streaming-friendly format
    • CSV → spreadsheet-ready format
    • XML → hierarchical structured output
    Each format is useful depending on downstream usage:
    • JSON → APIs & apps
    • CSV → Excel / analytics
    • XML → structured integrations
    • JSONL → big data pipelines
    🌍 2. Storage BackendsFeed exporters are not limited to local files.They can write directly to:
    • 💻 Local filesystem
    • 📡 FTP servers
    • ☁️ Amazon S3 (cloud storage)
    This makes Scrapy suitable for:enterprise-level data pipelines without extra storage code⚙️ 3. Pipeline + Export IntegrationA key concept in this episode is the separation of concerns:🔹 Pipelines (data filtering layer)Used to:
    • remove unwanted items
    • enforce business rules
    • clean or block data
    Example:
    • drop books above a certain price
    • filter invalid entries
    🔹 Feed Exporters (storage layer)Used to:
    • take final cleaned items
    • serialize them
    • write them to destination
    🧪 4. Configuration-Driven DesignInstead of writing export logic in code, everything is moved into:🛠️ settings.pyYou define:
    • output format
    • output destination (URI)
    • export behavior
    Example conceptually:FEEDS: output.json: format: json encoding: utf8 🔄 5. Full Data FlowSpider ↓ Item Extraction ↓ Pipelines (filter + clean) ↓ Feed Exporter (serialize) ↓ Storage (file / S3 / FTP) 🧪 6. Practical Demo InsightThe episode’s demo reinforces:✔ Filtering firstItems are removed before export via pipelines.✔ No manual savingNo open() or file handling needed.✔ Automatic export generationScrapy generates:
    • JSON output
    • XML output
    • structured datasets
    🧠 Key TakeawayThe main idea is:Scrapy becomes a configuration-driven data exporter, not just a scraper.You define:
    • what to extract (spider)
    • what to keep (pipelines)
    • where to store it (feed exporters)
    Everything else is automated.🚀 Big PictureThis module completes the Scrapy data pipeline:StageResponsibilitySpiderExtract dataPipelineClean/filter dataFeed ExporterSerialize + store data

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    18 min
  • Course 40 - Web Scraping with Python | Episode 31: From Item Loaders to Pipelines
    This episode is essentially about turning Scrapy from “just a scraper” into a full data processing system, where extraction, cleaning, validation, and storage are all structured and automated.🕷️ Scrapy Data Population & Processing Pipeline1. 📦 Item Loaders (Structured Data Population)Item Loaders are the layer between raw scraped HTML and structured Scrapy Items.Instead of manually assigning fields, you feed data through controlled methods:🔹 Core methods
    • add_xpath()
    • add_css()
    • add_value()
    These methods:
    • collect raw extracted values
    • pass them through processors automatically
    • build a clean final item via load_item()
    💡 Why this mattersInstead of:
    • messy manual parsing
    • scattered cleaning logic
    You get:A single controlled pipeline for building structured objects🔄 Item Loader FlowResponse HTML ↓ add_xpath / add_css / add_value ↓ Input Processors (cleaning + normalization) ↓ Item Fields (structured data) ↓ load_item() ⚙️ 2. Item Pipelines (Post-Extraction Processing Layer)Item Pipelines operate after scraping, acting like a processing conveyor belt.Each pipeline class can:
    • modify data
    • validate data
    • reject invalid items
    • store data
    🔹 Common Pipeline Responsibilities🧹 Data Cleaning
    • remove unwanted characters
    • normalize formats
    • fix inconsistent values
    ✅ Validation
    • check price formats
    • validate emails or URLs
    • ensure required fields exist
    🚫 Filtering
    • drop invalid or unwanted items
    • block duplicates
    • filter based on business rules
    💾 Storage
    • save to database
    • export to JSON / CSV
    • push into APIs
    📚 3. Practical Example: Book Scraping SystemThe episode demonstrates a real workflow using a book website.🔹 Data Transformation ExampleMapCompose usageUsed to transform raw fields like:
    • image URLs → full valid URLs
    • book links → normalized links
    • text cleanup (whitespace, symbols)
    🔹 Custom Pipeline LogicExample rule:“Flag or drop books where price > threshold”So the pipeline can:
    • mark expensive books
    • exclude them entirely
    • or route them differently
    🔹 Pipeline OrderingScrapy allows multiple pipelines:You define execution order in settings:Item Pipeline Order: 1. Cleaning Pipeline 2. Validation Pipeline 3. Filtering Pipeline 4. Storage Pipeline This ensures:Data always flows in a predictable transformation sequence🧠 Key Concept of the EpisodeThe main idea is:Scrapy is not a scraper — it is a data engineering pipeline frameworkYou are not just collecting data, you are:
    • structuring it (Item Loaders)
    • refining it (Processors)
    • validating it (Pipelines)
    • and storing it (Final output layer)
    🧩 Mental ModelLayerPurposeItem LoadersBuild structured itemsProcessorsClean + normalize fieldsPipelinesValidate + transform + storeSettingsControl execution order🚀 Big Picture InsightThis episode shows the shift from:❌ “scrape → print data”to:✅ “scrape → structure → clean → validate → store → scale”

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    24 min
  • Course 40 - Web Scraping with Python | Episode 30: Controlling URL Paths and Processing Scraped Data
    This episode is really about controlling Scrapy’s crawl scope and shaping data as it moves through the pipeline, so you’re not just collecting data—you’re actively engineering what gets collected and how it looks.🕷️ Scrapy Crawl Control & Data Processing Pipeline1. 🎯 URL Path Control (Allow / Deny Rules)In Scrapy, crawl behavior is tightly controlled using rule-based filtering, often inside spiders like CrawlSpider.🔹 Allow rules
    • Define what URLs the spider is allowed to follow
    • Typically based on regex patterns
    • Used to target specific sections of a site (e.g., product pages)
    🔹 Deny rules
    • Explicitly block unwanted paths
    • Useful for excluding:
      • irrelevant categories
      • admin pages
      • unwanted content types
    Example use cases:
    • Allow: /products/.*
    • Deny: /category/crime/.*, /adult/.*
    Key idea:You are shaping the crawler’s “attention span” using URL patterns.⚙️ 2. Data Processing Pipeline (Item Loaders)Once Scrapy extracts raw HTML data, it passes through a structured transformation system.This is where Item Loaders + Processors come in.🔄 Input vs Output Processors📥 Input Processors
    • Run immediately after extraction
    • Clean or normalize raw scraped values
    • Example: stripping whitespace, converting formats
    📤 Output Processors
    • Run after all values are collected
    • Produce final cleaned field value
    🧠 3. Built-in Processor ToolsScrapy provides reusable functions to transform scraped data efficiently:🔹 MapComposeApplies functions to every item in a list.Example use:
    • strip spaces
    • convert strings to integers
    • normalize URLs
    👉 Think of it as:“run this function on every extracted piece of data”🔹 JoinCombines multiple values into a single string.Example:["New", "York"] → "New York" Used when:
    • HTML splits text into multiple nodes
    • You want a single clean field
    🔹 TakeFirstReturns:
    • the first non-null value from a list
    Useful because:
    • Scrapy often returns multiple matches
    • You usually only want one final value
    🔗 4. Full Data Flow (Important Concept)This is the critical architecture idea in the episode:HTML Response ↓ Selectors (XPath / CSS) ↓ Item Loader ↓ Input Processors (cleaning stage 1) ↓ Output Processors (final formatting) ↓ Items ↓ Item Pipelines (storage / DB / export) 🧠 Core Insight of the EpisodeThe key idea is:Scrapy is not just scraping data — it is a data transformation pipeline systemYou don’t just extract data…
    You control how messy web data becomes structured business intelligence.📌 Mental ModelComponentPurposeAllow / Deny rulesControl crawl scopeInput ProcessorsClean raw extractionOutput ProcessorsFinal formattingMapComposeTransform listsJoinMerge textTakeFirstReduce noise

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    21 min
  • Course 40 - Web Scraping with Python | Episode 29: From Feed and Sitemap Spiders to CrawlSpider Demos
    This episode is really about choosing between manual control and automated crawling logic inside Scrapy, and understanding how specialized spider classes change your level of control.Here’s the structured breakdown:🕷️ Scrapy Spider Types — Practical Comparison & Feed Spiders1. Feed-Based Spiders (Structured Data Sources)These spiders are not designed for HTML pages — they target pre-structured data formats.📄 XMLFeedSpider ScrapyPurpose:Extract structured data from XML feeds.Key concept:
    • Works by iterating through XML nodes
    • Uses itertag to define which tag to extract
    • Uses iterator mode (itnodes) for performance
    Behavior:Instead of parsing a full page, it streams through XML elements one by one.📊 CSVFeedSpider ScrapyPurpose:Scrape structured CSV files directly.Key features:
    • Custom delimiters (, ; \t)
    • Configurable quote characters
    • Header mapping → fields become item keys
    Behavior:Each row becomes a structured item automatically.2. SitemapSpider (Automated URL Discovery)SitemapSpider ScrapyPurpose:Crawl websites using their sitemap instead of link discovery.How it works:
    • Reads sitemap.xml
    • Extracts all URLs listed
    • Filters URLs using:
      • regex rules
      • callback mapping rules
    Advantage:No need to manually discover or follow links.⚔️ 3. scrapy.Spider vs CrawlSpider (Core Comparison)🧱 A. scrapy.Spider (Manual Control)Behavior:
    • You define:
      • start_urls
      • parse() logic
      • pagination logic manually
    What you control:
    • Every request
    • Every page transition
    • Every extraction step
    Example characteristics:
    • CSS selectors used explicitly
    • Must manually follow “next page” links
    • Full control over flow
    Key idea:You are writing the crawling engine logic yourself.🤖 B. CrawlSpider (Automated Crawling)Behavior:
    • Uses Rules + LinkExtractors
    • Automatically follows links
    What it does for you:
    • Finds links automatically
    • Filters them using regex or CSS rules
    • Calls callbacks automatically
    Scope:
    • Much broader by default
    • Can crawl entire domains unless restricted
    Key idea:You define rules — Scrapy handles navigation.🔄 4. Real Demo Insight (Quotes Scraping Example)scrapy.Spider behavior:
    • Manually extract data
    • Manually handle pagination
    • Pages may finish in non-sequential order (async execution)
    CrawlSpider behavior:
    • Automatically follows links
    • Less manual parsing logic
    • More scalable for large websites
    🧠 Core Concept of the EpisodeThe real takeaway is:scrapy.Spider = precision control
    CrawlSpider = autonomous exploration📌 Mental ModelTypeStrengthWeaknessscrapy.SpiderFull controlMore codeCrawlSpiderAutomationLess fine-grained controlSitemapSpiderFast discoveryDepends on sitemapXML/CSV SpidersStructured feedsLimited flexibility

    You can listen and download our episodes for free on more than 10 different platforms:
    https://linktr.ee/cybercode_academy

    Get the books of the CyberCode Academy podcast:
    📚 The book series: 416 lessons, 12 books, hundreds of original diagrams: security, networking and programming.
    https://cybercodeacademy.gumroad.com/
    20 min

About CyberCode Academy

From the publisher's feed

Welcome to CyberCode Academy — your audio classroom for Programming and Cybersecurity.
🎧 Each course is divided into a series of short, focused episodes that take you from beginner to…