Data Science Tech Brief By HackerNoon

Data Science Tech Brief By HackerNoon

By HackerNoonNewsTech News
Download on the App Store

Data Science Tech Brief By HackerNoon episodes

  • 224 Blog Posts To Learn About Web Scraping

    This story was originally published on HackerNoon at: https://hackernoon.com/224-blog-posts-to-learn-about-web-scraping.


    Learn everything you need to know about Web Scraping via these 224 free HackerNoon blog posts.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #web-scraping, #learn, #learn-web-scraping, and more.


    This story was written by: @learn. Learn more about this writer by checking @learn's about page,
    and for more stories, please visit hackernoon.com.

    56 min
  • Mastering Databricks: A Developer’s Guide to Lakehouse Architecture & PySpark Pipelines

    This story was originally published on HackerNoon at: https://hackernoon.com/mastering-databricks-a-developers-guide-to-lakehouse-architecture-and-pyspark-pipelines.


    A complete developer guide to Databricks and Lakehouse architecture.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #databricks, #data-engineering, #apache-spark, #pyspark, #machine-learning, #architecture, #lakehouse-architecture, #pyspark-pipelines, and more.


    This story was written by: @jagan_489. Learn more about this writer by checking @jagan_489's about page,
    and for more stories, please visit hackernoon.com.


    Whether you're building real-time streaming pipelines or deploying production generative AI applications, this guide breaks down the core architecture, implementation patterns, and code needed to master Databricks.

    5 min
  • 1.89 Seasons: A Baseball Experiment About Hiring and Human Judgment

    This story was originally published on HackerNoon at: https://hackernoon.com/189-seasons-a-baseball-experiment-about-hiring-and-human-judgment.


    A 69-season baseball pilot tests when an individual’s own record becomes more predictive than their reference class, with careful caveats for hiring.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #predictive-modeling, #sports-analytics, #referential-evaluation, #lahman-baseball-database, #reference-class-prediction, #empirical-bayes, #irep, #resume-screening, and more.


    This story was written by: @elodieaishwarya. Learn more about this writer by checking @elodieaishwarya's about page,
    and for more stories, please visit hackernoon.com.


    The article uses public baseball data to test when an individual’s prior record beats reference-class prediction. It argues that hiring may need a similar shift away from static résumé categories, while clearly stating that the baseball number does not directly transfer to hiring.

    12 min
  • Knowledge Graphs: What They Are, Where They Live, and How to Build One

    This story was originally published on HackerNoon at: https://hackernoon.com/knowledge-graphs-what-they-are-where-they-live-and-how-to-build-one.


    A beginner-friendly guide to knowledge graphs covering triples, nodes, edges, graph databases, real-world uses, and Python examples.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #knowledge-graphs, #knowledge-graph, #graph-databases, #python-knowledge-graph, #networkx-tutorial, #knowledge-graph-python, #rdf-triplestore, #graph-data-modeling, and more.


    This story was written by: @subhash1986. Learn more about this writer by checking @subhash1986's about page,
    and for more stories, please visit hackernoon.com.


    A beginner-friendly guide to knowledge graphs covering triples, nodes, edges, graph databases, real-world uses, and Python examples.

    9 min
  • I Built an AI Job Tracker While Job Hunting: The AI Was the Easy Part

    This story was originally published on HackerNoon at: https://hackernoon.com/i-built-an-ai-job-tracker-while-job-hunting-the-ai-was-the-easy-part.


    A backend engineer's honest build story: aggregating 11 UK job-board APIs into one schema, with AI scoring and cover letters on top.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #python-programming, #fastapi, #artificial-intelligence, #job-search, #full-stack-development, #ai-job-tracker, #apis, #side-project, and more.


    This story was written by: @bogdus1k. Learn more about this writer by checking @bogdus1k's about page,
    and for more stories, please visit hackernoon.com.


    I built SearchWork - a tool that searches 11 job boards at once, scores each role against my CV, and generates cover letters. The AI was the easy part; normalising 11 messy job APIs into one schema was the real work.

    10 min
  • AI Agents Are Opening Up, but What About the Data?

    This story was originally published on HackerNoon at: https://hackernoon.com/ai-agents-are-opening-up-but-what-about-the-data.


    A2A and MCP are making AI agents easier to connect, but proprietary data remains harder to move, govern, and secure across enterprise AI systems.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #enterprise-data, #enterprise-ai, #data-interoperability, #data-portability, #ai-agent-permissions, #data-governance, #multi-agent-data-access, #good-company, and more.


    This story was written by: @jonstojanjournalist. Learn more about this writer by checking @jonstojanjournalist's about page,
    and for more stories, please visit hackernoon.com.


    A2A and MCP are making it easier for AI agents built by different providers to communicate and share tools. But agent interoperability doesn't solve the harder data problem. This article explores why proprietary data platforms, duplicated pipelines, fragmented permissions, and multi-agent access controls could limit enterprise AI flexibility even as open protocols make the agent layer more portable.

    9 min
  • Scaling a Kafka Consumer From 4K to 25K Events per Second While Preserving Ordering

    This story was originally published on HackerNoon at: https://hackernoon.com/scaling-a-kafka-consumer-from-4k-to-25k-events-per-second-while-preserving-ordering.


    A production redesign using batching, safe offset commits, and record-level fallback raised Kafka consumer throughput from 4K to 25K without adding partitions.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #apache-kafka, #software-architecture, #distributed-systems, #software-development, #scalability, #performance-optimization, #kafka-consumer, #event-streaming, and more.


    This story was written by: @manjushaguntur. Learn more about this writer by checking @manjushaguntur's about page,
    and for more stories, please visit hackernoon.com.


    A Kafka consumer processing 4,000 events per second had to meet an 8,000 event SLO. Rather than add partitions, we redesigned the consumer around batching, safe offset commits, and record-level fallback. The solution was validated in production at approximately 25,000 events per second while preserving ordering and at-least-once delivery for recoverable events. This article explains how the design handled successful batches, transient failures, and terminal failures without silently losing events or blocking a partition.

    6 min
  • Modeling Valve-Closure Pressure Spikes in Miniature Fluid Systems

    This story was originally published on HackerNoon at: https://hackernoon.com/modeling-valve-closure-pressure-spikes-in-miniature-fluid-systems.


    Estimate liquid-line pressure spikes from fast solenoid-valve closure with a simple model, Python code, limitations, and bench-test guidance.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #python-programming, #hardware-engineering, #iot, #fluid-dynamics, #ivd-analyzers, #pmic, #lab-automation, #solenoid-valves, and more.


    This story was written by: @alexhu-fluidics. Learn more about this writer by checking @alexhu-fluidics's about page,
    and for more stories, please visit hackernoon.com.


    A rigid-column model shows how density, liquid-column length, flow velocity, and valve closure time set the pressure-spike scale. Python makes the screening calculation repeatable, while bench testing and a transient model remain essential near component limits.

    11 min
  • Fivetran vs. Apache SeaTunnel: Managed ELT or Open-Source Control?

    This story was originally published on HackerNoon at: https://hackernoon.com/fivetran-vs-apache-seatunnel-managed-elt-or-open-source-control.


    A successful pipeline doesn’t guarantee trusted data.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-pipeline, #data-science, #data-security, #fivetran-vs-seatunnel, #apache-seatunnel, #fivetran-alternative, #managed-elt, #data-quality, and more.


    This story was written by: @zhoujieguang. Learn more about this writer by checking @zhoujieguang's about page,
    and for more stories, please visit hackernoon.com.


    See how to enable reliable movement, recovery, and schema evolution.

    18 min
  • The Narrative Wars: Tracking Public Web Influence Operations Across the Globe

    This story was originally published on HackerNoon at: https://hackernoon.com/the-narrative-wars-tracking-public-web-influence-operations-across-the-globe.


    NGOs use public web monitoring tools and web data analysis to reveal geopolitical influence operations in Europe and Asia.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #real-world-web-data, #digital-propaganda-networks, #geopolitical-tensions, #ai-monitoring, #information-warfare, #information-integrity, #south-korea-fimi, #hackernoon-top-story, and more.


    This story was written by: @cerniauskas. Learn more about this writer by checking @cerniauskas's about page,
    and for more stories, please visit hackernoon.com.


    Geopolitical conflicts are increasingly shifting to the information domain, requiring civil society organizations to have better tools for monitoring the public web. By analyzing data at scale, teams like FactCheck.LT and Doublethink Lab are able to expose influence operations ranging from large-scale state propaganda to subtle, early-stage inauthentic activities. Ultimately, closing the resource gap between state-backed actors and civil society requires a stronger coalition between technology providers and the public-interest organizations working to defend information integrity.

    7 min

About Data Science Tech Brief By HackerNoon

From the publisher's feed

Learn the latest data science updates in the tech world.