Data Science Tech Brief By HackerNoon

Data Science Tech Brief By HackerNoon

By HackerNoonNewsTech News
Download on the App Store

Data Science Tech Brief By HackerNoon episodes

  • Principal Components Analysis in TypeScript (Part 4): Turning PCA Into Interpretable Factor Analysis

    This story was originally published on HackerNoon at: https://hackernoon.com/principal-components-analysis-in-typescript-part-4-turning-pca-into-interpretable-factor-analysis.


    Remember how PCA collapses data with 100 dimensions into a single dimension, wouldn't it be cool if this dimension were interpretable. Factor Analysis does that
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-analysis, #typescript, #principal-component-analysis, #factor-analysis, #singular-value-decomposition, #interpretable-ai, #dimensionality-reduction, #exploratory-data-analysis, and more.


    This story was written by: @bitanath. Learn more about this writer by checking @bitanath's about page,
    and for more stories, please visit hackernoon.com.


    Now remember how PCA collapses data with 100 dimensions into a single dimension, wouldn't it be cool if this dimension was interpretable. For example, let's say the 100 columns were like stress, smoking frequency, alcohol ml etc etc.. you see where I am going with this, the final dimension would be something like cardiac arrest or premature demise. On that cheery note, let's figure out how PCA can actually be used to label this reduced dimension.

    6 min
  • Data Engineering Teams Need a Different Version of Agile

    This story was originally published on HackerNoon at: https://hackernoon.com/data-engineering-teams-need-a-different-version-of-agile.


    This article explores which Agile practices actually help data engineering teams and which ceremonies often become operational overhead.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-governance, #agile-data-engineering, #data-pipelines, #pipeline-monitoring, #backlog-management, #engineering-management, #pipeline-validation, #data-operations, and more.


    This story was written by: @kuladeepsandra. Learn more about this writer by checking @kuladeepsandra's about page,
    and for more stories, please visit hackernoon.com.


    Agile is useful for data engineering teams when it creates visibility, reduces context switching, and helps teams manage uncertainty. A visible backlog, regular delivery rhythm, and meaningful retrospectives usually help. Story point velocity tracking and status-report standups often become ceremony. The goal is not to “do Agile.” The goal is to create enough structure to prevent shortcuts, surface blockers early, and deliver reliable data work.

    13 min
  • The LLM Veneer: When AI Sounds Smart but Has Nothing Real to Reason Over

    This story was originally published on HackerNoon at: https://hackernoon.com/the-llm-veneer-when-ai-sounds-smart-but-has-nothing-real-to-reason-over.


    When AI sounds smart but has nothing real to reason over. A pet-tech case study in reference frames, longitudinal modeling, and missing data.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-science, #artificial-intelligence, #time-series, #ai-infrastructure, #data-engineering, #pet-tech-ai, #longitudinal-data-modeling, #hackernoon-top-story, and more.


    This story was written by: @elodieaishwarya. Learn more about this writer by checking @elodieaishwarya's about page,
    and for more stories, please visit hackernoon.com.


    Most AI products add a fluent interface before fixing the data model. The result: confident answers over the wrong structure. This is the LLM Veneer. A pet-tech case study in why data architecture matters more than conversational fluency.

    7 min
  • Bad Ingestion Architecture Generates Million Dollar Snowflake and Databricks Bills

    This story was originally published on HackerNoon at: https://hackernoon.com/bad-ingestion-architecture-generates-million-dollar-snowflake-and-databricks-bills.


    Enterprise data platforms often suffer from skyrocketing cloud bills caused not by user queries, but by bad ingestion architecture.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #dataengineering, #cloudcomputing, #finops, #snowflake, #databricks, #data-architecture, #bigdata, #bad-ingestion-architecture, and more.


    This story was written by: @abhilash-tech. Learn more about this writer by checking @abhilash-tech's about page,
    and for more stories, please visit hackernoon.com.


    Enterprise data platforms often suffer from skyrocketing cloud bills caused not by user queries, but by bad ingestion architecture. Issues like the "Small File Problem" from real-time micro-batching, lack of change data capture forcing massive full-table overwrites, and mismatched data clustering keys run up hidden compute charges. By implementing automated file compaction, tiered ingestion routing, and strict incremental data logic, engineers can achieve up to an 80% reduction in compute spend while maintaining high system performance.

    10 min
  • Optimizing Distributed Data Processing for ML at Scale

    This story was originally published on HackerNoon at: https://hackernoon.com/optimizing-distributed-data-processing-for-ml-at-scale.


    A practitioner's guide to ML data pipeline performance: read the query plan first, eliminate shuffle, fix file layout, handle skew, prune columns
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #spark, #pyspark, #machine-learning, #data-engineering, #performance-optimization, #distributed-systems, #distributed-data-processing, #optimizing-distributed-data, and more.


    This story was written by: @seshendranath. Learn more about this writer by checking @seshendranath's about page,
    and for more stories, please visit hackernoon.com.


    Stop tuning knobs on a broken foundation shuffle, file layout, skew, and column pruning do more for ML pipeline performance than any clever algorithm.

    8 min
  • Why Finance Data Quality Needs Rule Engines, Not ML Hype

    This story was originally published on HackerNoon at: https://hackernoon.com/why-finance-data-quality-needs-rule-engines-not-ml-hype.


    Why financial data quality depends less on ML hype and more on rule engines, governance, vendor controls and audit trails that regulators can understand.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-quality, #reference-data, #financial-data, #data-governance, #audit-trail, #data-validation, #regulatory-reporting, #auditability, and more.


    This story was written by: @nithish_6q9kh89. Learn more about this writer by checking @nithish_6q9kh89's about page,
    and for more stories, please visit hackernoon.com.


    Why financial data quality depends less on ML hype and more on rule engines, governance, vendor controls and audit trails that regulators can understand.

    15 min
  • 156 Blog Posts To Learn About Business Intelligence

    This story was originally published on HackerNoon at: https://hackernoon.com/156-blog-posts-to-learn-about-business-intelligence.


    Learn everything you need to know about Business Intelligence via these 156 free HackerNoon blog posts.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #business-intelligence, #learn, #learn-business-intelligence, and more.


    This story was written by: @learn. Learn more about this writer by checking @learn's about page,
    and for more stories, please visit hackernoon.com.

    38 min
  • Why Your Marketplace Scraper Keeps Getting Blocked (And Why It’s Not a Code Problem)

    This story was originally published on HackerNoon at: https://hackernoon.com/why-your-marketplace-scraper-keeps-getting-blocked-and-why-its-not-a-code-problem.


    Marketplace anti-bot systems increasingly score network identity instead of scraper logic, making rotating residential proxies essential infrastructure.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #web-scraping, #ai-web-scraping, #data-marketplace, #marketplace-scraping, #rotating-residential-proxies, #anti-bot-systems, #datacenter-proxies, #good-company, and more.


    This story was written by: @webintelligencehub. Learn more about this writer by checking @webintelligencehub's about page,
    and for more stories, please visit hackernoon.com.


    If your marketplace scraper keeps hitting 403s and CAPTCHAs, the problem isn't your code: it's your IP identity. Datacenter and static IPs fail anti-bot scoring systems. The fix: rotating residential proxies, geo-targeted to your marketplace's locale, with a rotation model matched to your target's session behavior.

    12 min
  • How I Decoded My Apple Watch Metrics: Taking a Look At The Raw Numbers (Part 2)

    This story was originally published on HackerNoon at: https://hackernoon.com/how-i-decoded-my-apple-watch-metrics-taking-a-look-at-the-raw-numbers-part-2.


    Learn how to parse Apple Health XML & GPX files. A technical guide to "streaming" large CDA files and extracting workout kinematics using Python.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-science, #python-notebook, #python, #apple-watch, #apple-health, #prediction-delta, #health-data, #apple-wearable-data, and more.


    This story was written by: @farzon. Learn more about this writer by checking @farzon's about page,
    and for more stories, please visit hackernoon.com.


    Exporting Apple Health data results in massive, messy XML files that are difficult to process. By using a "streaming" parser to filter specific LOINC codes and extracting GPS kinematics from GPX files, I converted 300MB of raw records into clean CSVs. This structured data is now ready to be fed into a custom machine learning model to reverse-engineer VO2 Max.

    4 min
  • Why AI Agents Are Creating a New Kind of Data Engineer

    This story was originally published on HackerNoon at: https://hackernoon.com/why-ai-agents-are-creating-a-new-kind-of-data-engineer.


    The role of data engineers is evolving faster than ever and this is the advent of intelligence engineers who will not only build AI agents but create governance
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-engineering, #ai-agents, #agentic-ai, #intelligence-engineer, #data-pipelines, #etl-automation, #agent-governance, #pipeline-monitoring, and more.


    This story was written by: @engineervarun0012. Learn more about this writer by checking @engineervarun0012's about page,
    and for more stories, please visit hackernoon.com.


    The role of data engineers is evolving faster than ever and this is the advent of intelligence engineers who will not only build AI agents but create governance around them along with strict guardrails.The blog sheds light on the next generation data leader

    14 min

About Data Science Tech Brief By HackerNoon

From the publisher's feed

Learn the latest data science updates in the tech world.