Data Science Tech Brief By HackerNoon

Data Science Tech Brief By HackerNoon

By HackerNoonNewsTech News
Download on the App Store

Data Science Tech Brief By HackerNoon episodes

  • How to Merge Multiple Excel Files Using Python

    This story was originally published on HackerNoon at: https://hackernoon.com/how-to-merge-multiple-excel-files-using-python.


    Unlock the power of Python in your data management journey! Learn how to effortlessly merge multiple Excel files into one cohesive excel.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-analysis, #merge-excel-files, #python-for-excel, #excel-tutorial, #excel-hacks, #excel-for-data-analysis, #merge-excel-files-with-python, and more.


    This story was written by: @luca1iu. Learn more about this writer by checking @luca1iu's about page,
    and for more stories, please visit hackernoon.com.


    Tired of wrestling with multiple Excel files? Python comes to the rescue! Explore the simplicity of merging Excel files using Python's pandas and os modules. The provided code effortlessly navigates through files in a specified folder, combining them into a structured 'merged_excel.xlsx' file. Whether you're dealing with straightforward data merging or nuanced scenarios like date-driven filenames, Python offers a powerful solution. Elevate your data management game and transform chaos into order with this comprehensive guide.

    3 min
  • Anomaly Detection with Privileged Information—Part 2

    This story was originally published on HackerNoon at: https://hackernoon.com/anomaly-detection-with-privileged-informationpart-2.


    Learn how SVDD encapsulates datasets within hyperspheres, and discover how SVDD+ leverages privileged information to optimize training.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #anomaly-detection, #machine-learning, #svdd+, #cvxopt-library, #oneclasssvm, #malware-detection, #quadratic-optimization, #svdd+-vs-oneclasssvm, and more.


    This story was written by: @dsmoliakov. Learn more about this writer by checking @dsmoliakov's about page,
    and for more stories, please visit hackernoon.com.


    Explore the intricacies of implementing Support Vector Data Description (SVDD) and its enhanced version, SVDD+, for anomaly detection. Understand how SVDD encapsulates datasets and learn how SVDD+ integrates privileged information in the training phase. Dive into the quadratic optimization process of SVDD+ with a step-by-step guide and insights into the underlying math. Compare SVDD+ with OneClassSVM for anomaly detection and discover the performance on the kdd99 dataset. Gain practical implementation insights using the cvxopt library and navigate the world of anomaly detection with and without privileged information.

    8 min
  • 5 Benefits of Dashboard Software for Business Intelligence in 2024

    This story was originally published on HackerNoon at: https://hackernoon.com/5-benefits-of-dashboard-software-for-business-intelligence-in-2024.


    Discover how dashboard software revolutionizes businesses—enhancing efficiency, empowering decisions, and driving cost-effective insights in 2024.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-visualization, #data-visualization-tools, #dashboard, #exploratory-data-analysis, #big-data-analytics, #data-science, #data-insights, #data-driven-decision-making, and more.


    This story was written by: @davidkostya. Learn more about this writer by checking @davidkostya's about page,
    and for more stories, please visit hackernoon.com.

    10 min
  • Anomaly Detection with Privileged Information—Part 1

    This story was originally published on HackerNoon at: https://hackernoon.com/anomaly-detection-with-privileged-informationpart-1.


    Delve into the world of anomaly detection with Support Vector Data Description (SVDD+).
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #anomaly-detection, #original-svdd, #svdd, #svdd+-theory, #machine-learning, #support-vector-data, #malware-detection, #hackernoon-top-story, #hackernoon-es, #hackernoon-hi, #hackernoon-zh, #hackernoon-fr, #hackernoon-bn, #hackernoon-ru, #hackernoon-vi, #hackernoon-pt, #hackernoon-ja, #hackernoon-de, #hackernoon-ko, #hackernoon-tr, and more.


    This story was written by: @dsmoliakov. Learn more about this writer by checking @dsmoliakov's about page,
    and for more stories, please visit hackernoon.com.


    Explore the advanced realm of anomaly detection through Support Vector Data Description (SVDD+). Uncover how the inclusion of privileged information during training enhances classification accuracy. Learn about the modified SVDD+ algorithm, its quadratic optimization task, and its applications in medical anomaly detection and malware analysis. Elevate your understanding of anomaly detection with SVDD+.

    8 min
  • Python Challenge Anyone?

    This story was originally published on HackerNoon at: https://hackernoon.com/python-challenge-anyone.


    Explore a Python exercise that addresses missing data gaps in the JustJoin.it job market dataset fetched using AWS Lambda.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #python-programming, #python-exercises, #python-tutorials, #python-coding-challenges, #python-practice-questions, #learn-python-programming, #programming-challenges, #python-developers, and more.


    This story was written by: @jszafran. Learn more about this writer by checking @jszafran's about page,
    and for more stories, please visit hackernoon.com.


    The Python exercise focuses on identifying missing dates in the JustJoin.it job market dataset fetched via AWS Lambda. The solution utilizes sets, date manipulation, and file traversal to pinpoint the gaps, ensuring a comprehensive and reliable dataset.

    5 min
  • Modern World Problems of ML in Production and How to Cope With Them

    This story was originally published on HackerNoon at: https://hackernoon.com/modern-world-problems-of-ml-in-production-and-how-to-cope-with-them.


    Discover the real-world challenges of deploying machine learning models and explore practical solutions.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #ml-model, #ml-in-production, #model-drift, #concept-drift, #model-drift-detection, #problems-of-ml-in-production, #model-interpretability, #ml-model-data-privacy, and more.


    This story was written by: @viktoriaerokhina. Learn more about this writer by checking @viktoriaerokhina's about page,
    and for more stories, please visit hackernoon.com.


    Deploying machine learning models in real-world scenarios brings challenges like model drift, scalability issues, interpretability concerns, data privacy, and the need for continuous integration/deployment (CI/CD) pipelines. Solutions involve monitoring and retraining for model drift, optimizing model architectures, leveraging hardware accelerators, and implementing explainable AI for model interpretability. Maintaining data privacy involves techniques like differential privacy and federated learning. Establishing robust CI/CD pipelines is crucial, with tools like MLflow and Kubeflow aiding in the process. Real-world examples from companies like Amazon, MobiDev, Citibank, Google, and Netflix illustrate the practical application of these solutions, emphasizing the evolving nature of machine learning challenges and solutions.

    12 min
  • A Comprehensive Comparison of Prometheus and Grafana in 2023

    This story was originally published on HackerNoon at: https://hackernoon.com/a-comprehensive-comparison-of-prometheus-and-grafana-in-2023.


    Explore a comprehensive assessment of Prometheus vs. Grafana - their offerings, where they overlap, and how they differ.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-observability, #observability, #grafana-and-prometheus, #grafana, #prometheus, #troubleshooting, #observability-tools, #data-observability-tools, and more.


    This story was written by: @vivekbadani77. Learn more about this writer by checking @vivekbadani77's about page,
    and for more stories, please visit hackernoon.com.


    Prometheus specializes in monitoring and efficient time-series data storage, while Grafana excels in advanced visualization, handling metrics, logs, and traces for comprehensive observability. Both tools integrate well but serve distinct, complementary functions in the observability ecosystem.

    16 min
  • How Big Data in IoV Analytics Prevents Accidents

    This story was originally published on HackerNoon at: https://hackernoon.com/how-big-data-in-internet-of-vehicles-analytics-prevents-accidents.


    As electric vehicles surge, the Internet of Vehicles (IoV) is reshaping the automotive landscape.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #big-data, #iov, #internet-of-vehicles, #iot, #data-science, #data-analytics, #iov-data, #iov-data-processing, and more.


    This story was written by: @zakilu. Learn more about this writer by checking @zakilu's about page,
    and for more stories, please visit hackernoon.com.


    The Internet of Vehicles (IoV) is generating an unprecedented volume of data, especially with electric vehicles dominating the auto market. This post delves into the critical role of Controller Area Network (CAN) within each vehicle, managing real-time feedback, data sharing, and network troubleshooting. With a car manufacturer handling 100 billion CAN data pieces daily, the transition to Apache Doris from Apache Hive marks a revolutionary move for real-time analytics. From quick data writing to cost-effective computation, this IoV analytics platform transforms massive data into strategic insights, enhancing driving safety and experience for 4 million connected cars.

    9 min
  • The Year of the Graph Newsletter Vol. 25: The Fusion of Generative AI and Graph Technologies

    This story was originally published on HackerNoon at: https://hackernoon.com/graphs-analytics-and-generative-ai-the-year-of-the-graph-newsletter-vol-25-winter-2023-2024.


    Graphs, analytics and Generative AI. An account of the different ways graphs and AI mingle, plus industry and research news.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #data-science, #ai, #analytics, #knowledge-graph, #rag, #emerging-technology, #generative-ai, #graph-database, and more.


    This story was written by: @linked_do. Learn more about this writer by checking @linked_do's about page,
    and for more stories, please visit hackernoon.com.


    The article discusses the intersection of graphs and AI, specifically focusing on knowledge graphs, graph databases, graph analytics, and generative AI. It explores the impact of generative AI on the mentioned areas and provides insights into product offerings and research efforts in this field. The article also highlights the use of vector databases and graph databases for retrieval augmented generation (RAG) and the role of data management vendors in integrating with large language models (LLMs). Additionally, it covers the advancements in graph analytics engines and the entry and exit of vendors in the graph database market. The article further delves into topics like assisted knowledge graph creation, ontology modeling, applied graph AI use cases, graph levels of detail, and the combination of graphs with large language models. It concludes with a discussion on the influence of knowledge graphs on improving LLM response accuracy in the enterprise and introduces various research efforts in this domain.

    1 min
  • Snowflake vs. BigQuery vs. ClickHouse: Mastering Cost-Effective Business Analytics

    This story was originally published on HackerNoon at: https://hackernoon.com/snowflake-vs-bigquery-vs-clickhouse-mastering-cost-effective-business-analytics.


    Explore Snowflake, BigQuery & ClickHouse's cost models in this deep dive. Unlock strategies for cost-effective analytics & transform your data approach.
    Check more stories related to data-science at: https://hackernoon.com/c/data-science.
    You can also check exclusive content about #business-analytics, #cloud-costs, #snowflake, #clickhouse, #google-bigquery, #manage-cloud-costs, #cost-effective-analytics, #good-company, and more.


    This story was written by: @altinityinc. Learn more about this writer by checking @altinityinc's about page,
    and for more stories, please visit hackernoon.com.


    TL;DR: Comparing the cost models of analytics platforms Snowflake, BigQuery, and ClickHouse reveals diverse pricing structures. Snowflake's costs lean towards computing, BigQuery's on-demand model can rack up with extensive data scans, while ClickHouse offers real-time analytics with efficient storage. Properly aligning with each platform's strengths and understanding their cost nuances can guide businesses to more cost-effective analytics solutions.

    12 min

About Data Science Tech Brief By HackerNoon

From the publisher's feed

Learn the latest data science updates in the tech world.