Software Daily

Software Daily

By SoftwareDaily.comNewsTech News
Download on the App Store

Software Daily episodes

  • Apache Airflow with Maxime Beauchemin, Vikram Koka, and Ash Berlin-Taylor

    Apache Airflow was released in 2015, introducing the first popular open source solution to data pipeline orchestration. Since that time, Airflow has been widely adopted for dependency-based data workflows. A developer might orchestrate a pipeline with hundreds of tasks, with dependencies between jobs in Spark, Hadoop, and Snowflake.

    Since Airflow’s creation, it has powered the data infrastructure at companies like Airbnb, Netflix, and Lyft. It has also been at the center of Astronomer, a startup that helps enterprises build infrastructure around Airflow. Airflow is used to construct DAGs–directed acyclic graphs for managing data workflows.

    Maxime Beauchemin is the creator of Airflow. Vikram Koka and Ash Berlin-Taylor work at Astronomer. They join the show to talk about the state of Airflow–the purpose of the project, its use cases, and open source ecosystem.

    0 min
  • Human in the Loop Data Analytics with Aditya Parameswaran

    The life cycle of data management includes data cleaning, extraction, integration, analysis and exploration, and machine learning models. It would be great if all of this data management could be handled with automation, but unfortunately that is not an option. For most applications, data management requires a human in the loop.

    A human in the loop might be responsible for working in a spreadsheet, or labeling data as a mechanical turk, or creating an algorithm for data labeling in Snorkel. Data scientists and data analysts are humans in the loop, studying large data sets.

    Aditya Parameswaran is an assistant professor at UC Berkeley. He studies human-in-the-loop data analytics, and he joins the show to talk about the work and the projects that he is focused on, including DataSpread, an alternative to Excel, and OrpheusDB, a relational database versioning system.

    0 min
  • Human in the Loop Data Analytics with Aditya Parameswaran

    The life cycle of data management includes data cleaning, extraction, integration, analysis and exploration, and machine learning models. It would be great if all of this data management could be handled with automation, but unfortunately that is not an option. For most applications, data management requires a human in the loop.

    A human in the loop might be responsible for working in a spreadsheet, or labeling data as a mechanical turk, or creating an algorithm for data labeling in Snorkel. Data scientists and data analysts are humans in the loop, studying large data sets.

    Aditya Parameswaran is an assistant professor at UC Berkeley. He studies human-in-the-loop data analytics, and he joins the show to talk about the work and the projects that he is focused on, including DataSpread, an alternative to Excel, and OrpheusDB, a relational database versioning system.

    0 min
  • Tilt: Kubernetes Tooling with Dan Bentley

    Kubernetes continues to mature as a platform for infrastructure management. At this point, many companies have well-developed workflows and deployment patterns for working with applications built on Kubernetes. The complexity of some of these deployments may be daunting, and when a new employee joins a company, that employee needs to get quickly onboarded with the custom dev environment. 

    Environment management is not the only issue with Kubernetes development. When a service gets updated, that update needs to be live and usable as fast as possible. When Kubernetes-related errors occur, those problems need to be easily accessible in a UI for triage.

    Dan Bentley is the CEO of Windmill Engineering, a company that makes a set of Kubernetes tools called Tilt. Dan joins the show to talk about the workflow for deploying Kubernetes infrastructure and the role of Tilt, the product he has been working on.

    0 min
  • Tilt: Kubernetes Tooling with Dan Bentley

    Kubernetes continues to mature as a platform for infrastructure management. At this point, many companies have well-developed workflows and deployment patterns for working with applications built on Kubernetes. The complexity of some of these deployments may be daunting, and when a new employee joins a company, that employee needs to get quickly onboarded with the custom dev environment. 

    Environment management is not the only issue with Kubernetes development. When a service gets updated, that update needs to be live and usable as fast as possible. When Kubernetes-related errors occur, those problems need to be easily accessible in a UI for triage.

    Dan Bentley is the CEO of Windmill Engineering, a company that makes a set of Kubernetes tools called Tilt. Dan joins the show to talk about the workflow for deploying Kubernetes infrastructure and the role of Tilt, the product he has been working on.

    0 min
  • Uber’s Data Visualization Tools with Ib Green

    Uber needs to visualize data on a range of different surfaces. A smartphone user sees cars moving around on a map as they wait for their ride to arrive. Data scientists and operations researchers within Uber study the renderings of traffic moving throughout a city.

    Data visualization is core to Uber, and the company has developed a stack of technologies around visualization in order to build appealing, highly functional applications. DeckGL is a library for high-performance visualizations of large data sets. LumaGL is a set of components that targets high performance rendering. These and other tools make up VisGL, the data visualization technology that powers Uber.

    Uber’s visualization team included Ib Green, who left Uber to co-found Unfolded.ai, a company that builds geospatial analytics products. He joins the show to discuss his work on visualization products and libraries at Uber, as well as the process of taking that work to found Unfolded.ai. Full disclosure: I am an investor in Unfolded.ai.

    0 min
  • Uber’s Data Visualization Tools with Ib Green

    Uber needs to visualize data on a range of different surfaces. A smartphone user sees cars moving around on a map as they wait for their ride to arrive. Data scientists and operations researchers within Uber study the renderings of traffic moving throughout a city.

    Data visualization is core to Uber, and the company has developed a stack of technologies around visualization in order to build appealing, highly functional applications. DeckGL is a library for high-performance visualizations of large data sets. LumaGL is a set of components that targets high performance rendering. These and other tools make up VisGL, the data visualization technology that powers Uber.

    Uber’s visualization team included Ib Green, who left Uber to co-found Unfolded.ai, a company that builds geospatial analytics products. He joins the show to discuss his work on visualization products and libraries at Uber, as well as the process of taking that work to found Unfolded.ai. Full disclosure: I am an investor in Unfolded.ai.

    0 min
  • Prisma: Modern Database Tooling with Johannes Schickling

    A frontend developer issuing a query to a backend server typically requires the developer to issue that query through an ORM or a raw database query. Prisma is an alternative to both of these data access patterns, allowing for easier database access through auto-generated, type-safe query building tailored to an existing database schema.

    By integrating with Prisma, the developer gets a database client that has query autocompletion, and an API server with less boilerplate code. Prisma also has a system called Prisma Migrate, which simplifies database and schema migrations.

    Johannes Schickling is CEO of Prisma, and he joins the show to talk about the developments of Prisma that have occurred since we last spoke, and where the company is headed.

    0 min
  • Prisma: Modern Database Tooling with Johannes Schickling

    A frontend developer issuing a query to a backend server typically requires the developer to issue that query through an ORM or a raw database query. Prisma is an alternative to both of these data access patterns, allowing for easier database access through auto-generated, type-safe query building tailored to an existing database schema.

    By integrating with Prisma, the developer gets a database client that has query autocompletion, and an API server with less boilerplate code. Prisma also has a system called Prisma Migrate, which simplifies database and schema migrations.

    Johannes Schickling is CEO of Prisma, and he joins the show to talk about the developments of Prisma that have occurred since we last spoke, and where the company is headed.

    0 min
  • Tecton: Machine Learning Platform from Uber with Kevin Stumpf

    Machine learning workflows have had a problem for a long time: taking a model from the prototyping step and putting it into production is not an easy task. A data scientist who is developing a model is often working with different tools, or a smaller data set, or different hardware than the environment which that model will be deployed to.

    This problem existed at Uber just as it does at many other companies. Models were difficult to release, iterations were complicated, and collaboration between engineers could never reach a point that resembled a harmonious “DevOps”-like workflow. To address these problems, Uber developed an internal system called Michelangelo.

    Some of the engineers working on Michelangelo within Uber realized that there was a business opportunity in taking the Michelangelo work and turning it into a product company. Thus, Tecton was born. Tecton is a machine learning platform focused on solving the same problems that existed within Uber. Kevin Stumpf is the CTO at Tecton, and he joins the show to talk about the machine learning problems of Uber, and his current work at Tecton.

    0 min

About Software Daily

From the publisher's feed

Technical interviews about software topics.