Building the Backend: Data Solutions that Power Leading Organizations

In this episode of Building The Backend we hear from Prukalpa Sankar – Co-founder of Atlan, where we talk all about data quality/governance, common issues organizations face when implementing data quality and much much more.

Below are top 3 value bombs: 

• Data Governance has a bad reputation. It should not be a bureaucratic controlling process that is pushed from the top down. 
• Active Metadata is key to modern data architectures, essentially it’s putting together all the human and machine generated metadata together to derive insights. 
• One of the most difficult metadata attributes to capture is the context for the data as this almost always requires input from humans and tribal knowledge is often lost and is not documented.

This is a podcast episode you do not want to miss with Stephen Brobst, CTO @ Teradata. We discuss all things Data Warehouses, the shift to the distributed cloud and, key principles to implementing  successful DW's.

Top 3 Value Bombs:

• Large organizations are shifting more to a distributed / inter-cloud architecture for many reasons, a couple of reasons are data sovereignty, increasing residency and reducing costs.
• Just because your DW does not support indexing does not mean you do not need them. 
• One of the most common reasons DW’s fail is they are led by IT and not the business. The DW should be led directly by the business needs and most important initiatives. 
  
  

“The hardest part of ETL is not building the connectors, it is maintaining them.” Truer words never spoken. Really enjoyed this episode with Michel Tricot CEO & Co-Founder of Airbyte where we discuss all things data integration and connectors.

Top 3 value bombs:

• The future of ETL/ELT integration connectors may lie with open source. Many closed source data integration tools only create connectors if the ROI is there, but this leaves many tools out and speed to market can be slow. Airbyte has created a modular open source framework that allows the community to quickly build reliable data connectors. 
• As Airbyte starts to monetize they have some innovative methods, one of which is if a developer from the open source community creates and maintains a connector they could potentially get a small percentage of revenue associated with that connector. 
• Data governance ang logging is  increasingly becoming more important in the coming years. 
  
  

This episode features Gleb Mezhanskiy Co-Founder & CEO @ Datafold, during our discussion we talk all about data observability and how to improve your data quality. Before Datafold, Gleb was a founding member of data teams at Lyft and Autodesk, where he built sophisticated data platforms and developed tooling to improve productivity and data quality.

Top 3 Value Bombs:

• The foundation of any data observability platform is the data catalog. 
• Data observability becomes increasingly difficult the more data sets you have if you do not define your process to track and monitor your data. 
• Do not surprise your report consumers. knowing how your metrics will change in prod before your deployment can be done with the right data observability process and regression testing. 
  
  

This episode features Arjun Narayan Co-Founder & CEO @ Materialize, during our discussion we talk all about transforming streaming data, the do’s the don’ts and how Materialize is changing the landscape of streaming. 

Top 3 Value Bombs:

1. When creating schema changes organizations should always strive to create forward compatible schema changes only. This means consumers will be able to consume your data model without impacting them, they just may be missing your newly added column.
   
   
2. Materialized computations are bound to change in the future, either due to bugs or requirement changes. Kafka allows you to replay all your previous messages to update the calculation. 
   
   
3. The cloud is still young, over the coming years we will see many more technologies that are specifically built with a cloud focus. 

This episode features Jean-Yves Stephan Co-Founder & CEO @ Data Mechanics (recently Acq. by Spot by NetApp), during our discussion we talk about optimizing Spark to run in the cloud at a low cost.

Top 3 Value Bombs:

• Running Spark CAN be expensive but there are ways to reduce your current operating costs by 50-75% by smart automations (i.e.  tune for node type, memory and CPU). 
• Spot instances can lower your costs by utilizing unused instances. 
• Creating serverless architectures and using containers will allow for more flexibility with deployment models and scalability. 

This episode features Josh Benamrum, who is the co-founder of Databand. Databand is a company that helps engineering teams achieve better observability and control over their tech stack.

Top 3 Value Bombs: 

• When observing our data we should be looking at our data and pipelines
• Don’t wait till the board meeting for an incorrect metric to make DQ a priority
• Having clear SLA’s on just what data quality means across the organization is essential 
  
  

Travis welcomes to his podcast Saket Saurabh, who provides a window into the world of data management and the self-service options that are democratizing it. Co-founder and CEO of Nexla, Saket has a passion for data and infrastructure and how to improve its flow among partners, customers and vendors. Nexla automates various data engineering tasks, intelligently creates an abstraction of data and enables collaboration among people at different skill levels. Named a 2021 Cool Vendor by Gartner, Nexla is a leader in data preparation, integration and tracking.

Top 3 value bombs: 

• Data architectures overall need to be more abstract to enable future flexibility
• The first stumbling block for most organizations is not knowing where to locate their data.
• ETL is dead. The ELT model has become central while streaming and real-time use cases are becoming prevalent.

In this episode, we speak with Rob Hedgpeth, a  director of developer developer relations at Maria DB.  We explore all things Maria DB, the capabilities it has and when you should consider it for your next project. 

Top 3 value bombs:

• MariaDB follows a shared nothing architecture and supports distributed SQL for unlimited scaling on demand.
• MariaDB can handle many types of storage (i.e. document store, graph and spatial)
• When deciding on your next relational database do not just look at options available within your cloud service provider, include Databases as a Service within your analysis (i.e. Sky SQL - Maris DB’s commercial product). 

In this episode, we speak with Lior Gavish, the co-founder of Monte Carlo to explore all things data quality. Monte Carlo is a data lineage and observability tool that lowers your data downtime.

Top 3 Value Bombs:

1. Data products should be thought of in it’s entirely from the source to the consumer.
2. No one data stakeholder can solve data quality issues, it’s a collaboration of the data engineers, business, data consumer and even software to help automate certain aspects of cataloging and capturing meaningful metadata. 
3. Good data quality processes should alert you to anomalies in your metrics before your data consumers do.