Data Mesh Radio

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Andrew's Medium post called Improving Data Quality with Data Contracts: https://medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698

LinkedIn: https://www.linkedin.com/in/andrewrhysjones/

Twitter: @andrewrjones / https://twitter.com/andrewrjones

In this episode, Scott interviews Andrew Jones, Tech Lead of the Data Infrastructure Team at GoCardless.

Andrew shares the story of how operational system changes kept breaking downstream data consumption (sound familiar?), especially using CDC. The software engineers couldn't easily use the CDC tooling and the data engineers could easily use the data from it either as CDC didn't structure the data for easy consumption.

Andrew wasn't really sure how other people were handling taking the API contract concept and leveraging it for data but started building out some generic simple tooling to let consumers and producers feel somewhat comfortable with their data contracts.

A big revelation was in helping data consumers make better asks for data. The data consumers weren't used to asking the producers for data, especially in a reliable and scalable way (sound familiar?). GoCardless now has an actual standard form for data consumers to use to request data and that is working quite well.

GoCardless plans to completely remove their CDC architecture by 3Q of this year to replace with data contracts. They are focusing on providing tooling to give domains the autonomy to serve data to consumers in their own way. While it isn't data mesh, especially with the lack of interoperability between data products and lack of source/producer-aligned data products, it seems to be working for GoCardless thus far.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman):

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Danilo's Webinar with Zhamak called "Data mesh and domain ownership": https://www.thoughtworks.com/en-us/about-us/events/webinars/core-principles-of-data-mesh/data-mesh-and-domain-ownership

Vladik Khononov, 7 Years of DDD: https://www.youtube.com/watch?v=h_HjtYAH0AI

Danilo LinkedIn: https://www.linkedin.com/in/danilosato/

Danilo Twitter: @dtsato / https://twitter.com/dtsato

Andrew LinkedIn: https://www.linkedin.com/in/andrewharmellaw/

Andrew Twitter: @al94781 / https://twitter.com/al94781

In this episode, Scott interviews two Domain Driven Design (DDD) experts from Thoughtworks - Danilo Sato Director and the Head of Data & AI (part of Office of the CTO) and Andrew Harmel-Law, Technical Principal. This was a further delve into Domain Driven Design for Data after the conversations with Paolo Platter and Piethein Strengholt.

Danilo and Andrew gave a lot of great information about Event Storming, domain definitions and boundaries, ubiquitous language, and so much more but the main theme was "just get people to talk to each other".

DDD is about bridging the gap between how the tech people talk and how the business/business people talk; if you are doing it right, both sides can understand each other and then the engineers can implement those business process learnings as part of the code.

For an initial PoC, Danilo recommends starting with 2-3 data products. It is better if you can do the PoC across multiple domains but it isn't necessary. Validate value and do it quickly. As Andrew mentions, the earlier you can show value, the less pressure there is overall. Look for the initial quick wins while also building for the long-term.

One key thing to remember, per Danilo, when doing DDD for data and data mesh in general: it is always an iterative process. Andrew briefly discussed a way to do DDD in more of a guerilla style than the blue/red books (well known DDD guides). Don't get ahead of yourself as Max Schultze mentioned in his episode. Do not let the size of the eventual task throw you into analysis paralysis.

Andrew talked a lot about how normalization and strong abstractions on the application side make it very difficult to re-add the context lost when you normalize. Both Andrew and Danilo talked about the need to...

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Angelo's LinkedIn: https://www.linkedin.com/in/angelomartelli/

In this episode, Scott interviews Angelo Martelli, Group Leader of Data Services at logistics company Vanderlande.

Angelo laid out his framework for driving data mesh buy-in internally at Vanderlande which helped them take the idea from a small group to a company-wide initiative:

Start with proving there is a problem that you are trying to solve - if everything is functioning well, why focus your efforts on that area instead of trying to fix another? Your proof should be as fact-based as possible, e.g. how long does it take to make a change to your data warehouse. Focus on proving that your incremental investments are driving sub-linear returns. Other areas to look to prove problems: how many people are involved in a change to your data warehouse, percent of time spent on regression testing versus development, mean time to resolution of challenges, etc.

Once you have some proof, you need to work towards understanding the problem you are trying to solve. It's not "deploying a data mesh", it's scaling the organization to be agile relative to data and be able to make more (and better) data-informed decisions.

Next, you need to understand your organization. Who are the right people that can help you? How does your organization work relative to culture and process? Which domains are struggling and how? Tie the implementation goals to the actual business challenges.

Then, you need to demystify data mesh, make it easy to understand for people not well versed in data - what are we actually trying to accomplish and why?

Last, make it concrete / prove it out. Make a few data products, make a simple platform for folks to use.

Angelo then recommends that once you have momentum, sharing a very clear vision is crucial. Not just sharing in a document but actually having conversations to really make sure the context and vision is understood. Data mesh is about collaboration, you must work together so it is imperative to make expectations very clear.

Similar to Abhi Sivasailam, Angelo also stressed the importance of the domain data model and abstracting that away from the application model(s). The business model is what matters for data.

All of that and so much more. Also, Angelo gives a shout out to the usefulness of the Data Mesh Learning community. 😎

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here:

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Brian is giving away 10 copies of his book for free to those who sign up for a chat to share more about your current challenges related to the book topic or shadow/domain-based IT. To claim your free book, fill out a contact form here and mention "Data Mesh Radio" in the comments.

Brian's contact info:

Email: brian at minimumviablearchitecture.com

LinkedIn: https://www.linkedin.com/in/brianmcmillan01/

Website: https://www.minimumviablearchitecture.com/

Scott interviews Brian McMillan a former Enterprise Architect who took time off to write a book called 'Building Data Products: Introduction to Data and Analytics Engineering for Non-Programmers'. You can learn more about the book - and get a free copy, see below - here: https://www.minimumviablearchitecture.com/

Brian's book lays out a path for the people who are doing the most with data in domains to elevate their skill sets and produce small-scale data products. They do this through a slow ramp from understanding SQL queries to learning data modeling to learning how to publish their data and use simple orchestration tooling. It isn't magic, it will take time and training, but it means you have more people with strong domain knowledge becoming part of the data and analytics engineering process, sharing their business context in scalable and repeatable ways.

Brian's approach can also be used for a pretty easy path to an exploratory platform. There isn't a lot of pre-build to get going so teams can much more easily test out a hypothesis or two rather than it being a lengthy and costly approval and build cycle. There is also an easy path once someone finds a "there" there, to move it to something far more scalable and reliable in the cloud.

There is a lot from the book and interview that can be adapted to help level up your teams' data literacy.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Somewhat short episode. Scott emphasizes that you aren't the only one facing challenges with data mesh. You aren't behind the curve, you are ahead of it - data mesh is bleeding edge. But with a bleeding edge, there is some blood...

Also, you need to think about how others' implementations works for them. Trying to copy-paste another organization's implementation is going to lead to a failed implementation in yours. Take the learnings away and think about how to apply them to your organization.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Sheetal's (and Divya and Madhu from Thoughtworks) Data Mesh Learning Meetup presentation: https://www.youtube.com/watch?v=5btUaLPdaNk

Sheetal's post re data mesh and using DataHub: https://blog.datahubproject.io/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f

Sheetal's LinkedIn: https://www.linkedin.com/in/sheetalpratik/

Data Governance using Data Mesh paper: https://easychair.org/publications/preprint/qZ3m

In this episode, Scott interviews Sheetal Pratik, Director of Engineering, Data Integration at Adidas. If Sheetal's name sounds familiar to data mesh officianados, she presented at the Data Mesh Learning meetup in August 2021.

Sheetal is passionate about giving companies the permission AND a workable plan for getting started with data mesh. She covered a wide range of things regarding getting starting but a few really stood out:

• Don't try to tackle tomorrow's challenges today
• Break your implementation into phases: development, adoption, and scaling
• Start your initial data mesh MVP with a simple data product with a simple schema - your goal is to develop the "muscles" around creating and deploying data products rather than shooting for a high-value product first
• Keep to a reasonable budget and prove viability and value

Sheetal also covered how much you really have to have in place to create and evaluate your MVP. There will always be evolution and change and your organization has to be ready for that. That can be frightening or inspirational. Sheetal chooses it as inspirational - it gives you the freedom to move quickly as long as the organization understands that things will change in the future.

She wrapped up with saying that data mesh shouldn't be scary, you should be excited about this journey and what it can mean for your organization. Get an MVP out the door, it will take time, don't get ahead of yourself. Data mesh success can happen if you let it.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here:

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Dr. Abadi's blog post on data federalization and data virtualization: https://blog.starburst.io/data-federation-and-data-virtualization-never-worked-in-the-past-but-now-its-different

Dr. Abadi's contact info:

LinkedIn:

Twitter: @daniel_abadi / https://twitter.com/daniel_abadi

Starburst blog posts: https://blog.starburst.io/author/daniel-abadi

Don't forget to catch Dr. Abadi at Datanova - the Data Mesh Summit on Feb 9-10th. Thanks to Starburst for sponsoring the transcripts for Data Mesh Radio, check out the transcript here. And check out Starburst's other free data mesh resources here.

In this episode, Scott interviewed Dr. Daniel Abadi, the Darnell-Kanal Computer Science Professor at the University of Maryland with a focus on scalable data management research. Dr. Abadi will be presenting next week at the Data Mesh Summit on Data Fabric and Data Mesh alongside Zhamak and Sanjeev Mohan.

This was a pretty wide ranging and free wheeling conversation about data virtualization in general and how it can be used in data mesh. Both agreed that there are many places where data virtualization can play in data mesh, whether in extracting information from operational systems, stitching together a data product once data processing has been done, or at the mesh experience plane re combining data across multiple data products. Dr. Abadi specifically mentions something like a query fabric that makes use of a data virtualization approach, not just tools that only do data virtualization.

There is a natural side effect of having multiple different technologies in use - when you give the domains the ability to use what they choose, the difficulty of combining data from multiple sources needs to be solved. There is always a balance between how much you just copy data and how much you can access in the source system and data virtualization can give a few more options rather than all or nothing.

As data virtualization has been around as a concept for 30+ years, there is a lot of baggage with the term but Dr. Abadi sees there being recent advancements that mean more people should take a second look at where they can be...

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Colleen's contact info:

Email: colleen at starburst.io

LinkedIn: https://www.linkedin.com/in/colleen-tartow-phd/

Twitter: @CTartow / https://twitter.com/CTartow

Colleen's blog: https://thesequel.substack.com/

The Data Buffet blog post: https://thesequel.substack.com/p/the-endless-data-buffet

Don't forget to catch Colleen at Datanova - the Data Mesh Summit on Feb 9-10th. Thanks to Starburst for sponsoring the transcripts for Data Mesh Radio, check out the transcript here. And check out Starburst's other free data mesh resources here.

In this episode, Scott interviews Dr. Colleen Tartow, Director of Engineering at Starburst. They chatted about the fun and usefulness of food-related analogies to data mesh - you want it to work like a brunch buffet as that is basically a perfect Saturday in Colleen's eyes. And Scott shared the concept of a grocery store - intentional food preparation with different degrees of ingredient and meal preparedness.

Colleen and Scott then covered the "Modern Data Stack" and its relevance to data mesh - in Colleen's eyes, the Modern Data Stack isn't all that modern, it is just the same old paradigm of data teams trying to do their best to work with the output of application/operational data stores and systems that aren't designed with the data in mind. Scott somewhat agreed with his own spin.

Colleen shared her 4 S paradigm re doing data well: speed, simplicity, scalability, and SQL. Yes, slightly Starburst self-serving (4 more S words!) but still relevant and interesting.

They wrapped up with trying to figure out how much of data mesh can companies get away with not doing, which seems to be the topic on everyone's mind. More questions than answers there...

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here:

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Max's contact info and related links:

LinkedIn: https://www.linkedin.com/in/max-schultze-b11996110/

Twitter: @mcs1408 / https://twitter.com/mcs1408

Presentations:

Similar to content of the book (Data Innovation Summit): https://www.youtube.com/watch?v=rqYFqtztWi4

Zalando's Story (Spark AI Summit): https://www.youtube.com/watch?v=UrM8yCjmzzw

Max and Arif's 'Data Mesh in Practice' book links:

O'Reilly: https://www.oreilly.com/library/view/data-mesh-in/9781098108502/

Starburst (free gated download): https://www.starburst.io/info/data-mesh-in-practice-ebook/

Max and Arif's upcoming training (Feb 14th): https://www.oreilly.com/live-events/data-mesh-in-practice/0636920508816/0636920068685/

Transcript (link) courtesy of Starburst; check out their other data mesh resources here

Part of Starburst's Datanova Data Mesh Summit takeover week.

Max and Arif's upcoming (Feb 9th) Datanova/Starburst Data Mesh Summit Presentation: https://www.starburst.io/info/datanova2022/

In this episode, Scott interviews one of the most prolific content producers in data mesh, Max Schultze, Data Engineering Manager at Fashion E-Tailer Zalando.

Max shares a LOT of very valuable advice while reflecting on Zalando's data mesh journey so far two years in. Max recommends starting with empathy building and knowledge sharing but looking for ways to scale versus only 1:1 or small group conversations. He had a clever way to force the hands of data consumers to speak to data producers via an Inverse Conway Maneuver that