Summary
Building data products are complicated by the fact that there are so many different stakeholders with competing goals and priorities. It is also challenging because of the number of roles and capabilities that are necessary to go from idea to delivery. Different organizations have tried a multitude of organizational strategies to improve the success rate of these data teams with varying levels of success. In this episode Jesse Anderson shares the lessons that he has learned while working with dozens of businesses across industries to determine the team structures and communication styles that have generated the best results. If you are struggling to deliver value from big data, or just starting down the path of building the organizational capacity to turn raw information into valuable products then this is a conversation that you don’t want to miss.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data managementWhat are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. Go to dataengineeringpodcast.com/97things to add your voice and share your hard-earned expertise.When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!Are you bogged down by having to manually manage data access controls, repeatedly move and copy data, and create audit reports to prove compliance? How much time could you save if those tasks were automated across your cloud platforms? Immuta is an automated data governance solution that enables safe and easy data analytics in the cloud. Our comprehensive data-level security, auditing and de-identification features eliminate the need for time-consuming manual processes and our focus on data and compliance team collaboration empowers you to deliver quick and valuable data analytics on the most sensitive data to unlock the full potential of your cloud data platforms. Learn how we streamline and accelerate manual processes to help you derive real results from your data at dataengineeringpodcast.com/immuta.Today’s episode of the Data Engineering Podcast is sponsored by Datadog, a SaaS-based monitoring and analytics platform for cloud-scale infrastructure, applications, logs, and more. Datadog uses machine-learning based algorithms to detect errors and anomalies across your entire stack—which reduces the time it takes to detect and address outages and helps promote collaboration between Data Engineering, Operations, and the rest of the company. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial. If you start a trial and install Datadog’s agent, Datadog will send you a free T-shirt.Your host is Tobias Macey and today I’m interviewing Jesse Anderson about best practices for organizing and managing data teamsInterview
IntroductionHow did you get involved in the area of data management?Can you start by giving an overview of how you view the mission and responsibilities of a data team?What are the critical elements of a successful data team?Beyond the core pillars of data science, data engineering, and operations, what other specialized roles do you find helpful for larger or more sophisticated teams?For organizations that have "small data", how does that change the necessary composition of roles for successful data projects?What are the signs and symptoms that point to the need for a dedicated team that focuses on data?With data scientists and data engineers in particular being in such high demand, what are strategies that you have found effective for attracting new talent?In the case where you have engineers on staff, how do you identify internal talent that can be trained into these specialized roles?Another challenge that organizations face in dealing with data is how the team is organized. What are your thoughts on effective strategies for how to structure the communication and reporting structures of data teams? (e.g. centralized, embedded, etc.)How do you recommend evaluating potential candidates for each of the necessary roles?What are your thoughts on when to hire an outside consultant, vs building internal capacity?For managers who are responsible for data teams, how much understanding of data and analytics do they need to be effective?How do you define success or measure performance of a team focused on working with data?What are some of the anti-patterns that you have seen in managers who oversee data professionals?What are some of the most interesting, unexpected, or challenging lessons that you have learned in the process of helping organizations and individuals achieve success in data and analytics?What advice or additional resources do you have for anyone who is interested in learning more about how to build and grow a successful data team?Contact Info
Website@jessetanderson on TwitterParting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workersJoin the community in the new Zulip chat workspace at dataengineeringpodcast.com/chatLinks
Data Teams BookDBA == Database AdministratorML EngineerDataOpsThree VsThe Ultimate Guide To Switching Careers To Big DataS-1 ReportJesse Anderson’s Youtube ChannelVideo about interviewing for data teamsUber Data Infrastructure Progression Blog PostThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
View all episodes
