May 23, 2022

Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy

1 hour 10 minutes

Summary

The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction. In this episode he explores the tension between code-first and no-code utilities and how he is working to balance the strengths without falling prey to their shortcomings.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management

When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription

So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.

Your host is Tobias Macey and today I’m interviewing Raj Bains about how improving the user experience for data tools can make your work as a data engineer better and easier

Interview

Introduction

How did you get involved in the area of data management?

What are the broad categories of data tool designs that are available currently and how does that impact what is possible with them?

What are the points of friction that are introduced by the tools?

Can you share some of the types of workarounds or wasted effort that are made necessary by those design elements?

What are the core design principles that you have built into Prophecy to address these shortcomings?

How do those user experience changes improve the quality and speed of work for data engineers?

How has the Prophecy platform changed since we last spoke almost a year ago?

What are the tradeoffs of low code systems for productivity vs. flexibility and creativity?

What are the most interesting, innovative, or unexpected approaches to developer experience that you have seen for data tools?

What are the most interesting, unexpected, or challenging lessons that you have learned while working on user experience optimization for data tooling at Prophecy?

When is it more important to optimize for computational efficiency over developer productivity?

What do you have planned for the future of Prophecy?

Contact Info

@_raj_bains on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.

Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.

If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.

To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

Prophecy

Podcast Episode

CUDA

Clustrix

Hortonworks

Apache Hive

Compilerworks

Podcast Episode

Airflow

Databricks

Fivetran

Podcast Episode

Airbyte

Podcast Episode

Streamsets

Change Data Capture

Apache Pig

Spark

Scala

Ab Initio

Type 2 Slowly Changing Dimensions

AWS Deequ

Matillion

Podcast Episode

Prophecy SaaS

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast

...more

View all episodes

By Tobias Macey

4.6

134134 ratings