“I am learning from the data scientist, and the data scientist is learning from the developer, constantly”

In Episode 3 of the The New Builders, David Taieb, a developer, is joined by Jorge Castañón, a data scientist, to explain how getting a drink together at Datapalooza led to a partnership in which they refined a sample application, built by Taieb, that predicts how much a flight will be delayed, based on weather conditions. Thanks to the rise of cloud computing, collaborative technology and complementary skill sets, Taieb and Castañón were able to achieve 60 percent accuracy with the app – far exceeding their expectations.

We get into the details behind their collaboration, including how open source tools like Simple Data Pipe helped Taieb move on-premises data sources into a cloud-based Apache Spark environment for analysis (8:32), the role of intuition in data science (13:23), how a common language of Spark and IPython notebooks enabled them to collaboratively execute and experiment with data (14:30), how deep learning could further improve the sample app (27:03), and why they think it’s never too late for developers and data scientists to start working together on their projects (29:49).

Check out the flight predictor sample app on Github.

You can find new episodes of The New Builders on developerWorks TV and SoundCloud. Find out more about IBM Cloud Data Services at IBM.biz/forbuilders. Contact hosts Doug Flora and Jim Young on Twitter (@DSFlora, @JW_Young) or email ([email protected], [email protected]).

The show’s music is provided by School for Robots. Check them out at schoolforrobots.bandcamp.com!