Software Engineering Daily

By Software Engineering DailyNews Technology Tech News

Software Engineering Daily episodes

December 11, 2015Data Science at Spotify with Boxun Zhang
“I normally try to sit together or very close to a product team or engineering team. And by doing so, I get very close to the source of all kinds of challenging problems.”
Spotify is a streaming music service that uses data science and machine learning to implement product features such as recommendation systems and music categorization, but also to answer internal questions.
Boxun Zhang is a data scientist at Spotify where he focuses on understanding user behavior within the product.
Questions
What is the overlap between distributed systems and data science?
How has Spotify’s big data architecture evolved over time?
As a data scientist do you need to understand this big data architecture well?
What were the benefits for starting to use Kafka?
What kinds of data science problems do you tackle at Spotify?
Could you describe what a random forest is?
Why are there so many streaming systems, and what do you use at Spotify?
How will data science change moving towards the future?
Links
The Evolution of Big Data at Spotify
Luigi
Project Jupyter
XGBoost
Automatic Statistician
Skytree
Sponsors
Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.
Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.
The post Data Science at Spotify with Boxun Zhang appeared first on Software Engineering Daily.
56 min
December 08, 2015Learning Machines with Richard Golden
“When I was a graduate student, I was sitting in the office of my advisor in electrical engineering and he said, ‘Look out that window – you see a Volkswagon, I see a realization of a random variable.’ ”
Richard Golden is the host of Learning Machines 101, a podcast that covers artificial intelligence and machine learning topics. Dr. Golden is also a full-time Professor of Cognitive Science and Electrical Engineering at UT Dallas.
Questions
What is machine learning?
What are the fundamental concepts to build artificial intelligence?
How do you define a rule in the domain of machine learning?
How can a machine learning system estimate the probability of something it has not seen?
Could you explain how ML could be applied to real world healthcare scenarios?
What is a neural network?
What is the difference between natural and artificial intelligence?
Links
Bayesian Model Averaging
Frequentist Model Averaging
McCulloch-Pitts Formal Neurons
Dr. Golden’s Professor Page
Sponsors
Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.
Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.
The post Learning Machines with Richard Golden appeared first on Software Engineering Daily.
56 min
November 17, 2015Machine Learning and Technical Debt with D. Sculley
“Changing anything changes everything.”
Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems.
D. Sculley is a software engineer at Google, focusing on machine learning, data mining, and information retrieval. He recently co-authored the paper Machine Learning: The High Interest Credit Card of Technical Debt.
Questions
How do you define technical debt?
Why does technical debt tend to compound like financial debt?
Is machine learning the marriage of hard-coded software logic and constantly changing external data?
What types of anti-patterns should be avoided by machine learning engineers?
What is a decision threshold in a machine learning system?
What advice would you give to organizations that are building their prototypes and product systems in different languages?
Links
Technical Debt
Adapter pattern and glue code
D’s research page
Sponsors
Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.
Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.
The post Machine Learning and Technical Debt with D. Sculley appeared first on Software Engineering Daily.
32 min
October 05, 2015Bridging Data Science and Engineering with Greg Lamp
Current infrastructure makes it difficult for data scientists to share analytical models with the software engineers who need to integrate them.
Yhat is an enterprise software company tackling the challenge of how data science gets done. Their products enable companies and users to easily deploy data science environments and translate analytical models into production code.
Greg Lamp is the Co-founder and CTO of Yhat and previously worked as a product manager in financial services. Yhat was part of the Y Combinator winter 2015 class.
Questions
At a software company, what is the typical relationship between data scientists and software engineers?
Does Yhat turn data scientists into HTTP endpoints?
What was the most counterintuitive advice you received at Y-Combinator?
What is the moonshot goal for Yhat?
Is it easier to teach data science to an engineer or engineering to a data scientist?
Links
Yhat’s Products
Yhat Blog
Greg’s Website
Beer Recommender Talk
The post Bridging Data Science and Engineering with Greg Lamp appeared first on Software Engineering Daily.
48 min
October 03, 2015Kaggle with Ben Hamner
Data science competitions are an effective way to crowdsource the best solutions for challenging datasets.
Kaggle is a platform for data scientists to collaborate and compete on machine learning problems with the opportunity to win money from the competitions’ sponsors.
Ben Hamner is the co-founder and CTO of Kaggle.
Questions
What is Kaggle?
How does the experience of an individual competitor compare to the experience of a data science team?
What is Kaggle’s tech stack?
Do companies collect too much data?
How do you use machine learning to convert neural patterns into control signals?
Links
Kaggle
Kaggle Scripts
Ben Hamner on Quora
Ben Hamner on Twitter
The post Kaggle with Ben Hamner appeared first on Software Engineering Daily.
50 min
September 30, 2015Teaching Data Science with Vik Paruchuri
There is a need for more data scientists to make sense of the vast amounts of data we produce and store.
Dataquest is an in-browser platform for learning data science that is tackling this problem.
Vik Paruchuri is the founder of Dataquest. He was previously a machine learning engineer at EdX and before that a U.S. diplomat.
Questions
What is data science?
How does data science compare to software engineering?
How does someone new to data science go about starting off at Kaggle?
In machine learning, there is unsupervised learning and supervised learning. Could you contrast these two?
What are the biggest world problems that will be solved with data science?
Links
Dataquest
How to actually learn data science
Kaggle
The post Teaching Data Science with Vik Paruchuri appeared first on Software Engineering Daily.
45 min