Greg Lewis is an economist and the Senior Principal Researcher at Microsoft and co-leads the ALICE (Automated Learning and Intelligence for Causation and Economics) and EconML projects, an effort to develop use AI and machine learning for economics research. He specializes in industrial organization, market design, applied econometrics and machine learning. His work is unified by the twin goals of making better sense of microeconomic data, and using those insights to optimize firm decision making and improve market performance. His research has spanned a range of industries – online retailing, online advertising, procurement, electricity, education. Before joining Microsoft, Prof. Lewis was a professor at Harvard for seven years. He has a Ph.D. from the University of Michigan.

In this interview, Harsh and Tiger ask with Prof. Lewis about his research on consumer’s shopping trends online, how tech platforms rank search results and products, how to do causal inference using neural nets, how machine learning is in some news “the new statistics” and how it’s being integrated into econometrics.

In his recent paper “You Can Lead a Horse to Water: Spatial Learning and Path Dependence in Consumer Search,” Prof. Lewis introduces a new model of consumer search known as spatial learning. He explains his main findings in layman’s terms, and discusses how online platforms have the ability to manipulate consumer purchases to some degree by playing an active role in ranking products during a consumer’s searching process. His key insights focus on how consumers develop beliefs about products they see online, and how those beliefs change through experience and observation of reviews. More importantly, his methods try to understand how a good or bad experience with a particular product influences a consumer’s buying decisions about related products.

Prof. Lewis is a leading advocate to use machine learning methods for fundamental research in economics. In this conversation, he breaks down his development of DeepIV: a method for instrumental variable (IV) analysis using deep neural networks. He starts off by explaining IV analysis using simple examples and guides us through the improvements offered by deep learning. He elaborates that while neural networks are incredibly powerful at pattern recognition, their black-box nature offers great challenges as it fails to explain their predictions.

Prof. Lewis also addresses some “big-picture” questions about research in general. He talks about application of theoretical models to real-world scenarios, the effectiveness of computer simulations, as well as the age-old debate about how to uncover the “truth” from data. He talks about conducting research in the private sector as opposed to at a university, being optimistic about technology, and even a paper by Bill Gates about pancake flipping.