NLP Highlights

By Allen Institute for Artificial Intelligence

What's NLP Highlights about?

Discussing recent and interesting work related to natural language processing. Matt Gardner and Waleed Ammar, research scientists at the Allen Institute for Artificial Intelligence, give short discussions of papers, mostly in interviews with authors about their work.

Download our free app to listen on your phone

LAST EPISODE

114 - Behavioral Testing of...

05.26.2020

We invited Marco Tulio Ribeiro, a Senior Researcher at Microsoft, to talk about evaluating NLP models using behavioral testing, a framework borrowed from Software Engineering. Marco describes three kinds of black-box tests the check whether NLP models satisfy certain necessary ...

NLP Highlights episodes:

05.26.2020 114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro We invited Marco Tulio Ribeiro, a Senior Researcher at Microsoft, to talk about evaluating NLP models using behavioral testing, a framework borrowed from Software Engineering. Marco describes three kinds of black-box tests the check whether NLP models satisfy certain necessary ...
05.22.2020 113 - Managing Industry Research Teams, with Fernando Pereira We invited Fernando Pereira, a VP and Distinguished Engineer at Google, where he leads NLU and ML research, to talk about managing NLP research teams in industry. Topics we discussed include prioritizing research against product development and effective collaboration with ...
05.13.2020 112 - Alignment of Multilingual Contextual Representations, with Steven Cao We invited Steven Cao to talk about his paper on multilingual alignment of contextual word embeddings. We started by discussing how multilingual transformers work in general, and then focus on Steven’s work on aligning word representations. The core idea is ...
04.27.2020 111 - Typologically diverse, multi-lingual, information-seeking questions, with Jon Clark We invited Jon Clark from Google to talk about TyDi QA, a new question answering dataset, for this episode. The dataset contains information seeking questions in 11 languages that are typologically diverse, i.e., they differ from each other in terms ...
04.06.2020 110 - Natural Questions, with Tom Kwiatkowski and Michael Collins In this episode, Tom Kwiatkowski and Michael Collins talk about Natural Questions, a benchmark for question answering research. We discuss how the dataset was collected to reflect naturally-occurring questions, the criteria used for identifying short and long answers, how this ...
03.30.2020 109 - What Does Your Model Know About Language, with Ellie Pavlick How do we know, in a concrete quantitative sense, what a deep learning model knows about language? In this episode, Ellie Pavlick talks about two broad directions to address this question: structural and behavioral analysis of models. In structural analysis, ...
03.23.2020 108 - Data-To-Text Generation, with Verena Rieser and Ondřej Dušek In this episode we invite Verena Rieser and Ondřej Dušek on to talk to us about the complexities of generating natural language when you have some kind of structured meaning representation as input. We talk about when you might want ...
02.24.2020 107 - Multi-Modal Transformers, with Hao Tan and Mohit Bansal In this episode, we invite Hao Tan and Mohit Bansal to talk about multi-modal training of transformers, focusing in particular on their EMNLP 2019 paper that introduced LXMERT, a vision+language transformer. We spend the first third of the episode talking ...
02.17.2020 106 - Ethical Considerations In NLP Research, with Emily Bender In this episode, we talked to Emily Bender about the ethical considerations in developing NLP models and putting them in production. Emily cited specific examples of ethical issues, and talked about the kinds of potential concerns to keep in mind, ...
02.10.2020 105 - Question Generation, with Sudha Rao In this episode we invite Sudha Rao to talk about question generation. We talk about different settings where you might want to generate questions: for human testing scenarios (rare), for data augmentation (has been done a bunch for SQuAD-like tasks), ...
02.03.2020 104 - Model Distillation, with Victor Sanh and Thomas Wolf In this episode we talked with Victor Sanh and Thomas Wolf from HuggingFace about model distillation, and DistilBERT as one example of distillation. The idea behind model distillation is compressing a large model by building a smaller model, with much ...
01.27.2020 103 - Processing Language in Social Media, with Brendan O'Connor We talked to Brendan O’Connor for this episode about processing language in social media. Brendan started off by telling us about his projects that studied the linguistic and geographical patterns of African American English (AAE), and how obtaining data from ...
01.20.2020 102 - Biomedical NLP research at the National Institute of Health with Dina Demner-Fushman What exciting NLP research problems are involved in processing biomedical and clinical data? In this episode, we spoke with Dina Demner-Fushman, who leads NLP and IR research at the Lister Hill National Center for Biomedical Communications, part of the National ...
01.14.2020 101 - The lottery ticket hypothesis, with Jonathan Frankle In this episode, Jonathan Frankle describes the lottery ticket hypothesis, a popular explanation of how over-parameterization helps in training neural networks. We discuss pruning methods used to uncover subnetworks (winning tickets) which were initialized in a particularly effective way. We ...
01.08.2020 100 - NLP Startups, with Oren Etzioni For our 100th episode, we invite AI2 CEO Oren Etzioni to talk to us about NLP startups. Oren has founded several successful startups, is himself an investor in startups, and helps with AI2's startup incubator. Some of our discussion topics ...
12.16.2019 99 - Evaluating Protein Transfer Learning, With Roshan Rao And Neil Thomas For this episode, we chatted with Neil Thomas and Roshan Rao about modeling protein sequences and evaluating transfer learning methods for a set of five protein modeling tasks. Learning representations using self-supervised pretaining objectives has shown promising results in transferring ...
12.09.2019 98 - Analyzing Information Flow In Transformers, With Elena Voita What function do the different attention heads serve in multi-headed attention models? In this episode, Lena describes how to use attribution methods to assess the importance and contribution of different heads in several tasks, and describes a gating mechanism to ...
11.27.2019 97 - Automated Analysis Of Historical Printed Documents, With Taylor Berg-Kirkpatrick In this episode, we talk to Taylor Berg-Kirkpatrick about optical character recognition (OCR) on historical documents. Taylor starts off by describing some practical issues related to old scanning processes of documents that make performing OCR on them a difficult problem. ...
11.12.2019 96 - Question Answering as an Annotation Format, with Luke Zettlemoyer In this episode, we chat with Luke Zettlemoyer about Question Answering as a format for crowdsourcing annotations of various semantic phenomena in text. We start by talking about QA-SRL and QAMR, two datasets that use QA pairs to annotate predicate-argument ...
10.07.2019 95 - Common sense reasoning, with Yejin Choi In this episode, we invite Yejin Choi to talk about common sense knowledge and reasoning, a growing area in NLP. We start by discussing a working definition of “common sense” and the practical utility of studying it. We then talk ...
09.30.2019 94 - Decompositional Semantics, with Aaron White In this episode, Aaron White tells us about the decompositional semantics initiative (Decomp), an attempt to re-think the prototypical approach to semantic representation and annotation. The basic idea is to decompose complex semantic classes such as ‘agent’ and ‘patient’ into ...
07.22.2019 93 - NLP/ML for clinical data, with Alistair Johnson In this episode, we invite Alistair Johnson to discuss the main challenge in applying NLP/ML to clinical domains: the lack of data. We discuss privacy concerns, de-identification, synthesizing records, legal liabilities and data heterogeneity. We also discuss how the MIMIC ...
07.05.2019 92 - Computational Humanities, with David Bamman In this episode, we invite David Bamman to give an overview of computational humanities. We discuss examples of questions studied in computational humanities (e.g., characterizing fictionality, assessing novelty, measuring the attention given to male vs. female characters in the literature). ...
06.26.2019 91 - (Executable) Semantic Parsing, with Jonathan Berant In this episode, we invite Jonathan Berant to talk about executable semantic parsing. We discuss what executable semantic parsing is and how it differs from related tasks such as semantic dependency parsing and abstract meaning representation (AMR) parsing. We talk ...
05.31.2019 90 - Research in Academia versus Industry, with Philip Resnik and Jason Baldridge How is it like to do research in academia vs. industry? In this episode, we invite Jason Baldridge (UT Austin => Google) and Philip Resnik (Sun Microsystems => UMD) to discuss some of the aspects one may want to consider ...
05.31.2019 89 - Dialog Systems, with Zhou Yu In this episode, we invite Zhou Yu to give an overview of dialogue systems. We discuss different types of dialogue systems (task-oriented vs. non-task-oriented), the main building blocks and how they relate to other research areas in NLP, how to ...
05.07.2019 88 - A Structural Probe for Finding Syntax in Word Representations, with John Hewitt In this episode, we invite John Hewitt to discuss his take on how to probe word embeddings for syntactic information. The basic idea is to project word embeddings to a vector space where the L2 distance between a pair of ...
04.25.2019 87 - Pathologies of Neural Models Make Interpretation Difficult, with Shi Feng In this episode, Shi Feng joins us to discuss his recent work on identifying pathological behaviors of neural models for NLP tasks. Shi uses input word gradients to identify the least important word for a model's prediction, and iteratively removes ...
04.15.2019 86 - NLP for Evidence-based Medicine, with Byron Wallace In this episode, Byron Wallace tells us about interdisciplinary work between evidence based medicine and natural language processing. We discuss extracting PICO frames from articles describing clinical trials and data available for direct and weak supervision. We also discuss automating ...
03.29.2019 85 - Stress in Research, with Charles Sutton In this episode, Charles Sutton walks us through common sources of stress for researchers and suggests coping strategies to maintain your sanity. We talk about how pursuing a research career is similar to participating in a life-long international tournament, conflating ...
03.26.2019 84 - Large Teams Develop, Small Groups Disrupt, with Lingfei Wu In a recent Nature paper, Lingfei Wu (Ling) suggests that smaller teams of scientists tend to do more disruptive work. In this episode, we invite Ling to discuss their results, how they define disruption and possible reasons why smaller teams ...
03.13.2019 83 - Knowledge Base Construction, with Sebastian Riedel In this episode, we invite Sebastian Riedel to talk about knowledge base construction (KBC). Why is it an important research area? What are the tradeoffs between using an open vs. closed schema? What are popular methods currently used, and what ...
03.06.2019 82 - Visual Reasoning, with Yoav Artzi In this episode, Yoav Artzi joins us to talk about visual reasoning. We start by defining what visual reasoning is, then discuss the pros and cons of different tasks and datasets. We discuss some of the models used for visual ...
02.06.2019 81 - BlackboxNLP, with Afra Alishahi and Tal Linzen Neural models recently resulted in large performance improvements in various NLP problems, but our understanding of what and how the models learn remains fairly limited. In this episode, Tal Linzen and Afra Alishahi talk to us about BlackboxNLP, an EMNLP’18 ...
01.29.2019 80 - Leaderboards and Science, with Siva Reddy Originally used to entice fierce competitions in arcade games, leaderboards recently made their way into NLP research circles. Leaderboards could help mitigate some of the problems in how researchers run experiments and share results (e.g., accidentally overfitting models on a ...
01.21.2019 79 - The glass ceiling in NLP, with Natalie Schluter In this episode, Natalie Schluter talks to us about a data-driven analysis of career progression of male vs. female researchers in NLP through the lens of mentor-mentee networks based on ~20K papers in the ACL anthology. Directed edges in the ...
01.15.2019 78. Where do corpora come from?, with Matt Honnibal and Ines Montani Most NLP projects rely crucially on the quality of annotations used for training and evaluating models. In this episode, Matt and Ines of Explosion AI tell us how Prodigy can improve data annotation and model development workflows. Prodigy is an ...
01.07.2019 77. On Writing Quality Peer Reviews, with Noah A. Smith It's not uncommon for authors to be frustrated with the quality of peer reviews they receive in (NLP) conferences. In this episode, Noah A. Smith shares his advice on how to write good peer reviews. The structure Noah recommends for ...
11.27.2018 76 - Increasing In-Class Similarity by Retrofitting Embeddings with Demographics, with Dirk Hovy EMNLP 2018 paper by Dirk Hovy and Tommaso Fornaciari. https://www.semanticscholar.org/paper/Improving-Author-Attribute-Prediction-by-Linguistic-Hovy-Fornaciari/71aad8919c864f73108aafd8e926d44e9df51615 In this episode, Dirk Hovy talks about natural language as social phenomenon which can provide insights about those who generate it. For example, this paper uses retrofitted embeddings to improve ...
11.21.2018 75 - Reinforcement / Imitation Learning in NLP, with Hal Daumé III In this episode, we invite Hal Daumé to continue the discussion on reinforcement learning, focusing on how it has been used in NLP. We discuss how to reduce NLP problems into the reinforcement learning framework, and circumstances where it may ...
11.16.2018 74 - Deep Reinforcement Learning Doesn't Work Yet, with Alex Irpan Blog post by Alex Irpan titled "Deep Reinforcement Learning Doesn't Work Yet" https://www.alexirpan.com/2018/02/14/rl-hard.html In this episode, Alex Irpan talks about limitations of current deep reinforcement learning methods and why we have a long way to go before they go mainstream. ...
11.13.2018 73 - Supersense Disambiguation of English Prepositions and Possessives, with Nathan Schneider ACL 2018 paper by Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Jakob Prange, Austin Blodgett, Sarah R. Moeller, Aviram Stern, Adi Bitan, Omri Abend. In this episode, Nathan discusses how the meaning of prepositions varies, proposes a hierarchy for classifying ...
10.16.2018 72 - The Anatomy Question Answering Task, with Jordan Boyd-Graber Our first episode in a new format: broader surveys of areas, instead of specific discussions on individual papers. In this episode, we talk with Jordan Boyd-Graber about question answering. Matt starts the discussion by giving five different axes on which ...
10.12.2018 71 - DuoRC: Complex Language Understanding with Paraphrased Reading Comprehension, with Amrita Saha ACL 2018 paper by Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, Karthik Sankaranarayanan Amrita and colleagues at IBM Research introduced a harder dataset for "reading comprehension", where you have to answer questions about a given passage of text. Amrita joins ...
09.18.2018 70 - Measuring the Evolution of a Scientific Field through Citation Frames, with David Jurgens TACL 2018 paper (presented at ACL 2018) by David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Daniel Jurafsky David comes on the podcast to talk to us about citation frames. We discuss the dataset they created by painstakingly ...
09.10.2018 69 - Second language acquisition modeling, with Burr Settles A shared task held in conjunction with a NAACL 2018 workshop, organized by Burr Settles and collaborators at Duolingo. Burr tells us about the shared task. The goal of the task was to predict errors that a language learner would ...
09.04.2018 68 - Neural models of factuality, with Rachel Rudinger NAACL 2018 paper, by Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme Rachel comes on to the podcast, telling us about what factuality is (did an event happen?), what datasets exist for doing this task (a few; they made ...
08.27.2018 67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman Paper by Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Sam comes on to tell us about GLUE. We talk about the motivation behind setting up a benchmark framework for natural language understanding, how ...
08.20.2018 66 - Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods, with Jieyu Zhao NACL 2018 paper, by Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Jieyu comes on the podcast to talk about bias in coreference resolution models. This bias makes models rely disproportionately on gender when making decisions for ...
08.13.2018 65 - Event Representations with Tensor-based Compositions, with Niranjan Balasubramanian AAAI 2018 paper by Noah Weber, Niranjan Balasubramanian, and Nathanael Chambers Niranjan joins us on the podcast to tell us about his latest contribution in a line of work going back to Shank's scripts. This work tries to model sequences ...

Older episodes

NLP Highlights

What's NLP Highlights about?

Download our free app to listen on your phone

NLP Highlights episodes:

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro

113 - Managing Industry Research Teams, with Fernando Pereira

112 - Alignment of Multilingual Contextual Representations, with Steven Cao

111 - Typologically diverse, multi-lingual, information-seeking questions, with Jon Clark

110 - Natural Questions, with Tom Kwiatkowski and Michael Collins

109 - What Does Your Model Know About Language, with Ellie Pavlick

108 - Data-To-Text Generation, with Verena Rieser and Ondřej Dušek

107 - Multi-Modal Transformers, with Hao Tan and Mohit Bansal

106 - Ethical Considerations In NLP Research, with Emily Bender

105 - Question Generation, with Sudha Rao

104 - Model Distillation, with Victor Sanh and Thomas Wolf

103 - Processing Language in Social Media, with Brendan O'Connor

102 - Biomedical NLP research at the National Institute of Health with Dina Demner-Fushman

101 - The lottery ticket hypothesis, with Jonathan Frankle

100 - NLP Startups, with Oren Etzioni

99 - Evaluating Protein Transfer Learning, With Roshan Rao And Neil Thomas

98 - Analyzing Information Flow In Transformers, With Elena Voita

97 - Automated Analysis Of Historical Printed Documents, With Taylor Berg-Kirkpatrick

96 - Question Answering as an Annotation Format, with Luke Zettlemoyer

95 - Common sense reasoning, with Yejin Choi

94 - Decompositional Semantics, with Aaron White

93 - NLP/ML for clinical data, with Alistair Johnson

92 - Computational Humanities, with David Bamman

91 - (Executable) Semantic Parsing, with Jonathan Berant

90 - Research in Academia versus Industry, with Philip Resnik and Jason Baldridge

89 - Dialog Systems, with Zhou Yu

88 - A Structural Probe for Finding Syntax in Word Representations, with John Hewitt

87 - Pathologies of Neural Models Make Interpretation Difficult, with Shi Feng

86 - NLP for Evidence-based Medicine, with Byron Wallace

85 - Stress in Research, with Charles Sutton

84 - Large Teams Develop, Small Groups Disrupt, with Lingfei Wu

83 - Knowledge Base Construction, with Sebastian Riedel

82 - Visual Reasoning, with Yoav Artzi

81 - BlackboxNLP, with Afra Alishahi and Tal Linzen

80 - Leaderboards and Science, with Siva Reddy

79 - The glass ceiling in NLP, with Natalie Schluter

78. Where do corpora come from?, with Matt Honnibal and Ines Montani

77. On Writing Quality Peer Reviews, with Noah A. Smith

76 - Increasing In-Class Similarity by Retrofitting Embeddings with Demographics, with Dirk Hovy

75 - Reinforcement / Imitation Learning in NLP, with Hal Daumé III

74 - Deep Reinforcement Learning Doesn't Work Yet, with Alex Irpan

73 - Supersense Disambiguation of English Prepositions and Possessives, with Nathan Schneider

72 - The Anatomy Question Answering Task, with Jordan Boyd-Graber

71 - DuoRC: Complex Language Understanding with Paraphrased Reading Comprehension, with Amrita Saha

70 - Measuring the Evolution of a Scientific Field through Citation Frames, with David Jurgens

69 - Second language acquisition modeling, with Burr Settles

68 - Neural models of factuality, with Rachel Rudinger

67 - GLUE: A Multi-Task Benchmark and Analysis Platform, with Sam Bowman

66 - Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods, with Jieyu Zhao

65 - Event Representations with Tensor-based Compositions, with Niranjan Balasubramanian

Other shows like NLP Highlights: