In this episode, we invite John Hewitt to discuss his take on how to probe word embeddings for syntactic information. The basic idea is to project word embeddings to a vector space where the L2 distance between a pair of words in a sentence approximates the number of hops between them in the dependency tree. The proposed method shows that ELMo and BERT representations, trained with no syntactic supervision, embed many of the unlabeled, undirected dependency attachments between words in the same sentence.

Paper: https://nlp.stanford.edu/pubs/hewitt2019structural.pdf

GitHub repository: https://github.com/john-hewitt/structural-probes

Blog post: https://nlp.stanford.edu/~johnhew/structural-probe.html

Twitter thread: https://twitter.com/johnhewtt/status/1114252302141886464

John's homepage: https://nlp.stanford.edu/~johnhew/