Share #111: Formal Algorithms for Transformers

Copy link

April 04, 2023

#111: Formal Algorithms for Transformers

38 minutes

勤務先への脅威に怯える森田が Transformer を復習しました。ご意見ご感想などはおたより投書箱や Reddit にお寄せください。iTunes のレビューや星も歓迎です。

今回は録音に際し Adobe Podcast (beta) のバグを引き当ててしまい、向井と森田の音声トラックがずれてしまいました。ごめんなさい。次回からは non-beta の手堅いツールで録音しようと思います・・・。

[2207.09238] Formal Algorithms for Transformers

#15 – Neural Machine Translation by Jointly Learning to Align and Translate

#38 – Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

#51 – Attention Is All You Need

#53 – BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jay Alammar – YouTube

GitHub – openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI’s models.

GitHub – karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs.

Let’s build GPT: from scratch, in code, spelled out. – YouTube

...more

By Hajime Morrita, Jun Mukai

66 ratings