Main Greg Channel!

Demystifying attention, the key mechanism inside transformers and LLMs.

Demystifying self-attention, multiple heads, and cross-attention.

The first pass for the translated subtitles here is machine-generated, and therefore notably imperfect. To contribute edits or fixes, visit https://www.criblate.com

Звуковая дорожка на русском языке: Влад Бурмистров.

And yes, at 22:00 (and elsewhere), "breaks" is a typo.

Here are a few other relevant resources

Build a GPT from scratch, by Andrej Karpathy

If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic:

If you're interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources.

Site with exercises related to ML programming and GPTs

History of language models by Brit Cruise,  @ArtOfTheProblem 

An early paper on how directions in embedding spaces have meaning:

Timestamps:

These animations are largely made using a custom Python library, manim. See the FAQ comments here:

All code for specific videos is visible here:

The music is by Vincent Rubinetti.

3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly.

Mailing list: https://3blue1brown.substack.com

Unpacking the multilayer perceptrons in a transformer, and how they may store facts

AI Alignment forum post from the Deepmind researchers referenced at the video's start:

Anthropic posts about superposition referenced near the end:

Some added resources for those interested in learning more about mechanistic interpretability, offered by Neel Nanda

Mechanistic interpretability paper reading list

Getting started in mechanistic interpretability

An interactive demo of sparse autoencoders (made by Neuronpedia)

Coding tutorials for mechanistic interpretability (made by ARENA)

Звуковая дорожка на русском языке: Влад Бурмистров.

Sections:

These animations are largely made using a custom Python library, manim. See the FAQ comments here:

All code for specific videos is visible here:

The music is by Vincent Rubinetti.

3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly.

Mailing list: https://3blue1brown.substack.com

The example-driven, practical walkthrough of Large Language Models and their growing list of related features, as a new entry to my general audience series on LLMs. In this more practical followup, I take you through the many ways I use LLMs in my own life.

Chapters

Links

Tiktokenizer https://tiktokenizer.vercel.app/

OpenAI's ChatGPT https://chatgpt.com/

Anthropic's Claude https://claude.ai/

Google's Gemini https://gemini.google.com/

xAI's Grok https://grok.com/

Perplexity https://www.perplexity.ai/

Google's NotebookLM https://notebooklm.google.com/

Cursor https://www.cursor.com/

Histories of Mysteries AI podcast on Spotify https://open.spotify.com/show/3K4LRyMCP44kBbiOziwJjb

The visualization UI I was using in the video: https://excalidraw.com/

The specific file of Excalidraw we built up: https://drive.google.com/file/d/1DN3LU3MbKI00udxoS-W5ckCHq99V0Uqs/view?usp=sharing

Discord channel for Eureka Labs and this video: https://discord.gg/3zy8kqD9Cp

Educational Use Licensing

The pain of missing that 1 black key that makes the beginning so catchy 🤦‍♂️

#GenshinImpact #Thunderstruck #WindsongLyre #Xinyan #Gadget #song #ACDC

Want to know how google docs would work, if it was decentralised and therefore better? Lets make one!

In this video I implement a text CRDT (That is, a conflict-free replicating data type) for collaborative text editing, from scratch. And I explain how it all works along the way.

Which CRDT? FugueMax! Which is the gold standard for text editing. And instead of doing it the tree way, like in the paper, I show how to implement it as a list - which is essential if you want to optimise the algorithm.

Here's the code I wrote on camera: https://github.com/josephg/crdt-from-scratch

The FugueMax paper is here: https://arxiv.org/abs/2305.00583

And here's the blog post talking about how to make this algorithm go really fast: https://josephg.com/blog/crdts-go-brrr/

00:00:00 Intro

Functional or Combinator Parsing explained by Professor Graham Hutton.

Professor Hutton's Functional Parsing Library: http://bit.ly/C_FunctParsLib

https://www.facebook.com/computerphile

This video was filmed and edited by Sean Riley.

Computer Science at the University of Nottingham: https://bit.ly/nottscomputer

Computerphile is a sister project to Brady Haran's Numberphile. More at http://www.bradyharan.com

Listen all Songs of the Album “Musik Mit Hertz” here: https://youtube.com/playlist?list=PLd657Dme_2QYh0VyRW7ZfE-MkSlenVYdP&si=jQwCxHGAfJjJ0hMu

LYRICS:

We do not make money from these videos, all income and rights go to the respective artist or label.

a simple question with a very complicated answer, there is no satisfying single answer at the end of this video, but I hope it will help you understand the complexities behind how best to position yourself for one of the most important activities you'll be doing during your day, your work. by getting this right, you will improve your concentration and performance. while no two rooms are alike, try your best and take a moment to look around you, feel the energy and environment around, and be sure you're sat in the best possible spot in your office!

This episode of our Weekly Guided Meditation podcast is a practice in mindfulness of the body. Rooted in the First Foundation of Mindfulness, this is a practice that helps us build awareness, understanding, and clarity of the body as we see it in the present moment.

The period of meditation begins with a few minutes of breath awareness. Mindfulness of the breath is one aspect of mindfulness of the body, and a great place to start. The breath is always with us, always changing, and serves as a powerful object of awareness.

We then open the attention up quite a bit, tuning into other physical sensations in the body. We rest in a relaxing open awareness of the body practice, noticing what is arising and passing in the body. Without judgement, we simply observe the sensations in the body.

Mindfulness of the body is the first of the Four Establishments of Mindfulness, and a foundational Buddhist meditation practice. We can build insight into the nature of our experience, namely the Three Marks of Existence. It also can be a calming and relaxing practice that helps us change our relationships with our bodies.

You can always visit us on Instagram at https://instagram.com/oneminddharma or email us at [email protected] with any questions. We also have a Facebook group at https://web.facebook.com/groups/TESWmeditation/.

For some more reading and resources on mindfulness of the physical body, you can read these articles and try these meditations:

To support this podcast, visit https://oneminddharma.com/donate/

Today I offer a guided meditation on bringing the qualities of loving-kindness and gentleness to your mindfulness practice. This practice accompanies the dharma talk "Talk: Self-Compassion & Mindfulness."

For full show notes, links, and contact information, visit https://OneMindDharma.com/.

For individual mindfulness counseling sessions, please visit https://themindfulcounselor.me