This study investigates the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, revealing distinct patterns in encoders and decoders. Initial training expands dimensionality, while later training refines into more compact representations.

https://arxiv.org/abs//2311.05928

YouTube: https://www.youtube.com/@ArxivPapers

TikTok: https://www.tiktok.com/@arxiv_papers

Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016

Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers