Sign up to save your podcasts
Or
Share Relational Graph Transformer for Multi-Table Learning
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Relational Graph Transformer for Multi-Table Learning
This episode explores the Relational Graph Transformer paper and asks whether transformer-based models can outperform standard graph neural networks for prediction tasks over real multi-table databases such as customers, orders, products, claims, and shipments. It explains how the method turns a relational warehouse into a heterogeneous temporal graph, then builds five-part tokens for sampled neighbors that encode row features, table type, hop distance, relative time, and a learned local-structure signal. The discussion focuses on the model’s local-global attention design, where dense attention over timestamp-safe two-hop neighborhoods is paired with learned global centroids to capture broader database patterns without full all-pairs cost. It is especially interesting because it frames both the promise and the friction of relational deep learning: strong motivation to beat hand-engineered SQL features and message-passing bottlenecks, but real skepticism about whether such graph-heavy systems are practical enough for ordinary industrial stacks.
Sources:
1. Relational Graph Transformer — Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico López, Charilaos I. Kanatsoulis, Rishi Puri, Matthias Fey, Jure Leskovec, 2025
http://arxiv.org/abs/2505.10960
2. Heterogeneous Graph Transformer — Ziniu Hu, Yuxiao Dong, Kuansan Wang, Yizhou Sun, 2020
https://scholar.google.com/scholar?q=Heterogeneous+Graph+Transformer
3. Temporal Graph Networks for Deep Learning on Dynamic Graphs — Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, Michael Bronstein, 2020
https://scholar.google.com/scholar?q=Temporal+Graph+Networks+for+Deep+Learning+on+Dynamic+Graphs
4. Relational Deep Learning: Graph Representation Learning on Relational Databases — Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, Jure Leskovec, 2023
https://scholar.google.com/scholar?q=Relational+Deep+Learning%3A+Graph+Representation+Learning+on+Relational+Databases
5. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Matthias Fey, Jure Leskovec, et al., 2024
https://scholar.google.com/scholar?q=RelBench%3A+A+Benchmark+for+Deep+Learning+on+Relational+Databases
6. Graph-Bert: Only Attention is Needed for Learning Graph Representations — Jiawei Zhang, Haopeng Zhang, Congying Xia, Li Sun, 2020
https://scholar.google.com/scholar?q=Graph-Bert%3A+Only+Attention+is+Needed+for+Learning+Graph+Representations
7. Do Transformers Really Perform Bad for Graph Representation? — Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu, 2021
https://scholar.google.com/scholar?q=Do+Transformers+Really+Perform+Bad+for+Graph+Representation%3F
8. Pure Transformers are Powerful Graph Learners — Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong, 2022
https://scholar.google.com/scholar?q=Pure+Transformers+are+Powerful+Graph+Learners
9. NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs — Jinsong Chen, Kaiyuan Gao, Gaichao Li, Kun He, 2023
https://scholar.google.com/scholar?q=NAGphormer%3A+A+Tokenized+Graph+Transformer+for+Node+Classification+in+Large+Graphs
10. Representing Long-Range Context for Graph Neural Networks with Global Attention — Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, Ion Stoica, 2021
https://scholar.google.com/scholar?q=Representing+Long-Range+Context+for+Graph+Neural+Networks+with+Global+Attention
11. Recipe for a General, Powerful, Scalable Graph Transformer — Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini, 2022
https://scholar.google.com/scholar?q=Recipe+for+a+General%2C+Powerful%2C+Scalable+Graph+Transformer
12. Exphormer: Sparse Transformers for Graphs — Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, Ali Kemal Sinop, 2023
https://scholar.google.com/scholar?q=Exphormer%3A+Sparse+Transformers+for+Graphs
13. Centroid Transformers: Learning to Abstract with Attention — Lemeng Wu, Xingchao Liu, Qiang Liu, 2021
https://scholar.google.com/scholar?q=Centroid+Transformers%3A+Learning+to+Abstract+with+Attention
14. Learning Efficient Positional Encodings with Graph Neural Networks — Charilaos I. Kanatsoulis et al., 2025
https://arxiv.org/abs/2502.01122
15. ContextGNN: Beyond Two-Tower Recommendation Systems — Yiwen Yuan et al., 2024
https://arxiv.org/abs/2411.19513
16. RelGNN: Composite Message Passing for Relational Deep Learning — Tianlang Chen, Charilaos Kanatsoulis, Jure Leskovec, 2025
https://arxiv.org/abs/2502.06784
17. Are Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNs — Jeongwhan Choi et al., 2025
https://arxiv.org/abs/2511.13010
18. Beyond Message Passing: Neural Graph Pattern Machine — Zehong Wang et al., 2025
https://arxiv.org/abs/2501.18739
19. Transformers Meet Relational Databases — Jakub Peleska and Gustav Sir, 2024
https://arxiv.org/abs/2412.05218
20. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan et al., 2025
https://arxiv.org/abs/2510.06377
21. Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification — Zijie Zhou et al., 2024
https://arxiv.org/abs/2412.15302
22. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3
View all episodes
By mcgrofThis episode explores the Relational Graph Transformer paper and asks whether transformer-based models can outperform standard graph neural networks for prediction tasks over real multi-table databases such as customers, orders, products, claims, and shipments. It explains how the method turns a relational warehouse into a heterogeneous temporal graph, then builds five-part tokens for sampled neighbors that encode row features, table type, hop distance, relative time, and a learned local-structure signal. The discussion focuses on the model’s local-global attention design, where dense attention over timestamp-safe two-hop neighborhoods is paired with learned global centroids to capture broader database patterns without full all-pairs cost. It is especially interesting because it frames both the promise and the friction of relational deep learning: strong motivation to beat hand-engineered SQL features and message-passing bottlenecks, but real skepticism about whether such graph-heavy systems are practical enough for ordinary industrial stacks.
Sources:
1. Relational Graph Transformer — Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico López, Charilaos I. Kanatsoulis, Rishi Puri, Matthias Fey, Jure Leskovec, 2025
http://arxiv.org/abs/2505.10960
2. Heterogeneous Graph Transformer — Ziniu Hu, Yuxiao Dong, Kuansan Wang, Yizhou Sun, 2020
https://scholar.google.com/scholar?q=Heterogeneous+Graph+Transformer
3. Temporal Graph Networks for Deep Learning on Dynamic Graphs — Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, Michael Bronstein, 2020
https://scholar.google.com/scholar?q=Temporal+Graph+Networks+for+Deep+Learning+on+Dynamic+Graphs
4. Relational Deep Learning: Graph Representation Learning on Relational Databases — Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, Jure Leskovec, 2023
https://scholar.google.com/scholar?q=Relational+Deep+Learning%3A+Graph+Representation+Learning+on+Relational+Databases
5. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Matthias Fey, Jure Leskovec, et al., 2024
https://scholar.google.com/scholar?q=RelBench%3A+A+Benchmark+for+Deep+Learning+on+Relational+Databases
6. Graph-Bert: Only Attention is Needed for Learning Graph Representations — Jiawei Zhang, Haopeng Zhang, Congying Xia, Li Sun, 2020
https://scholar.google.com/scholar?q=Graph-Bert%3A+Only+Attention+is+Needed+for+Learning+Graph+Representations
7. Do Transformers Really Perform Bad for Graph Representation? — Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu, 2021
https://scholar.google.com/scholar?q=Do+Transformers+Really+Perform+Bad+for+Graph+Representation%3F
8. Pure Transformers are Powerful Graph Learners — Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, Seunghoon Hong, 2022
https://scholar.google.com/scholar?q=Pure+Transformers+are+Powerful+Graph+Learners
9. NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs — Jinsong Chen, Kaiyuan Gao, Gaichao Li, Kun He, 2023
https://scholar.google.com/scholar?q=NAGphormer%3A+A+Tokenized+Graph+Transformer+for+Node+Classification+in+Large+Graphs
10. Representing Long-Range Context for Graph Neural Networks with Global Attention — Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, Ion Stoica, 2021
https://scholar.google.com/scholar?q=Representing+Long-Range+Context+for+Graph+Neural+Networks+with+Global+Attention
11. Recipe for a General, Powerful, Scalable Graph Transformer — Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini, 2022
https://scholar.google.com/scholar?q=Recipe+for+a+General%2C+Powerful%2C+Scalable+Graph+Transformer
12. Exphormer: Sparse Transformers for Graphs — Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, Ali Kemal Sinop, 2023
https://scholar.google.com/scholar?q=Exphormer%3A+Sparse+Transformers+for+Graphs
13. Centroid Transformers: Learning to Abstract with Attention — Lemeng Wu, Xingchao Liu, Qiang Liu, 2021
https://scholar.google.com/scholar?q=Centroid+Transformers%3A+Learning+to+Abstract+with+Attention
14. Learning Efficient Positional Encodings with Graph Neural Networks — Charilaos I. Kanatsoulis et al., 2025
https://arxiv.org/abs/2502.01122
15. ContextGNN: Beyond Two-Tower Recommendation Systems — Yiwen Yuan et al., 2024
https://arxiv.org/abs/2411.19513
16. RelGNN: Composite Message Passing for Relational Deep Learning — Tianlang Chen, Charilaos Kanatsoulis, Jure Leskovec, 2025
https://arxiv.org/abs/2502.06784
17. Are Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNs — Jeongwhan Choi et al., 2025
https://arxiv.org/abs/2511.13010
18. Beyond Message Passing: Neural Graph Pattern Machine — Zehong Wang et al., 2025
https://arxiv.org/abs/2501.18739
19. Transformers Meet Relational Databases — Jakub Peleska and Gustav Sir, 2024
https://arxiv.org/abs/2412.05218
20. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan et al., 2025
https://arxiv.org/abs/2510.06377
21. Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification — Zijie Zhou et al., 2024
https://arxiv.org/abs/2412.15302
22. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3