This episode explores how a transformer trained on raw bank transaction histories can model customer behavior for financial product recommendation, and why that may outperform pipelines built from hand-engineered tabular features alone. It explains the paper’s core idea of turning each transaction into a tokenized sequence that mixes inflow or outflow, amount buckets, calendar signals, source metadata, and natural-language merchant descriptions, then pretraining the model with self-supervised learning to produce reusable customer embeddings. The discussion argues that transaction text and long-range patterns such as pay cycles, bill timing, and abrupt behavior changes carry signal that conventional tabular systems often flatten away, while a practical deployment can still combine learned embeddings with legacy banking features downstream. A listener would find it interesting because it connects transformer-style representation learning to a concrete banking use case and shows how foundation-model ideas can be adapted to messy, real-world financial behavior.

Sources:

1. Your Spending Needs Attention: Modeling Financial Habits with Transformers — D. T. Braithwaite, Misael Cavalcanti, R. Austin McEver, Hiroto Udagawa, Daniel Silva, Rohan Ramanath, Felipe Meneses, Arissa Yoshida, Evan Wingert, Matheus Ramos, Brian Zanfelice, Aman Gupta, 2025

http://arxiv.org/abs/2507.23267

2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2019

https://scholar.google.com/scholar?q=BERT%3A+Pre-training+of+Deep+Bidirectional+Transformers+for+Language+Understanding

3. CoLES: Contrastive Learning for Event Sequences with Self-Supervision — Dmitrii Babaev, Ivan Kireev, Nikita Ovsov, Mariya Ivanova, Gleb Gusev, Ivan Nazarov, Alexander Tuzhilin, 2020

https://scholar.google.com/scholar?q=CoLES%3A+Contrastive+Learning+for+Event+Sequences+with+Self-Supervision

4. Dynamic Customer Embeddings for Financial Service Applications — Nima Chitsazan, Samuel Sharpe, Dwipam Katariya, Qianyu Cheng, Karthik Rajasethupathy, 2021

https://scholar.google.com/scholar?q=Dynamic+Customer+Embeddings+for+Financial+Service+Applications

5. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences — Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, Jason Wong, 2024

https://scholar.google.com/scholar?q=Towards+a+Foundation+Purchasing+Model%3A+Pretrained+Generative+Autoregression+on+Transaction+Sequences

6. Self-Attentive Sequential Recommendation — Wang-Cheng Kang, Julian McAuley, 2018

https://scholar.google.com/scholar?q=Self-Attentive+Sequential+Recommendation

7. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer — Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang, 2019

https://scholar.google.com/scholar?q=BERT4Rec%3A+Sequential+Recommendation+with+Bidirectional+Encoder+Representations+from+Transformer

8. S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization — Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, Ji-Rong Wen, 2020

https://scholar.google.com/scholar?q=S%5E3-Rec%3A+Self-Supervised+Learning+for+Sequential+Recommendation+with+Mutual+Information+Maximization

9. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba — Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, Wenwu Ou, 2019

https://scholar.google.com/scholar?q=Behavior+Sequence+Transformer+for+E-commerce+Recommendation+in+Alibaba

10. Text Is All You Need: Learning Language Representations for Sequential Recommendation — Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, Julian McAuley, 2023

https://scholar.google.com/scholar?q=Text+Is+All+You+Need%3A+Learning+Language+Representations+for+Sequential+Recommendation

11. PinnerFormer: Sequence Modeling for User Representation at Pinterest — Nikil Pancha, Andrew Zhai, Jure Leskovec, Charles Rosenberg, 2022

https://scholar.google.com/scholar?q=PinnerFormer%3A+Sequence+Modeling+for+User+Representation+at+Pinterest

12. Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models — Chengkai Liu et al., 2024

https://scholar.google.com/scholar?q=Mamba4Rec%3A+Towards+Efficient+Sequential+Recommendation+with+Selective+State+Space+Models

13. SSD4Rec: A Structured State Space Duality Model for Efficient Sequential Recommendation — Haohao Qu et al., 2024

https://scholar.google.com/scholar?q=SSD4Rec%3A+A+Structured+State+Space+Duality+Model+for+Efficient+Sequential+Recommendation

14. DynLLM: When Large Language Models Meet Dynamic Graph Recommendation — Ziwei Zhao et al., 2024

https://scholar.google.com/scholar?q=DynLLM%3A+When+Large+Language+Models+Meet+Dynamic+Graph+Recommendation

15. Personalized Elastic Embedding Learning for On-Device Recommendation — Ruiqi Zheng et al., 2023

https://scholar.google.com/scholar?q=Personalized+Elastic+Embedding+Learning+for+On-Device+Recommendation

16. A Survey on Deep Tabular Learning — Shriyank Somvanshi et al., 2024

https://scholar.google.com/scholar?q=A+Survey+on+Deep+Tabular+Learning

17. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp3

18. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp3