I am so excited to host Dmitry Kan on the Weaviate Podcast!! Dmitry is a world class expert on emerging trends in search technology! This podcast reflects on Dmitry's latest characterization of the field, the Neural Search Pyramid. This describes the different components involved with building a Deep Learning-powered Search experience from the Approximate Nearest Neighbor index algorithms, to Database functionality,  LLM orchestration, Vectorization optimization, Data preprocessing, User Interface, and many more! We also concluded the podcast with an interesting debate around renaming "Vector Search" to something else that reaches a broader audience. I really hope you enjoy the podcast, thank you so much for listening! Please see the links below to Dmitry's recent content and the Weaviate Podcast Search App!

Links:

Dmitry's Keynote at Haystack Europe 2022, Where Vector Search is Taking Us - https://www.youtube.com/watch?v=2o8-dX__EgU

Dmitry's latest blog post on Neural Search Frameworks: A Head-to-Head Comparison - https://dmitry-kan.medium.com/neural-search-frameworks-a-head-to-head-comparison-976aa6662d20.

Search through this episode of the Weaviate Podcast! - https://github.com/weaviate/weaviate-podcast-search

Chapters

0:00 Neural Search Pyramid Visual

0:40 Weaviate Podcast Search!

1:35 Welcome Dmitry!!

2:02 Where is Vector Search taking us?

5:40 Retail and Search

11:02 Neural Search Frameworks

17:10 Data Preprocessing, e.g. PDF to Text / OCR

24:15 Vectorizing Data

31:18 ANN Index and Database Entanglement

37:25 Hardware Accelerators for Vector Search

46:02 Reader Layers, Q&A, Ranking, …

51:20 ChatGPT in Neural Search Frameworks

1:03:40 Search Result Summarization with ChatGPT

1:12:55 User Interfaces for Neural Search

1:26:30 Renaming “Vector Search”

1:46:10 Thank you Dmitry!!