How AI Is Built

Data Integration and Ingestion for AI & LLMs, Architecting Data Flows | changelog 3

Listen Later

In this episode, Kirk Marple, CEO and founder of Graphlit, shares his expertise on building efficient data integrations.

Kirk breaks down his approach using relatable concepts:

  1. The "Two-Sided Funnel": This model streamlines data flow by converting various data sources into a standard format before distributing it.
  2. Universal Data Streams: Kirk explains how he transforms diverse data into a single, manageable stream of information.
  3. Parallel Processing: Learn about the "competing consumer model" that allows for faster data handling.
  4. Building Blocks for Success: Discover the importance of well-defined interfaces and actor models in creating robust data systems.
  5. Tech Talk: Kirk discusses data normalization techniques and the potential shift towards a more streamlined "Kappa architecture."
  6. Reusable Patterns: Find out how Kirk's methods can speed up the integration of new data sources.

    7. Kirk Marple:

    • LinkedIn
    • X (Twitter)
    • Graphlit
    • Graphlit Docs

      • Nicolay Gerold:

      • ⁠LinkedIn⁠
      • ⁠X (Twitter)

        • Chapters

        00:00 Building Integrations into Different Tools

        00:44 The Two-Sided Funnel Model for Data Flow

        04:07 Using Well-Defined Interfaces for Faster Integration

        04:36 Managing Feeds and State with Actor Models

        06:05 The Importance of Data Normalization

        10:54 Tech Stack for Data Flow

        11:52 Progression towards a Kappa Architecture

        13:45 Reusability of Patterns for Faster Integration

        data integration, data sources, data flow, two-sided funnel model, canonical format, stream of ingestible objects, competing consumer model, well-defined interfaces, actor model, data normalization, tech stack, Kappa architecture, reusability of patterns

        ...more
        View all episodesView all episodes
        Download on the App Store
        How AI Is BuiltBy Nicolay Gerold