Elixir Wizards

Explorer: Data Frames in Elixir with Chris Grainger


Listen Later

In this episode of Elixir Wizards, Charles Suggs sits down with Chris Grainger, co-founder and CTO of Amplified and creator of the Explorer library. Chris explains how Explorer brings the familiar data-frame workflows of R’s dplyr and Python’s pandas into the Elixir world. We explore (pun intended!) how Explorer integrates with Ecto, Nx, and LiveView to build end-to-end data pipelines without leaving the BEAM, and how features like lazy evaluation and distributed frames let you tackle large datasets.

Whether you’re generating reports or driving interactive charts in LiveView, Explorer makes tabular data accessible to every Elixir developer. We wrap up by looking ahead to SQL-style backends, ADBC connectivity, and other features on the Explorer roadmap.

Key topics discussed in this episode:
  • dplyr- and pandas-inspired data manipulation in Elixir
  • Polars integration via Rust NIFs for blazing performance
  • Immutable data frames and BEAM-friendly concurrency
  • Lazy evaluation to work with arbitrarily large tables
  • Distributed data-frame support for multi-node processing
  • Seamless integration with Ecto schemas and queries
  • Zero-copy interoperability between Explorer and Nx tensors
  • Apache Arrow and ADBC protocols for cross-language I/O
  • Exploring SQL-style backends for remote query execution
  • Building interactive dashboards and charts in LiveView
  • Consolidating ETL workflows into a single Elixir API
  • Streaming data pipelines for memory-efficient processing
  • Tidy data principles and behavior-based API design
  • Real-world use cases: report generation, patent analysis, and more
  • Future roadmap: new backends, query optimizations, and community plugins
  • Links mentioned:

    https://hexdocs.pm/explorer/Explorer.html

    https://www.amplified.ai/
    https://www.r-project.org/
    https://vita.had.co.nz/papers/tidy-data.pdf
    https://www.tidyverse.org/
    https://www.python.org/
    https://dplyr.tidyverse.org/
    https://go.dev/
    https://hexdocs.pm/nx/Nx.html
    https://github.com/pola-rs/polars
    https://github.com/rusterlium/rustler
    https://www.rust-lang.org/
    https://www.postgresql.org/
    https://hexdocs.pm/ecto/Ecto.html
    https://www.elastic.co/elasticsearch
    https://arrow.apache.org/
    Chris Grainger & Chris McCord Keynote ElixirConf 2024: https://youtu.be/4qoHPh0obv0
    https://dbplyr.tidyverse.org/
    https://spark.posit.co/
    https://hexdocs.pm/pythonx/Pythonx.html
    https://hexdocs.pm/vega_lite/VegaLite.html
    10 Minutes to Explorer: https://hexdocs.pm/explorer/exploring_explorer.html
    https://github.com/elixir-nx/scholar
    https://scikit-learn.org/stable/
    https://github.com/cigrainger
    https://erlef.org/slack-invite/erlef
    https://bsky.app/profile/cigrainger.bsky.social
    https://github.com/cigrainger

    ...more
    View all episodesView all episodes
    Download on the App Store

    Elixir WizardsBy SmartLogic LLC

    • 4.9
    • 4.9
    • 4.9
    • 4.9
    • 4.9

    4.9

    22 ratings


    More shows like Elixir Wizards

    View all
    Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

    Software Engineering Radio - the podcast for professional software developers

    272 Listeners

    The Changelog: Software Development, Open Source by Changelog Media

    The Changelog: Software Development, Open Source

    284 Listeners

    Software Engineering Daily by Software Engineering Daily

    Software Engineering Daily

    621 Listeners

    Python Bytes by Michael Kennedy and Brian Okken

    Python Bytes

    215 Listeners

    Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

    Syntax - Tasty Web Development Treats

    987 Listeners

    CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

    CoRecursive: Coding Stories

    189 Listeners

    Elixir Outlaws by Elixir Outlaws

    Elixir Outlaws

    24 Listeners

    The Stack Overflow Podcast by The Stack Overflow Podcast

    The Stack Overflow Podcast

    62 Listeners

    Thinking Elixir Podcast by ThinkingElixir.com

    Thinking Elixir Podcast

    33 Listeners

    Beam Radio by Lars Wikman

    Beam Radio

    11 Listeners

    Software Unscripted by Richard Feldman

    Software Unscripted

    26 Listeners

    Oxide and Friends by Oxide Computer Company

    Oxide and Friends

    47 Listeners

    Elixir Mentor by Jacob Luetzow

    Elixir Mentor

    2 Listeners

    Risky Business with Nate Silver and Maria Konnikova by Pushkin Industries

    Risky Business with Nate Silver and Maria Konnikova

    261 Listeners

    The Pragmatic Engineer by Gergely Orosz

    The Pragmatic Engineer

    63 Listeners