🎙 Sign up for The Future of Machine Learning Livestream!
Sign up
🗞️ Signup for Our Newsletter
Sign Up

Episode 246:

AI, Abstractions, and the Future of Data Engineering with Pete Hunt of Dagster

May 28, 2025

This week on The Data Stack Show, Brooks and John welcome Pete Hunt, CEO of Dagster Labs. During this conversation, Pete takes listeners on a fascinating journey through the evolution of data platforms, sharing insights from his experiences at Facebook and Twitter. Pete discusses the critical challenges of managing data complexity, emphasizing the importance of software engineering best practices and strong abstractions in data orchestration. He explores how emerging technologies like AI are transforming data workflows, breaking down traditional team boundaries, and enabling more collaborative, self-service data engineering. The conversation also highlights Dagster’s innovative approach to building a data control plane, with a focus on empowering different stakeholders through flexible, composable tools. Listeners will gain valuable perspectives on the future of data engineering, the role of AI in development, the importance of creating adaptable, user-friendly data infrastructure, and so much more.

Notes:

Highlights from this week’s conversation include:

  • Pete’s Background and Journey in Data (1:36)
  • Evolution of Data Practices (3:02)
  • Integration Challenges with Acquired Companies (5:13)
  • Trust and Safety as a Service (8:12)
  • Transition to Dagster (11:26)
  • Value Creation in Networking (14:42)
  • Observability in Data Pipelines (18:44)
  • The Era of Big Complexity (21:38)
  • Abstraction as a Tool for Complexity (24:41)
  • Composability and Workflow Engines (28:08)
  • The Need for Guardrails (33:13)
  • AI in Development Tools (36:24)
  • Internal Components Marketplace (40:14)
  • Reimagining Data Integration (43:03)
  • Importance of Abstraction in Data Tools (46:17)
  • Parting Advice for Listeners and Closing Thoughts (48:01)

 

The Data Stack Show is a weekly podcast powered by RudderStack, customer data infrastructure that enables you to deliver real-time customer event data everywhere it’s needed to power smarter decisions and better customer experiences. Each week, we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

About the Podcast

Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

To keep up to date with our future episodes, subscribe to our podcast on Apple, Spotify, Google, or the player of your choice.

Sign Up for The Data Stack Show Newsletter

Get a monthly newsletter from The Data Stack Show team with a TL;DR of the previous month’s shows, a sneak peak at upcoming episodes, and curated links from Eric, John, & show guests. Follow on our Substack below.

Sign Up

⚡ Powered by

© RudderStack Inc.