🎙 Sign up for The Future of Machine Learning Livestream!
Sign up
🗞️ Signup for Our Newsletter
Sign Up

Episode 158:

The Orchestration Layer as the Data Platform Control Plane With Nick Schrock of Dagster Labs

October 4, 2023

This week on The Data Stack Show, Eric and Kostas chat with Nick Schrock, Founder of Dagster Labs. During the episode, Nick discusses his background at Facebook and his involvement in successful open-source technologies like React and GraphQL. Nick explains the mismatch between the complexity and available tools in the data infrastructure space, leading him to start Dagster Labs. The group also talks about the challenges and fragmentation in the data engineering industry, the need for better abstraction layers, the role of orchestrators, comparing it to GraphQL’s role in product engineering, the importance of data orchestration in the future of data infrastructure and engineering, and more.

Notes:

Highlights from this week’s conversation include:

  • Nick’s background and journey in data (2:28)
  • Founding Dagster Labs (7:50)
  • The evolution of data engineering (12:32)
  • Fragmentation in data infrastructure (15:04)
  • The role of orchestration in data platforms (19:53)
  • The importance of operational tools for data pipelines (25:01)
  • Lessons learned from working with GraphQL (26:19)
  • The role of the orchestrator in data engineering (34:51)
  • The boundaries between data infrastructure and product engineering (37:33)
  • Different orchestrators in the data infrastructure landscape(42:03)
  • The role of MLOps in data engineering (46:04)
  • Data Quality and Orchestration (51:04)
  • Future of Data Teams and Orchestration (54:27)
  • Final thoughts and takeaways from (58:01)

 

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

About the Podcast

Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

To keep up to date with our future episodes, subscribe to our podcast on Apple, Spotify, Google, or the player of your choice.

Sign Up for The Data Stack Show Newsletter

Get a monthly newsletter from The Data Stack Show team with a TL;DR of the previous month’s shows, a sneak peak at upcoming episodes, and curated links from Eric, John, & show guests. Follow on our Substack below.

Sign Up

⚡ Powered by

© RudderStack Inc.