🎙 Sign up for The Future of Machine Learning Livestream!
Sign up
🗞️ Signup for Our Newsletter
Sign Up

Episode 148:

Exploring the Intersection of DAGs, ML Code, and Complex Code Bases: An Elegant Solution Unveiled with Stefan Krawczyk of DAGWorks

July 26, 2023

This week on The Data Stack Show, Eric and Kostas chat with Stefan Krawczyk, the Co-Creator of Hamilton and Co-Founder of DAGWorks. During the episode, Stefan shares his journey working in data for NextDoor, StitchFix, and others on his journey to founding DAGWorks. The conversation also includes much discussion around Stefan’s creation of Hamilton, how the platform works with definitions and time-series data, how it improves pipelines, what makes Hamilton an ML oriented framework, the importance of unit testing, and more.

Notes:

Highlights from this week’s conversation include:

  • Stefan’s background in data (2:39)
  • What is DAGWorks? (3:55)
  • How building point solutions influenced Stefan’s journey (5:03)
  • Solving the tooling problems of self-service at an organization (11:44)
  • Creating Hamilton (15:53)
  • How Hamilton works with definitions and time-series data (19:34)
  • What makes Hamilton an ML oriented framework? (23:39)
  • Navigating the differences between ML teams and other data teams (26:27)
  • Understanding the fundamentals of Hamilton (28:25)
  • Dealing with types and conflicts in programming (33:18)
  • How Hamilton helps improve pipelines and maintaining data (37:11)
  • Why unit testing is important for a data scientist (44:54)
  • The ups and downs of founding building a data solution (46:32)
  • Connecting with DAGWorks and trying out Hamilton (50:01)
  • Final thoughts and takeaways (52:46)

 

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

About the Podcast

Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

To keep up to date with our future episodes, subscribe to our podcast on Apple, Spotify, Google, or the player of your choice.

Sign Up for The Data Stack Show Newsletter

Get a monthly newsletter from The Data Stack Show team with a TL;DR of the previous month’s shows, a sneak peak at upcoming episodes, and curated links from Eric, John, & show guests. Follow on our Substack below.

Sign Up

⚡ Powered by

© RudderStack Inc.