🎙 Sign up for The Future of Machine Learning Livestream!
Sign up
🗞️ Signup for Our Newsletter
Sign Up

Episode 184:

Kafka Streams and Operationalizing Event Driven Applications with Apurva Mehta of Responsive

April 3, 2024

This week on The Data Stack Show, Eric and Kostas chat with Apurva Mehta, Co-Founder and CEO of Responsive, about event-driven applications and the necessary infrastructure. Apruva shares his journey from LinkedIn to Confluent and eventually founding Responsive, focusing on managing event-driven applications in the cloud. The discussion covers the definition of event-driven applications, the significance of latency and state in event processing, and the evolution of Kafka and Kafka Streams. They also explore the challenges of managing Kafka in production, the developer experience with Kafka Streams, and the operational complexities of running distributed stateful applications. Apruva highlights Responsive’s approach to simplifying the management of these applications, the potential for innovation in event-driven architectures, and more. 

Notes:

Highlights from this week’s conversation include:

  • Apruva’s background in streaming technology (0:48)
  • Developer experience and Kafka streams (2:47)
  • Motivation to bootstrap a startup (4:09)
  • Meeting the Confluent founders and early work at Confluent (6:59)
  • Projects at Confluent and transition to engineering management (10:34)
  • Overview of Responsive and event-driven applications (12:55)
  • Defining event-driven applications (15:33)
  • Importance of latency and state in event-driven applications (18:54)
  • Low Latency and Stateful Processing (21:52)
  • In-Memory Storage and Evolution of Kafka (25:02)
  • Motivation for KSQL and Kafka Streams (29:46)
  • Category Creation and Database-like Interface (34:33)
  • Developer Experience with Kafka and Kafka Streams (38:50)
  • Kafka Streams Functionality and Operational Challenges (41:44)
  • Metrics and Tuning Configurations (43:33)
  • Architecture and Decoupling in Kafka Streams (45:39)
  • State Storage and Transition from RocksDB (47:48)
  • Future of Event-Driven Architectures (56:30)
  • Final thoughts and takeaways (57:36)

 

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

About the Podcast

Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

To keep up to date with our future episodes, subscribe to our podcast on Apple, Spotify, Google, or the player of your choice.

Sign Up for The Data Stack Show Newsletter

Get a monthly newsletter from The Data Stack Show team with a TL;DR of the previous month’s shows, a sneak peak at upcoming episodes, and curated links from Eric, John, & show guests. Follow on our Substack below.

Sign Up

⚡ Powered by

© RudderStack Inc.