Turning All Data at Grofers into Live Event Streams

Collecting and storing data

Grofers is based on AWS services and uses RedShift as its primary warehouse. Ingestion occurs through batch jobs and streaming jobs. AirFlow is utilized for the orchestration layer to help manage those jobs. They use a HUDI lake for all their source data and have a spark cluster running to process all the different event data and a lot of their MLAI workflows. “Our people query these data marks on Redshift primarily using an open source tool called Redash, and we also have a self-serve analytics use case where we use Tableau.”

Because Grofers is a transactional business, their source databases are mostly Postgres and MySQL. They have a microservice architecture with different services like card orders, last mile delivery, and different e-commerce components like a delivery app, a picker and an app shopper. A lot of those services have their own independent databases. “We capture the replication logs from these databases and we use Debezium which captures those CDC changes and dumps into a Kafka stream,” he said. “From Kafka we dump it into a raw location in S3 where we process it and convert it into a HUDI lake.”

Satyam added, “We use RudderStack to basically capture all of our impression needs on our consumer application.”

When asked about the comparison of the volume of event data versus transactional data collected, Satyam observed that, no question, the volume of event data is much higher. “We capture around five to six billion events every month,” he said. “Whereas our transactional data, we must be generating some terabytes of data every month. The scale of the event data is on a completely different level.”

For the transactional data they are using CDC while for event data they take data from vendors dumped into S3 in a raw format, convert it into compressed files partitioned by date that can be accessed through Redshift Spectrum. “We don’t even keep it in Redshift because of the size of the data.”

Challenges faced

Satyam identified some common use cases inside the company and some of the challenges that accompany that. Inside Grofers, there’s a common engineering team that manages the data product and the data platforms, but each team has its own data analysts in a decentralized way. He offered an example of a way that event data are used. “When they have to test rolling out a feature, they want to see how the users are using it, they are running an A/B test and they want to see the conversions happening on it.” Other examples he noted were sponsored product recommendations and personalized homepage feeds that use data from past purchases and other consumer behavior.

With a decentralized data analysis structure, Satyam noted that one of the challenges can revolve around data discoverability. “That brings in a lot of repetition of data and at times, people don’t know if they’re looking at the right data or not. I would say data discoverability is a big problem.” But through data cataloging practices, they are working on solving this problem. “That can basically help you know, ‘What does this table mean? Who created it and when was it last updated?’” This process involves building a system that create alerts that inspire more confidence in the data. “You get alerted internally rather than someone else reaching out to you,” he said. “So it brings in more confidence that the data that you’re looking at is right. If you start losing confidence in the data, you can never trust your metrics. And then the business decisions that you’re taking, it gets more and more difficult to be sure what you’re actually impacting or making positive changes.”

Of course, Grofers also faced challenges in light of the COVID pandemic. “I think one of the challenges we faced was more in terms of the communication internally in the team,” he said. “If you’re locked down for months and cooped up in a room, you want to ensure that that energy gets released We figured out games that we can play in our typical meetings and at the end of the day.”

The company also had to determine how to scale up the need for delivery services while also factoring in increased cost for delivery. “Ground operations were impacted massively.” He recounted questions they had to ask themselves. “How do we ensure the safety and security of our customer? How do we reduce touchpoint and how are able to serve all the customers?”

He said their response was to better batch their products and clump deliveries so that there are less delivery routes and the ability to deliver more orders. “We added capabilities in our application around how you can edit your order multiple times, which was not even supported before,” he said. “That was one of the biggest changes for us and there were a lot of new needs around reporting. Once you change your business model, you also want to track metrics at a much much faster pace. There were a lot of new real time reporting needs that came with that situation.”

The Data Stack Show is a weekly podcast powered by RudderStack. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.