Why Big Query Was a Big Deal, Observability AI, and How AI is Like a Guy at the Bar, Featuring David Wynn of Edge Delta

Eric Dodds 00:05
Welcome to The Data Stack Show.

John Wessel 00:07
The Data Stack Show is a podcast where we talk about the technical business and human challenges involved in data work.

Eric Dodds 00:13
Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. Welcome back to the show. We’re here with Dave Wynn. Dave, welcome to the Data Stack Show. We’re excited to chat with you.

David Wynn 00:36
Absolutely. Glad to be here.

Eric Dodds 00:38
All right. Well, we know you worked for Edge Delta in observability. But give us the brief overview of where you came from before that.

David Wynn 00:46
Oh man. Well, if we want to go back far enough, there was a cold and snowy night in February of 1984. And a cry rang out at four in the morning, which is unusual for that time of year. But if we fast forward a little bit from there. So I’ve been a lifelong geek. And I’ve bounced around a number of different places from doing econometrics at UPS headquarters in Atlanta to hopping around a few startups in Silicon Valley with some ETL software and some observability software. And then I was at Google Cloud for a number of years, doing all things there both on the computer and on the data side. By what until I finally am here at a new startup, where we’re doing observability. So little by little boy,

Eric Dodds 01:32
That is very cool. Nice.

John Wessel 01:35
So one of the topics we talked about before the show was your time at Google. And we talked a little bit about BigQuery. So I’m interested in digging in a little bit more there. Because you were at Google, I think, during some of the crucial years where that was, you know that that came to be a really, Pinnacle product. So what are some topics you want to dive in on?

David Wynn 01:56
Oh, man, you guys, tell me what you want to talk about. But BigQuery is definitely in my opinion, one of the best, if not the best data warehouse products that exists in the clouds, because it’s the closest you can get to a sequel API with really not having to worry about any of the back end knocks on Athena, Athena is great for what it does, don’t get me wrong. But if I’m gonna open another tab and start managing all of the S3 stuff and have all my Parquet files in just the right format, I’m not having the best day, if that’s what’s happening. So BQ just makes it super easy to dump data in there at the appropriate time in the right increment in the right spot. And you can just go about and start querying it, whether you’re talking to Ed the gig scale, or the petabyte scale, it just doesn’t matter. So I found that really slick.

Eric Dodds 02:48
Awesome. Well, tons to talk about. Let’s dive in. Yeah,

John Wessel 02:52
let’s do it. Dave,

Eric Dodds 02:54
There are so many topics that we want to dig into. I’m actually interested in you so you studied economics, and then right out of school, you did econometrics work at UPS. What did you do there? And what does an economist hired by UPS figure out for them? Ah,

David Wynn 03:13
So it’s a great question. I joined actually, I was hired to be part of one group and was transferred to a different group before my first day. Yeah, nice. But the way this works is that I joined the Forecasting team. And this was 2008, where we were coming into a giant recession. And the thing that this team was responsible for was predicting as far out into the future as possible. How do we know how much we need to ship where and when? Great, so we need to maintain all these time series that give us forecasts around all these things. In the past, the models didn’t actually have to be that sophisticated, because UPS more or less tracks GDP period within a percent or two. So not that hard to predict. You could just sort of hit the button. Until 2008. Wait a minute, suddenly, some things aren’t quite working, right? So number

Eric Dodds 04:09
look slightly different.

David Wynn 04:13
So there was a guy who thought that maybe we could bring in some more econometrics into the forecast to figure that out, they hired a PhD in order to implement some of the key ideas. And they hired a grunt to do all of the work in Excel to make scale happen. And I’m not going to say which role I served. I don’t have a PhD. That’s kind of

Eric Dodds 04:36
wild. Okay. So I mean, dig into that a little bit more. What did you mean, what did you find? How did you address the problem of, you know, sort of your core metric that had been used to forecast that business changing?

David Wynn 04:49
Oh, my gosh, well, so we’re almost thinking about it more on the operational side of what I was responsible for. So we had a number of time series that we published as a team to the wider organization. And this is before we were ever using a database like someone passed around a CD of SQL, you know, SQL Server 2005 as like, oh, maybe we should try this? Yes, we were before that. And in that sense, we were sharing Excel files about what were the different series that we maintained. The PhD is still there, and is doing tremendous, tremendous work. As I understand it, her name is Juana Mazarin. And she’s great. She would literally read academic papers and books and things that were published and translate the different processes that were available into stuff for time for a time series, that would matter. And so that would work once, and then I had to figure out how to do it more for all of them. And this involves a lot of VBA, which I was very grateful for IntelliSense. No matter what level of intelligence you would consider 2008 IntelliSense, to have just helped me keep moving and get going. But a lot of these operations were very annoying, right, so you can’t just fill down across the whole row because you’ve got month resets, right, and you’ve got different moving averages, and you’ve got different, all different like little operations that if I was doing it today, I probably would have functionalized everything and go about it that route. But I wasn’t quite that smart. Get younger Dave that had fewer I’m gonna call these blonde hairs on my chin. Younger Dave that had fewer blonde hairs didn’t quite know that. So we were trying to do everything half and GUI land and half in VBA. And that led to a lot of files with a lot of very particular changes. And, you know, it was a job. Yeah.

Eric Dodds 06:43
Okay, just out of curiosity, because I want to jump actually, to the wit there’s so much to talk about the Google I O. John has a ton of questions. But time series data at that scale is very interesting. Right. And so this the toolset that you’re talking about sounds, you know, particularly painful, what would what stack would you use today, right? I mean, they’re like time series databases, like influx and other tools like that, that are really good. You know, maybe that’s not the right tool for what you were doing. But what stack would you put together today to do that? And

John Wessel 07:15
Before you answer that, what percentage of the time did you spend, like waiting for Excel to become responsive again, after you made it? And you didn’t hit save?

David Wynn 07:26
So I’ll start with the latter question first. None at all. Because I was managing 250 Excel books. Why would I put these all in one workbook when I could do this copy and paste 250 times? So performance problems? So yes.

Eric Dodds 07:42
That is the original brute force.

John Wessel 07:45
Wow. Okay. Yeah. I wasn’t expecting that. To answer your former

David Wynn 07:49
question. I became a and this was actually true with a different issue that we had when I was there, where we had a Microsoft Access application. That wasn’t I didn’t potent, and it needed to be run on a scheduled basis. And or there was a problem with it that we didn’t notice, because the non item potency had messed it up. And it wasn’t until a different consumer that was rather important. noticed that our forecasts were the same week to week for a few weeks now. And we were like, oh, so simple. redoing that now. I mean, I’ve become much more on the item potent train much more on the functional train where I would be trying to make as much of those as we can. storage is cheap, which wasn’t even true enough, then. But it’s much more true. Now you’re and so why would we bother trying to mutate state in place when we could just have a much clearer lineage about how these things get transformed from place to place. So it’s all of the really cool and interesting stuff we talk about, like organizing a project correctly, and making sure that your tables are well named, and all that other good stuff, where you don’t just hope that someone else can read VBA and has your brain.

John Wessel 09:01
I often wonder if weather forecasts ever work that way? We’re like somebody somewhere. It’s like, oh, like, you know, the weather forecast is the same as it was last week, and somebody didn’t run something. Yeah, I absolutely

David Wynn 09:13
would.

John Wessel 09:14
But at a local level, that’s gotta be possible. Sure.

David Wynn 09:16
I often wonder if the precision is cut off because there’s like moss on the thermometer. And it’s just like, sorry, we can only get to one decimal point. Hopefully that’s good enough. Yeah. Like,

Eric Dodds 09:27
rotation. Yeah, good. Good. It’s probably why they did it at airports. Because you know, they have to Yeah, as instruments clean. Yeah, good point. Yeah. I mean, you got to think about the meteorologists you know who it’s raining outside. And the forecast comes in, and it’s different from the actual Yeah, absolutely.

David Wynn 09:45
Have you guys read that article? Gosh, I was on Hacker News the other week about like crazy real life bugs and the bug was the WiFi works when it’s raining and not when it’s not. Not fantastic. So it’s worth looking at To hear the whole story, but basically there was a guy who came back in from college. He was always tech support. First thing, but his dad was also very capable. And he was like, Yeah, I don’t know why, but the WiFi only works when it rains. I haven’t looked into it yet. And

Eric Dodds 10:12
right, which, depending on where you live is like what you’re not knocking on the internet that much, for

David Wynn 10:16
sure. Well, and also that’s backwards than what you would expect and a whole host of things. So the long story short, is that they installed their internet from a microwave beam that they were getting from an opposing house. And what had happened 20 years ago, was that someone had planted a tree. And so when it rained, it weighed the leaves down enough to be clear. And when it stopped, it was just enough to block most of

John Wessel 10:45
the signal. That’s awesome. I love that it ruins its nose. It’s just like, perfect Internet. Oh, yeah.

David Wynn 10:53
Exactly. Yeah. Recall, the article wasn’t there for wintertime. So I can’t speak okay. But what do you imagine?

Eric Dodds 11:00
Man that is so great. Okay, well, just a reminder to the listeners. We’re here with David Wynn, from Edge Delta, and we’re chatting about Wi-Fi signals VBA and forecasting, and breaking Excel workbooks. But John, you had a bunch of Google questions. I have some too. But David, tell us just give us a brief overview of your time at Google, because you’ve worked on all sorts of stuff. But how long were you there? And what were the sort of the biggies? Yeah, I

David Wynn 11:28
I was there for about seven years. We started when there were enough customer engineers to fit in one training room across the entire world, which we did once. And we didn’t have enough training material to last the entirety of two days. So half of a day was scheduled for five minute lightning talks from every person in the room, which was fascinating, because it could be on any topic that you want. And so that’s, that was the vibe, it was young, and it was fun. We also didn’t have all of the enterprise things that people routinely demand when I joined, like being able to directly peer to Google, that’s that that was a pretty big one that didn’t exist at that time. This was also before Kubernetes. And before GK and before various other things. So I was out there talking to people directly about architecture, how they could migrate to the cloud, how they could re-architect so that things might be more effective. And across the entire suite of products. I like to joke that the job was not that hard. All you had to do was not the some 300 products that we had, no, there’s some 400 products that AWS had no some 500 Open Source offerings, and how they all fit together in every conceivable scenario. It’s not that big a deal. But that, you know, that led to an interest in basically all different flavors of stuff. Because at some points I was territorial, where I would cover the entirety of the West Coast, because that’s how territories go when you’re early to smaller and smaller territories. And then I started focusing on an industry because I have tried to quit video games several times. And I’m sure I’ll succeed one of these days. But I figured maybe I should make that more of a job thing. Because we had several notable gaming customers at GCP Niantic makers of Pokemon Go were probably the biggest ones earlier on. But there have been others, like Unity, and Apex legends and various other things. I’ve also used different degrees of Google Cloud, which I may or may not have had a hand in. And so yeah, like, that’s where I was for most of that time doing the customer facing architecture side, and then also doing a lot of partner stuff as well.

John Wessel 13:34
Nice. So Google Cloud, I mean, that that is probably the most, like, fundamental, like, if I picked like seven years, you know, to be a Google Cloud, it seems like those were some of the most transformational years. Totally like what, so we want to talk about more about

David Wynn 13:50
today’s riti.

John Wessel 13:53
We’ll see. We’ll see. We’ll see. Yes.

David Wynn 13:56
Oh, we could probably have a chat about that, for sure. But yes, definitely big times.

John Wessel 14:00
Yeah, for sure. So, we want to talk about BigQuery. But were there any other like, you know, in your years there were like, a product comes out, you guys are introducing a new product? Oh, and you’re like, like, wow, this is gonna be incredible. Or maybe we can just talk about BigQuery? Or if there are some other products that you felt that way about inside the ecosystem?

David Wynn 14:19
I don’t think so. I think BQ is really the one that I’m the most enamored with, just because it delivers so well on the core promise that solves so many key problems. Dataflow, which is Google’s implementation of Apache Beam, which is the ETL framework, has promise, but it’s too complicated. And I tried to understand it for a couple of halves. There. I had it on my OKRs to try and figure this thing out. And usually when I had trouble, I would go ask the team and they’d be like, go read the source code. And I’m like, the last job that I saw was at JL Mann High School AB and I cannot read that was in July, Javis, five or something. And where they didn’t have decorators. And I don’t know what any of this syntax means anymore. So no, thank you. But I also knew enough about it to advise people on what the architecture patterns they needed and common pitfalls were. But even the Python SDK they built, I think, was just a little bit beyond what’s pretty reasonable for people to get. So I think that BQ hits the right thing. I think VMs are very commoditized. I think GK is great, and is definitely probably the best Kubernetes platform. But I mean, that’s borderline commoditized as well, because everybody’s doing Kubernetes. Well, but I think

John Wessel 15:42
you bring up a good point that I think a lot of companies struggle with is like they can have a brilliant solution to a problem that is not accessible enough to enough people to make a difference. For sure. Right. And it seems like you’re saying that the Big Query, a kind of hit that is a brilliant solution to a problem and very accessible to a large number of people. Yeah,

David Wynn 16:04
The only challenge that you would really have is migrating the data and getting it in there. That was really the only one because if you have a petabyte capable system, your next problem is getting a petabyte of data in there, sir, in order to make use Yeah,

Eric Dodds 16:19
so sure. What I’m interested in, we actually, interestingly enough, we have not talked about BigQuery. Much of the show, which I love, is that in recent shows, we’ve covered a bunch of topics we got into the other day. We got to the SAP HANA details. Yeah, that, you know, that was great. Yeah, that’s great. QCon some hardware? Yeah, totally. That was awesome. Good stuff. Oh, yeah, totally. It was great. But in terms of, you know, when you, I think there’s sort of this perception of like, you know, you have Snowflake, you have Databricks, and then big queries, you know, the third one on the list. But all the headlines go towards Snowflake and Databricks. And, you know, I mean, part of that could be because Snowflake and Databricks are sort of that’s the main thing they do. Whereas the alphabet is gigantic. And Google Cloud is, you know, a sprawling list of products, only to be dethroned by the AWS, you know, portfolio of products. But in terms of sort of the Snowflake Databricks, Big Query, give your perspective on that. I’d be interested in

John Wessel 17:30
one other thing here, like, think about what we’re talking about. We shouldn’t be talking about Microsoft Azure SQL. Yeah, AWS, Athena, whatever, and BigQuery that that is, should be the conversation,

Eric Dodds 17:43
man, I gotta get out of the bus, I need to get out of the social datasphere and stop reading the headlines about, you know, Battle Royale.

John Wessel 17:52
And I think it’s a really significant conversation that there’s like, clearly, like, who should be and then an Oracle, like we just skipped over Oracle, like, those are the four people that would be in this conversation. And only one of them is, which is a big deal. Like, like Snowflake and Databricks. Their claims are great, too. But it’s a big deal, the big queries in that conversation. And I’d be interested if you have any thoughts on like, why? How did that team win? How did that team beat out, you know, all these other products that should be just as viable theoretically. So

David Wynn 18:26
First, I’m not going to dunk on Databricks, or Snowflake, those are both great products. And last two, I’m not going to say which one, but I’ve lost to one of them more than I would care to admit when I was there and BigQuery. Sir, the challenge that I think comes with it is, especially when you’re talking about hyper scalar, there’s a question of how much do I have to commit in terms of getting a return on what this is, right? Because if you’re running most of your applications in AWS or Azure, you probably will just use whatever they have and kind of suck it up and deal with it. Right? And GCP, for most people, for better or worse, I would argue worse, but that’s not what we’re here to talk about, will not have GCP as their default. And so we’ll miss out on what goodness that this could provide. So I think that really is what holds people back. Whereas you look at Snowflake, you look at Databricks and Pimcore value prop is multi cloud do the whole thing. Doesn’t matter where it’s like, yeah, that’s not a thing that BigQuery could do for a long time. Talk about they just recently got it towards that in the last couple of years. I was there with federated queries and stuff. But even then, now you have even less of a tie to this platform. I don’t want to know if I have to go learn and figure it all out. And I have to give some empathy to that because here’s a little bit of humble pie, then I’m gonna go ahead and talk about eating. I was in Google Cloud for like a bunch of years. I think I’m a pretty sharp guy. Mostly. I thought I understood what the cloud was. I hadn’t really dabbled with AWS until about a month and a half ago, too terribly much. And I was very humbled at how different these two things were, in so many respects. And I could see perhaps a lot of the architectural decisions they’ve made like, Oh, I see how they got there. I don’t understand why I have to open so many browser tabs. And I don’t understand why all of the instructions are out of order. I don’t know if you guys, I don’t know if you guys already know AWS but trying to learn AWS in 2024 is insane. Because there is no ground zero tutorial out there that is up to date. They’re all half old with API’s and stuff. We’re a monster,

Eric Dodds 20:40
like a monster, an absolute monster. So all I have to say

David Wynn 20:45
is that a GCP like that doesn’t really exist. Someone is in charge of making sure it all works together. And boy, is that a change? But I recognize that other people don’t want to take on what they expect to be that madness times to you, if not more? So, this Oh,

John Wessel 21:01
I hear it. Yeah, this was almost 10 years ago, but I had to buy Pluralsight classes to get through some of it. So I did a bunch of digital, like modernization efforts, you know, almost 10 years ago now. And even then the documentation was either fairly inaccessible, or just like you said, I like I don’t know. So I just got a Pluralsight subscription. And like, you know, walk through some of the classes.

David Wynn 21:26
Yeah, chat GPT-3 definitely failed me in terms of trying to get me up to speed on AWS. It was some number of versions of Alexa, you should have asked Alexa. Right. Don’t Own it, Alexa.

John Wessel 21:41
Yeah, one of the anthropic models maybe would be better. Yeah. If they train those on the Amazon manual.

David Wynn 21:47
I should really sign up for Claude, I understand that one to be a bit more linguistically advanced, though, not technically advanced. Yeah.

Eric Dodds 21:55
Yeah, I’ve heard it is, I mean, one thing that I just want to return to what you said about the system working together, and then also considering what you said about I don’t remember the specific name, but Google’s implementation of Bing. We

David Wynn 22:14
don’t have a lot of people naming we’re not, they’re never good at it. Yeah,

Eric Dodds 22:17
That’s like the hardest, that’s a very difficult thing to get really good at, especially with that level of product catalog. But I mean, there is a lot to be said for, okay, this is in a combined platform. And if I were just gonna go to the market, and I could buy anything I wanted, and, you know, build this perfect thing. That’s great. But the reality for a lot of people is like, woof, procurement is a beast, and like, these things work together. And so even if it’s ideal, like, it’s still just a pipeline, right? It’s gonna run. And so did you see that dynamic a lot, where it’s like, the advantage of a connected ecosystem can outweigh the challenge maybe or like the rough edges of an individual product?

David Wynn 23:04
Yeah, I think a lot of GCP customers can testify to that for sure. And I think that has to do with the different development approaches that the different hyperscalers have. So AWS famously built two pizza teams, right? You got to features and stuff that need to be shipped on like a relatively small number of teams. What that means is that your interface boundaries grow a lot. And what we see in 2024, if again, you’re coming to this new is there are so many checkboxes, they are so out of order, they so have different expectations around all these because this team built that check box, this team built that check box, this one did this and you can feel it. Whereas in GCP, like it, just someone is in charge of the console and the flow of it. And it just makes so much more top down type of coherent sense that, yeah, whether or not the dashboarding solution inside of GCP is like the greatest thing since sliced bread. It definitely works. And it definitely plugs straight into BigQuery and takes advantage of a ton of optimizations that they have under the hood that is totally like keeps everything fresh in a way that is harder to do when you’re not shout out to all of the dashboarding solutions that do great stuff, not trying to knock any of them. But you know, there’s just more cohesion that you can take from that perspective. Yeah,

Eric Dodds 24:19
for it. Okay, one more question for me on Google Cloud. Do you think that Google’s and this is a I don’t want to ask this. So Google’s different business units, you know, at least I’ve never worked for Google. But just from my experience, you know, sort of building some technology on Google in a previous life like, even products like individual products can have parts of them that are pretty disconnected, not to the level of the Amazon sort of to pizza, like checkboxes are out of order. But one interesting thing, at least as a user of BigQuery, is that I use the Google Cloud to scaffold out a bunch of personal projects and it is very approachable. You know, just even using, like, their different API’s and other things is like, very approachable. And so you can build out a project really quickly, right? And just about everything works with BigQuery. And it is super nice. Do you think that comes from Google’s competency in consumer facing products? Right? I mean, that’s really where they came from, like deeply consumer facing Gmail. Yeah, exactly, you know, search Gmail, where there’s a significant emphasis on, you know, sort of emphasizing, like, simplicity and flow, or is that disconnected? Because you could tell me either way, and I wouldn’t necessarily be surprised. But I’m curious, obligatory

David Wynn 25:37
disclaimer here that all opinions announced here in this podcast are solely the property of David Wynne and not have any particular analysis of any entity.

Eric Dodds 25:47
And this does not constitute as investment advice does

David Wynn 25:49
not constitute investment, legal or technical advice, please consider everything I say stupid. I don’t think so. I think cuz, here’s a really interesting thing about Google Cloud. What was Google clouds first product?

John Wessel 26:03
Do you guys remember? Storage, but I actually don’t know. So

David Wynn 26:08
that’s S3, you’re thinking of S3 was the first product for AWS, which was released in 2004. Yeah, but

John Wessel 26:14
not been offered by Google buckler. Now for Google there? No, we haven’t. Alright,

David Wynn 26:18
The first product was App Engine, which is the entire development platform built in one. Now the reason for that is, that is how Google Developers work internally. And so the idea was down to the part where they actually run it on infrastructure inside of Borg inside of the thing that runs Google. This is the development model that we use here. Everyone should use the development model here. That didn’t catch on, for a lot of reasons. Partly, at least, because people would have to rewrite a love for applications they didn’t want to rewrite. So Oh, okay. Maybe if we want to go get this market faster, and, and more directly, I think AWS had a much better approach to that where it’s like, let us give you exactly what you are familiar with IT teams. And we will slice it up for you and charge you by the slice and have a nice little thing right here. Whereas Google tried to bring a bit more of the Google way of doing things. Yeah. So when we talk about projects, which I do think is a meaningful boundary that they drew in GCP, early on. I think that was more of a happy accident from the way that App Engine was structured. Because I do think it’s a much more coherent way to organize stuff. Yeah, then. I mean, does AWS have a project boundary? Now I feel like you can do some things with Ackles, and stuff like that. But mostly, it’s still just like, I hope you logged in with the right account, because here it goes. I don’t know,

John Wessel 27:53
Azure has resources that are kind of like projects. Google has the most clear boundary. I mean, you can tag things in AWS and you can have different accounts, and you can have a unified account with sub accounts. But beyond that, I don’t know. Yeah, it

Eric Dodds 28:09
is really nice. And Google though, like the other day, I had this, you know, 250 page PDF, exported from a note taking app on an iPad. And I was like, I don’t know why I wanted to experiment with OCR stuff. But Google has some really very cool products around that. Yeah. Yeah. And I mean, spinning up a project, like, because you ‘re required, because those are pretty heavy. And so like they require you to do well. I mean, we could discuss why they require you to add that one makes sense. Because you’re, you know, if you really hammer the system, but it was like, This is unbelievable. You know, like I ran a test in a couple minutes. You know, it was like, super cool.

David Wynn 28:51
Yeah, it’s good stuff. Highly recommend it, especially if you’d like light blue. It’s got a pretty tight theme there. I will, I will go on record. I assume Sundar is listening to this. Sundar, I’m gonna go ahead and tell you something that I didn’t get a chance to tell you in person, which is that the old logo for Google Cloud was better with the rivets, it should come back. I recognize it didn’t have all four colors. And that maybe is branding standards and like is a thing. So nice. Anyway, that’s my high horse. I’ll just step up and step off real quick.

Eric Dodds 29:22
And, Sundar, The Data Stack Show has a message for you. We would love for you to come on the show and talk about data at Google.

John Wessel 29:30
The Google logo in

Eric Dodds 29:32
the Google logo. really

David Wynn 29:34
use the agenda. It’s great. Yeah, yes, yes.

Eric Dodds 29:39
Okay, that was great. That was great. Just as a reminder to the listeners, who were driving and trying to look at maps and wonder at the same time. We are here with David Wynn from edge delta. And we’re talking about all things Google Cloud. What was next on the list though?

John Wessel 30:01
Well, we got to talk about observability. Yes, I want to put this in here because I’m curious about your take on BigQuery. So, open source table format, specifically Iceberg is making a lot of splashes, big splashes, which in the concept is great, right? Like, you can have this open storage concept that can be in S3 or GCP, like whatever storage you want, and then you’re less locked in all these products. So then that pushes all the, like, battles up to the compute layer. Right? So you got a Snowflake engine, you got a Databricks engine, it’s

Eric Dodds 30:33
good for the consumer,

John Wessel 30:34
it’s good for the consumer, allegedly. So where does GCP stand with that? You think? I, this could be a thing today. Can you use GCP today to access data and Iceberg? GAML?

David Wynn 30:49
That’s a great question. But I’m afraid Iceberg came along a little bit after I left GCP. So I am, I’m not sure that I’m really equipped to answer it.

Eric Dodds 31:01
I’m asking Google, okay, perfect.

John Wessel 31:05
Gemini, it seems like directionally right. That would be that okay, okay, Amazon, Oracle, Microsoft, like, this is your chance, like have a query engine that basically is just compute and access data and Iceberg like ready to go? Do you think any of them will do it? I

David Wynn 31:24
mean, functionally, that’s what Athena and BigQuery already do. Right? They have separate compute stacks on top of some proprietary or non proprietary formats that they can spin up well. So

John Wessel 31:37
embrace, embracing a new open source standard is really the question. Obviously, they’re capable technologically but will they play in the Iceberg? Yeah,

David Wynn 31:44
I mean, they took on Parquet, I don’t see any reason they wouldn’t take on Iceberg, right? Like, it’s invariable that much more interesting questions to me are, as we evolve our understanding and our practice of what we need to do as data analysts and data engineers, how does that change what we need to do, right, because I just came back from monitor Rama in Portland last week, shout out to the organizers, they always great conference much appreciate it. They’re one of the dominant themes, because in observability, we’ve got this concept of observability data. But it’s not in tables. It’s not an open format. So it’s not anything like that, like there is this concept of open telemetry, which does standardize the line protocol a little bit and has an agent associated with it. But mostly, there’s a whole bunch of all kinds of stuff floating around here from log lines to time series data to trace information, which is sort of logs but with parent IDs, and back and forth and stuff. The old approach was what I call the Patrick Starr model love, why don’t we take all of the data over here and put it over here, so that at least then I don’t have to go to 80,000 machines, if I want to see if something went wrong, that that’s like a level one improvement for sure. And was viable at gigs of data per day. But now we’ve got terabytes of data per day. Now we’ve got hundreds of terabytes of data per day. Now we’ve got some of the bigger organizations generating a petabyte of observability data per day. And

Eric Dodds 33:16
it’s like, we’ve

David Wynn 33:17
got to take this one step back and think about okay, what are we doing shear? Yeah, cuz, cuz you just can’t move that all across the wire fast enough to matter. And so, Edge delta is obviously helping to push this forward. But there are other people that are in the same vein of like, we need to push that distribution down as far to where the data can be as possible. So we can do aggregations, and filtering and routing and stuff, where all of that data is created. That’s one method to think about it. But you know, we might even have to think about what kind of data it is we’re making. And how do we use it? Because, man, we’ve got to, we’ve got to tie the dots on these things. I’ll, can I talk about my worst meeting? You don’t have to say yes, but maybe I’ll go ahead and say it anyway. When I was at UPS, one of the things that inclined me to get out of data analysis, was I had been given a charge and put together a dashboard and an analytical report on I honestly don’t even remember what I remember working on it for, I think it’s a week or something, you know, a good chunk of time, particularly as a young guy who didn’t know what I was doing. And then I walk into the meeting, and I start talking, and I can see the guy’s face change right away. And within 30 seconds, instead, he stops me. So, David, this looks great. But I wanted to let you know, this isn’t what we’re looking for. I wanted to see this, and this and this. And I was like, huh, that was a week’s worth of effort for a whole host of things that I thought were interesting bubble up style, that I’m now being told Do a little bit more top down style in a different direction. And it’s like, Hmm, something about this went wrong. Did I

John Wessel 35:07
waste didn’t? Did he just ask for hard coded values? He’s like, can you just code these values? I want it to be often to the right, man,

David Wynn 35:14
I really wish I remembered the specifics, but I mostly remember his face.

Eric Dodds 35:21
Like, you know, if you go in with data to present something, and in the first 30 seconds, that big, blinking red light on your internal dashboard is like, we’ve lost the audience here. Yes,

David Wynn 35:34
there’s, there’s such a thing that I’m always interested in ways that we can tie this type of stuff closer together. And I feel like as analysts and engineers, we can get a little bit caught up in the properties of what this is, without thinking enough about how it ties back to the greater objectives of what we have, it’s actually going on here, right? And I think the next wave, you’ll see an observability. But honestly, a little bit from the analytical side as we start taking more and more control of our data via open formats, or what have you, this needs to line up to the thing that we all need to do here. So how do we tie those things together so that we don’t burn any cycles that we can miss on?

Eric Dodds 36:16
For those keeping score at home, by the way, talking about open table formats? Straight from Gemini? Yes, you can query Apache Iceberg data in Google Cloud using BigQuery. BigQuery. Supports Iceberg format through big lake meta store, big lake meta store. That’s been their naming game. Big Lake. Yeah, that’s, yeah,

David Wynn 36:36
I kinda liked that. They made sure you knew it was big. Yeah.

Eric Dodds 36:39
And Lake, they got the data lake, it gets a little, what’s the largest

David Wynn 36:44
customers that wanted to change the name of data lake. We don’t want a data lake. We want a data ocean. We want that data galaxy. And you’re just like, yeah, man, absolutely. Keep going. So,

John Wessel 36:59
so our conversation about observability reminds me of something that we’ve talked about with RudderStack and autotrac. You remember, oh, auto track, right? So there’s this problem, I’ll let Eric describe it. But it’s what you’re saying. We’re like, hey, let’s just go in and collect everything, right. And you have this decoupled, technical team, like, I don’t know what’s useful, I’ll collect and store everything. And then like, downstream, you know, and business team that’s like, I don’t care about any of this stuff. And it’s not like I might care about some of it if I literally will never care about this piece of it. So every moment wasted engineering, collecting, tracking, storing, retrieving is complete waste. Well,

Eric Dodds 37:40
The interesting thing about that, is that the context is, you know, RudderStack collects user behavioral data, you know, so telemetry from things like, your website, or app, etc. And early on in the life of the company, they had experimented with an auto track feature, which is basically you install the script on your site, and it just tracks every change in the DOM on your website, and just sends that as a payload. It’s so noisy, you know, something really interesting. I don’t know if you listened to the show, but we had someone from the analytics company Heap on The Data Stack Show. And heaps, one of their big differentiators was auto track. And they stuck with it, and actually ended up figuring out how to make it work. But listening to this, this astounded me. It took their engineers, because they like our model to be very different. We send everything you know, to the warehouse or whatever, we don’t actually store any of the data, we have, you know, sort of standardized schemas or whatever. But Heap is an analytics tool. So not only do you collect, but they actually provide things like analytics, visualization, you know, layers or whatever. But I think the guy, I think the guy said it took their engineering team, like five or six years to build a system that had reasonable SaaS cogs on auto track. Wow. And then they did an immense amount of work to reduce the noise. And now it allowed them to do some very interesting things. Because if you can actually solve those two problems, you know, then you do have an interesting data set to work with. Right. But it was astounding, right? And it was actually, I mean, I seem to remember, I can’t remember the exact details of the conversation. But the founders had to be extremely opinionated, both with operators and investors to say we’re going to have really bad coughs until we figure this problem out, you know. And so not only is there noise, but like the infrastructure impacts and sort of like under utilization of what is required under the hood to even process that is significant. For Shame with observability Yeah, absolutely.

David Wynn 40:00
Well, I mean, you’re basically talking about a different form of observability. Right? When you’re zeroing in on user and behavior, that’s not what we traditionally think of and observability. Because we’re looking at the application and its actions, AI, but you’re, presumably, usually users initiate those actions. It’s a

Eric Dodds 40:16
yeah, a lot of times it measures. Okay, so let’s talk about we can’t not talk about AI. When we’re talking about petabytes of disparate data. I

David Wynn 40:25
was just gonna say, can you imagine how disappointed everyone’s gonna be that we’ve gotten this far without putting a and it gets

Eric Dodds 40:31
like a game every week? Honestly, we’re like, how long can we push this conversation? Without talking about AI? We do pretty good. Although we did, like we did, you know, we did disregard Microsoft and Oracle, you know, in favor of the, you know, the darlings of the valley. So we at least checked that box. And we talked about the Iceberg. You know, so, okay, but legitimate question. I mean, when you think about petabytes of data, petabytes of different types of data, in a context of observability, like, of course, you go to I mean, of course, the default is it can AI solve that problem, right. But it’s a machine learning application, right, where you’re looking for. You’re looking for anomalies in like a giant, you know, stream of data. But what do you think about that at the edge of Delta? Yeah,

David Wynn 41:27
so we are using a very hybrid approach of traditional, traditional approaches with sort of your standard type of alerts and search and various other things that go on that people would expect, as well as some machine learning driven sort of dynamic behavior. But it just makes alerting a little bit easier, because we’re re-baselining everything for you. So in that sense, we’re letting the model be. It is very hard for a constantly changing application to have fixed alerts that make sense over a long enough period of time, because you get drift. That there just is no, there’s not a great way to do it. And currently, we’re solving it by the fact that SMEs hopefully remember what alerts they have. And if they have gotten quiet for too long, they’ll go in and check them or nothing’s gotten to them and they will fix them and not just send them straight to spam. Yeah, we’ve, we’ve recently experimented as well with putting some LLM AI on top of our anomaly detection. So that is a very high signal to noise type stuff. I call it almost like the to am checklist of if you get paged at three in the morning, because something has gone wrong. And you are like, Oh my God, how did I ever keep those monitors this bright? I just want to do a thing and get back to sleep. Like the AI, we’ve added a little LMS in there to just give you hey, maybe you want to look at this, maybe want to look at this. Yeah, it doesn’t auto do anything on purpose. Because I mean, that’s, you say, Sure. There are people that think that’s the way forward,

John Wessel 43:00
but not, not at 2am.

David Wynn 43:06
If it gets the alert to go away, they’re absolutely SRS, they would push the button and do that there’s no roll back button. But there’s a fix button. They would absolutely know. That’s the reason it’s not a salute. Yeah. So it just makes suggestions for you along the lines of hey, you might want to look at this, you might want to look at this based on the anomaly in the information that we could all correlate across the different information. So that’s the direction that we are taking with it. Yep. Personally, I am on record as thinking that AI is not going to quote unquote, fix observability, which I likened to, hey, we’ve got a petabyte of data. Let’s dump it in there. And it’s like, Yeah, are you going to train that model? Are you ready to spend all that information? And that’s just the cog side. Even more importantly, if developers are good at one thing, first of all, any developer listening to the podcast right now is amazing and never makes these errors, but other developers write the other ones. If you’ve met any other developers, they’re really good at creating new ways for software to screw up. And the idea that you will have a dataset that has all of the errors that you could want to track in the future is comical. Yeah. And I’ve told the story several times of my favorite database error when I was working at the ETL company was a database error that I got that said, you haven’t paid us. And I was like, what? And it turns out, we were using a Salesforce syncing tool that to go from local database writing to you basically circumvents force.com Because you could write to your local database and they would handle the syncing into Salesforce. But we forgot to pay them. So I was like, I’ve never seen that error again. Is it useful to have any n LM is trained on that No. And that analog holds true infinite ways that we can combine bits together. So I’m very skeptical of the idea that AI is coming to fix observability, in particular. And similarly, I’m a little bit skeptical of it sort of in broad terms, that’s a bit more of an open question.

John Wessel 45:13
So the idea just in this one example would be like, Okay, we’ve got an AI in place, it is going to be able to have never having seen this before, read this error message. It says something very generic, like you haven’t paid us to know that there’s some vendor out there that you’re using to sync from A to B, and to prompt somebody like, hey, you need to go pay them like that. Yeah, that makes sense that that wouldn’t work

Eric Dodds 45:32
in St. Context requirement. Yeah,

John Wessel 45:35
exactly. Yeah.

David Wynn 45:36
Yeah. Well, I think that I really think one of the big challenges we have as it’s that if we’re not directly at the frontier of research, we’re getting a lot of second degree assessments of what AI can do and can’t do. And so I think what we really need, even for people in our position, and I can’t speak to how familiar you guys are with it as well, I’m not making any disparagement. It’s just we need better mental models of what it’s like. And so the one that I give to everybody is this an LLM, our current understanding of MLMs, as of time of recording, is, it’s a bit like a guy in a bar who has overheard 10,000 hours of conversations about motorcycles. So he’s never seen one, he’s never touched one. He’s never written one. But if you ask him any question about a motorcycle, he probably knows the answer. But occasionally, he might compliment how your torque smells. And so you’ve got to you he doesn’t know the difference. It’s not his fault. And the way they work is not at least based on our current understanding, again, at time of recording, if they don’t reason. But they’re associative, the reason that chain of thought works is that it snaps the words into a reason like a looking object. And so when a lot of AI products, and startups pitch themselves on the idea that AI will be able to think, or reason, or do the decision making part. I’m pretty skeptical. But if it can do some of the things that computers are very good at, like computers never get tired. So I think they’re very good at brainstorming, pulling together different associative ideas. You know, a lot of baseline stuff that can help clear the blank page problem. I think I’ll be great for like, 100 different paper cuts in normal everyday life, just like data was, you know, 20 years ago, or something like date is gonna change everything. It hasn’t knocked out the economy. It’s just made all of the little things that we do a little bit different. I think we’ll see that too.

John Wessel 47:39
And of the 10,000 hours of the guy listening at the bar, he was drunk for several 100 of them, but we don’t know which ones are right. Yeah. And he got, I guess, not so sure about, he’s got some hazy gaps, right. Or he

David Wynn 47:53
wasn’t paying attention to who was drunk and who wasn’t. Oh, yeah. Every conversation about motorcycles, including the one I wear, Guys, I just had the most amazing day. And I’m just like, okay, that sounded like he had fun. I’m gonna remember that. Yes,

John Wessel 48:09
training on Reddit data is now

Eric Dodds 48:13
too good. All right. Well, we’re at the buzzer. I think one of my big takeaways is that being an SRE is like having children in that, you know, that really bad gut feeling that you get when you’re like, our house is way too quiet.

David Wynn 48:30
So this is an excellent analogy. Unless there is a type of maybe a type of pet owner that has a very defined cage and fence where they’re okay with silence because they know exactly where everything is. Yeah, those SRAs are out there.

Eric Dodds 48:50
All right. Well, Dave, thanks so much for joining us on the show. It was an absolute blast. And we’d love to have you back sometime soon.

David Wynn 48:57
Absolutely. We’ll do it. Take care guys.

Eric Dodds 49:01
The Data Stack Show is brought to you by RudderStack. The warehouse native customer data platform RudderStack has purpose built to help data teams turn customer data into competitive advantage. Learn more at rudderstack.com.