When Data Alone Is Not Enough – Reinventing Book Shopping at Bookshop.org with Mason Stewart

Eric Dodds: [00:00:00] Welcome back to The Data Stacks Show with Eric Dodds and Kostas Pardalis. Today, we have a really interesting guest, someone who is near and dear to me because we’ve started several businesses together.

Mason Stewart is the lead engineer at Bookshop.org, which is a really cool company he’ll tell us about.

Because I know Mason so well, I know that, you know, they do some pretty interesting things with data, specifically book related data because they sell books. Kostas, as an engineer, what do you, what, what are you most interested in? I mean, on the surface, it seems like a pretty straightforward e-commerce concept, but you’re always good at sort of looking below the surface and, and thinking through what data issues it might be facing.

Kostas Pardalis: [00:00:56] Yeah. Actually, [00:01:00] I’m very excited to hear about all the details of implementing such a symbol in a way system, as it is in common site, because I think that, both us and our, listeners, will figure out that working with even the simplest things and stuff that we take for granted like a book, it can be quite complex from a data perspective.

And especially when you do that on scale, and especially when we are talking about scaling in terms of transactions, where people, they pay their  money to, to get a specific product. So you have to do, I mean, you’re  dealing with, the money and the desires of people and you need to keep them satisfied.

So I think in general, we, pretty much  underestimate, like the technical complexity of building, something like that, in terms of scaling and also because of my background in data modeling. [00:02:00] I’m quite aware of like all the research around how, to catalog, books and archives and all that stuff, which is more of the academic, let’s say a way of dealing with this data.

I know that this can become quite complicated and tricky and very dirty also. Like this data usually, are not, let’s say the cleanest data to work with, but, every piece of data that is not clean can affect, the purchase of a person. So it’s quite important to have like the best possible quality on your data.

So, yeah, I think that’s people will be surprised, both on the complexity and the technical solutions around dealing with, an eCommerce site that works with books. So yeah, let’s, let’s see, let’s see what we can learn today.

Eric Dodds: [00:02:53] Great. Let’s dive in and talk with Mason.

All right, Mason. Welcome to The Data Stacks Show.

[00:03:00] I’ve worked with you for many years in the past, so excited to, to have you on the show.

Welcome.

Mason Stewart: [00:03:08] Thanks. Thanks for having me.

Eric Dodds: [00:03:11] Why don’t we start out by, we’d love to just hear a little bit about what you do at Bookshop and then. What Bookshop does. And, just kind of what makes it’s a pretty unique model and would love to learn a little bit more about, just the company.

Mason Stewart: [00:03:28] Yeah, totally. So, I am the lead engineer at Bookshop.org. I have a lot of very typical engineering duties. I do a lot of production work. I deal with a lot of dev ops and, you know, developing features, also just kind of trying to build some sort of a longterm roadmap for how this app is going to function, especially as it continues to grow and gain more and more traction is how do we make sure that we’ve got a really robust [00:04:00] app that can withstand the amount of, books being  purchased and, the amount of interest from the community, which has been really, really cool, but it’s also, it also is, it’s a lot of books being sold, on Bookshop.

And so, you know, our responsibility to keep, keep the systems functioning and be, you know, a really a great experience for booksellers, for readers, customers, and communities that we’re trying to serve. And it’s, it’s a big responsibility, but so far we’re having quite a bit of fun. As far as what Bookshop.org is, Bookshop is, is, on the surface a way to buy books online, but there are quite a few twists in the way that we do this that are pretty different from, really anybody else, especially when you think of like buying books from Amazon. So with Bookshop, we are specifically designed to support [00:05:00] local indie bookstores.

That’s a, that’s a huge part of what we do and also to support our affiliates who may not necessarily be indie bookstores, but they are still promoting books, promoting authors. The way that we do that is we take, a really big chunk of the cover price of a book, essentially all the profit that we would make from a book and if you have set a local bookstore, as your bookstore, you went and selected, you know, looked at your bookstore by zip code or, you know, address, geolocation, we’ll, we’ll share the profit, all the profit from that book with that bookstore that you’ve selected. The interesting thing there is that you don’t, that bookstore doesn’t have to have the book in stock.

They don’t have to ship it. They don’t really even have to know that the book exists. We send that request to our fulfillment partner and drop ship the [00:06:00] book to the bookstores. I’m sorry, not to the bookstore, to the user. The bookstore doesn’t have to have touched any of this and we’ll still give them that commission on the sale and then using Stripe Connect, they can cash out.

And, we really are basically designed as a company to, to like send the vast majority of our profits back into the indie bookstore  ecosystem. The same thing happens with affiliates. You know,  we’ll take the cut, it is a little bit less there, but we’ll take, still pretty generous commissions on any books in those to, to affiliates, to, share the link on social media or, you know, with a book club and, we also take a part of that sale and put it in the, global, I guess you would call it for lack of a better term, global pot of funds raised for bookstores that we take every six months and distribute to indie [00:07:00] bookstores and really just write them a check every six months.

I mean, it’s technically an ACH transfer, but we send that money out, every six months and if you don’t select a bookstore or don’t use an affiliate link to buy a book, we still take, a big chunk of that and put that into the global pot for all bookstores. So there’s only, only some transactions that we actually make money on but that’s kind of by design. So it’s designed to, to take as much money as we can and infuse that back into local bookstores. We believe that, that, you know, you have look at the, the about page on the site and see it written better than I’ll say it here, but, you know, local bookstores are a really big part of communities.

They’re anchors in our downtown, downtown areas, I think is what it says on our about page. And I think that’s a really good way of putting it. You know, that’s where kids are going to learn to love books. It’s where, you know, people are going to get engaged with the community. [00:08:00] They’re going to meet authors at their signings and readings.

It’s a really important thing. And personally, I have really, really fond memories of my local bookstore, growing up, being exactly that for me, I, I was able to take my son there for the first time when I went back to visit last year and it was like the bookstore I grew up going to, and I was able to take him.

So that’s something that I think everybody wants to have cares about. And, so far the, you know, we’ve really exceeded anyone’s expectations as far as the popularity. So we, we are selling enormous amounts of books every day. And, that’s, you know, it’s been really exciting to see that see that happening.

I joined a little bit after the launch, full time. So I wasn’t there for the very beginning and some of the early development stages, but even then coming into it where, you know, we’ve got, on average, a book selling every, on average, maybe 15 seconds and orders being placed. So, you know, just kind of [00:09:00] stepping directly into that scale and the company, I think only having really started opening its doors to customers, I think maybe in March. So it’s a, it’s already grown to a pretty significant scale and that’s, that’s been really exciting to see the community so excited about that.

Eric Dodds: [00:09:17] That’s crazy fifth, every 15 seconds.

Mason Stewart: [00:09:20] Yeah. I mean, that’s, that’s some rough average math, off the top of my head, but, yeah, we get about 8 million requests, 9 million requests a day.

So if you average that out, you know, it’s about a hundred requests a second. Only about half of that is cached. So, you know, our, our application on its own is handling  about 50 requests, a second all in. And so we’re, we’ve just got a lot of volume. We’ve got a lot of people on the site doing a lot of things and buying a lot of books.

So yeah, it’s, it’s quite a bit of volume.

[00:10:00] Eric Dodds: [00:10:00] Well, I know Kostas, will have a bunch of technical questions, but I’m interested to know the, you know, when you go from launching to, you know, selling a product every 15 seconds from a technical standpoint, I mean, from a business standpoint, that’s incredible, but from a technical standpoint,  that’s a lot of infrastructure to manage, to make sure that the site doesn’t go down that, you know, there aren’t latency issues with people trying to check out. Could you just explain a couple of the sort of technical challenges that you faced making sure that the app can actually still function with the pace of expansion so high.

Mason Stewart: [00:10:45] Yeah, definitely. So, you know, I definitely want to say thanks to, Happy Fund Corp, in New York, they are the agency that kind of built the initial version of Bookshop and I joined, you know, after they had launched the app [00:11:00] and they, you know, they were the initial muscle behind the app and, you know, I think it’s a testament to their ability that they were able to get the app off the ground in a relatively short amount of time and keep, keep the app running.

But you know, as far as the longterm maintenance and growth and success of this, luckily the app itself is fairly simple. So it’s, it’s a Rails application, which, you know, there may be some people who are listening, who, who find that immediately boring but I actually think that’s a good thing, right?

Well, you know, we’re not doing anything wildly exciting here. We’re, we’re doing fairly straightforward e-commerce and we’re using a really battle tested and, and very normal, framework. And really Ruby, you know, while Ruby is a slower language, as far as CPU optimization, I don’t think Ruby is our bottleneck at this point.

But you know, we, the way that book data is stored is a little bit [00:12:00] complex because it really isn’t just like, here’s a book and here’s the author and here’s the name of the boat and here’s an ISBN, there’s quite a bit going on. Right. So you have. You have, the different, you know, you would think an international, like an ISBN would be usable internationally.

It actually isn’t. So you have a lot of issues where you have different variants of books that are not necessarily, there’s no way to know, except as a human looking at them, that they are the same book, just in different editions. Dealing with, you know, I think there’s about 14 million books in our, in our books database, you know, including all variants of all books, but at that scale, you know, we have to be, we’re obviously just going to have some data that doesn’t make sense. We’re going to accidentally have authors that are misattributed and that may be because of the data from the warehouse we got had a mistake in it and maybe because of during our import, we made a [00:13:00] mistake. So managing, you know, managing the amount of data that we need to import pretty regularly is challenging because we can run up against some very long running background jobs and we have to be very careful that those are as interruptible as possible.

So making sure that the app is reasonably fast, making sure that we, you know, that we’re able to keep up with not only the demand of orders that we need to send to the fulfillment partner, but then we’re able to import, you know, the pretty vast deltas on inventory every couple of hours and that not take days complete.

So I think the thing for me, you know, to wrap that all up is that the, the issue is that just about anything that you could normally get away with in a Ruby application or any web application becomes really hard when every table has hundreds of thousands of things [00:14:00] in it if not million. I tried to load up a drop down with a bunch of, even just the, list of publishers, not thinking about the fact that, you know, there’s some 15,000 publishers in the database and, you know, a normal, a normal select box with 15,000 items is not really a great user experience for anybody. So dealing with sort of the, just the, the massive amount of even the smallest task can be a little bit challenging.

It’s definitely, it takes a lot of extra thought to even do something as simple as a, as a migration on the database.

Kostas Pardalis: [00:14:36] That’s very interesting Mason. Actually. I wanted to ask you, I mean, apart from the front end parts of, the website, which is like the later phase that your customers have with interacting and trying like to find the books that they need there’s a lot of work that needs to happen behind the scenes. And I think you’ve talked to this a little bit about like gathering all the different data that is needed around the [00:15:00] product. And when we are talking about books, I mean, they are not like the simplest kind of product to represent, right. But it’s a lot of metadata that you need to have there in order to, for the customers, like to find what they need, like descriptions, titles, ISBNs like, right. There’s a wealth of, well-structured data there that is needed. And I assume that also in your case, because you are more of a de-centralized marketplace, let’s say like you have many different, bookshops that are represented there, you probably have like some kind of like different challenges compared to, I don’t know, like a company centralized, super centralized company, like Amazon, that controls everything. So how do you deal with that? How do you collect this data? What’s the importance of this data? Where do you find this data?

Like I’m going to Amazon and to be honest, like, I don’t know how they find them, how they collect them, how they validate, all these information around the books and the different [00:16:00] titles. So can you give us a little bit more information around that? I think it’s going to be very interesting.

Mason Stewart: [00:16:04] Yeah, for sure. So one thing that’s interesting, and it takes a little while for everyone to kind of wrap their mind around the fact that, you know, we may be supporting a, you know, let’s say a bookstore downtown and your city, but while they may need to have an affiliate account so that we can pay them via Stripe Connect, we don’t need to know their inventory.  It’s really, you know, it’s, I mean, I’m sure they have wonderful inventory, but we don’t actually need to know it for the transaction because they’re not responsible for shipping. They’re not responsible for ever even touching the book. you know, it sounds like we’re just sort of giving free money to them, but that’s, that’s actually the intention is they don’t have to have done anything other than  exists as an indie bookstore. And, you know, we, we’re trying to sort of make right some of the wrongs of how, you know [00:17:00] how we’ve shifted so much our commerce away from local bookstores, because the convenience of being able to order it online and, so in that, in that sense, we don’t really need data from the bookshops, which I think is good and has allowed us to do what we do in a relatively short amount of time.

You know, they’re not having to manage their inventory on some kind of tool that we’ve built. We’re not having to import their, their inventory. So that is good. However, you know, dealing with our fulfillment partner, we, we, you know, we have a gigantic set of data. A full import can take an enormous amount of time.

And so, you know, luckily we were able to get deltas on the inventory, but even then, you know, we have to do some, some, ETL type things with it, where we take a giant inventory file. We split it into discrete files. We process them in parallel. [00:18:00] We have to, you know, deal with the fact that we may be getting an import of book cover images. not necessarily after we have gotten the books, author data. So we’re, we’re absorbing all of this different data at sort of in parallel and not necessarily in any sort of ideal order. So making sure that we store that in sort of a temporary place that can be safely written to, and then ETL into our actual, You know, sort of available inventory and making sure that’s indexed correctly with elastic search.

It’s a pretty big headache because even, even a very, very small negligible amount of additional time loading something or writing something into the database, you know, when it’s multiplied times million, millions of items. We can end up adding days to an import if we’re not careful. So optimizing those queries are [00:19:00] really, really challenging and, yeah, and dealing with the sheer amount of metadata that we have to use and get right. And the reality is that even the best fulfillment partner in the world is going to still just have human typos on an author name or an author bio. And so having an ability in our own system to go in and override that. So if an author says, hey, I went to go look at my book on Bookshop.org, but you have the wrong book listed under my works; I didn’t write this one particular book. So being able to override that, but make sure that on the next import we don’t destroy that override is also, there’s also some interesting challenges there. So, you know, trying to treat the fulfillment partner as the one source of truth is great, but also realizing that you know, that there’s going to be times where we’re just simply going to have to provide overrides for that data. Yeah, it’s, it can definitely be, [00:20:00] it could be a big challenge. It’s, you know, it’s also a challenge to make sure that we are, we are when we’re dealing with warehouses, so we’re dealing with two, two, two different fulfillment partners, one in the United States, and we’re working on one with the United Kingdom, as we’re getting ready to roll out in the UK, and one of the, the thing that we probably anyone who deals with warehouse integrations knows is that every warehouse wants to receive their order in a slightly different way, in a different format. So that may be something like generating an actual file, maybe a fixed width file and actually sending it to an FTP server.

And that’s how the order is placed, which, you know, sounds incredible in 2020, but that’s perhaps how some of these warehouses work forever, and there’s no reason, you know, they don’t have any, any reason to do anything differently, but when you start sending orders at the rapid pace that we’re sending them on, you know, there’s, there’s a responsibility on the warehouse to, you know, to try to receive those orders in a reasonable [00:21:00] amount of time, and reliably, but also on our part to make sure that we built in really robust retrying mechanisms and fail safes so that a bad FTP connection doesn’t wreck the business, you know, so yeah, it’s it’s, as I know that that’s, it’s probably, you know, just barely scratching the surface of all the things that you wanted to hear, but it is, it is a challenging thing to deal with at just about every level because of the scale, but also because of the amount of metadata that needs to be ETLed and the sometimes, you know, non-typical non rest API sort of ways of interacting with some of these vendors where, it may, it may be that we’re developing some sort of parser for a type of file that exists nowhere else except with this one warehouse, you know.

Kostas Pardalis: [00:21:57] Yeah. Yeah. Yeah. From the sounds of [00:22:00] what you’re describing I think that’s most of the industries out there and especially if you start seeing what markets are doing outside, you know, like the Silicon Valley or like the high-tech sector, you will see that like, things are very, very different from what we read on Hacker News and all these places.

And there is a reason behind that, I mean, they are building their systems like for all these decades. And there are things that are working and change might cause a lot of mistakes. So whenever you try like to expand in the market, like this you always have to deal with that. And it’s something that we keep forgetting of how important this is. So I totally  understand your point. If I understood correctly, like the way that you have structured your, your product, you have Elasticsearch, but it’s, I assume it’s like used for searching for text searching. And then you probably also use like some kind of like database system where you keep all the [00:23:00] records there. Is this correct?

Mason Stewart: [00:23:03] Yeah. So we use MySQL, as our primary data store and, I think most everyone on the team is very, very used to Postgres SQL and, during the very, very initial phases of the application, there were some features in Google cloud platform’s mySQL that were not yet available for, general use, that the team needed to deal with some of the large file format, large file imports.

So, or sorry, they were not available in Postgres, but they were available in mySQL. So we actually ended up going with mySQL, which is fine. And for the, you know, for the most part, I don’t think, you know, there’s, there’s, an enormous difference. There’s definitely some things to get used to in the different software.

But yeah, generally speaking, our data storage is fairly straightforward. We’re using, my SQL and, Elasticsearch is [00:24:00] indexed for search on the site.

Kostas Pardalis: [00:24:02] Yeah. Can you give us like a bit of an idea of like the complexity of the data model or the schema that you have to deal with for the books.

I’m not talking about the business logic that you have as a bookstore, but, for the books itself and how this is like differently represented on an inverted index like Elasticsearch and how you can, you might not like keeping, seeing, your index with your database, which is a very classical problem with these systems, but, it would be very interesting to see what kind of challenges in your case you have. Exactly, because of like these deltas that you try to introduce of the amount of volume of the data that you have to work with and, all that stuff.

Mason Stewart: [00:24:46] Sure. Yeah. So the schema for the books themselves is not particularly–so if you take the sort of e-commerce, you know, the idea of order and line items, so you take those things [00:25:00] out of it–the schema for a book is not particularly complicated, but it’s not maybe as simple as most people would guess either, right? So we have, you know, we have a contributors table because you’ll have a particular book is going to have, who wrote the forward, who wrote the afterword, who edited and also who wrote the book and were there multiple authors and you’ll even have one book maybe in different  editions, different variants having different contributors. So there’s a need to make sure that all of the contributors are correctly, not only correctly, joined to the variants, but that we’re listing them in an order that makes sense. Right. you know, if you’re, if you’re, the primary contributor to a book that’s important, you probably need to be listed differently than someone who, you know, just wrote the forward.

[00:26:00] There’s contributor,  metadata. There’s the metadata for the book itself. There are all sorts of different categories that are called BISAC categories and they are nested categories that can go pretty deep on a book as far as how a book is categorized. So it may be,  there’s a BISAC for say science fiction, but then there’s hundreds of subcategories of science fiction, right?

So a book can belong to multiple BISAC categories and, you have a different language variance of the books. It’s not particularly crazy from a data structure, but it’s a little bit more complicated than, than would probably be guessed. As far as indexing with Elasticsearch, I, you know, we, I’m of the opinion that you could probably have somebody’s full time job, be just working on Elasticsearch for this application. And that would be a really good use of time. We [00:27:00] definitely don’t have that. So, you know, we, we definitely have to deal with issues where let’s say a warehouse has just simply attached the wrong ISBN to a book.

And we, we have imported that book. We have set up all the data structure has been indexed in Elasticsearch and  then in a delta they’ve corrected their mistake, but the deltas are large enough that we’re not necessarily going to be paying attention to every you know, every, every small change like this, but we can end up in situations where a book is still indexed and elastic.

It’s not actually available anywhere with the warehouse our system may still have a record of it. It may not. It may be that a book order has been placed and is waiting to be transmitted. And then the warehouse removes that ISBN from their database. So there are a bunch of a tricky issues, not only with the, the, having such a large, large Elasticsearch index that [00:28:00] re-indexing can take a significant amount of time, but also, Having, you know, trying to cache this stuff effectively. Right? So as we’re trying to store things in the cache and really reduce the load on our actual database, it can be very easy with this amount of, products to just simply tell somebody that something is still available when it may not be, or may not be available in the same way.

So trying to mitigate that can consume an enormous amount of time. But, yeah, it’s, it’s, it’s funny because you know, for such a simple data structure, it really can still end up in a lot of complication. I think too, the e-commerce, Ruby gem that we use, Solidus, is, you know, it, has  a lot of features it’s been around for a very long time, but it has a lot, it tries to deal with a lot of concerns that don’t actually matter in our particular case because we’re selling books and because they, they vary in very specific [00:29:00] ways. So we definitely have some aspects of Solidus the Ruby gem that we don’t actually use, but that aren’t necessarily simple to remove. Right. So we may end up having to touch quite a few tables, just to update a book, to make sure that all of our eCommerce data has been updated and the pricing and the availability and the stock and everything has been updated, but also then dealing with all of the metadata of the book. So, yeah dealing with Elastic is it can be very easy to really ruin the search results on the website. Yeah. We have done that a couple of times, even since I’ve started and, and, you know, the, the, the problem is, is that it’s really not simple to see that mistake until someone starts to point out that.

You know, for whatever searches you’re doing to make sure that the end debt re-indexing makes sense. There’s going to be somebody else. Who’s doing a [00:30:00] completely normal search on the site as a customer and they’re like, “This is a New York Times bestseller, why is it not in the search results?” And we have to go realize that the, the, the particular combination of title and everything just happened to be words that are deprioritized or ignored, in our search.

So it’s a challenge. I mean, it’s, I, I barely scratched the surface on everything that we’re doing as far as our Elastic goes. And even then it’s, it’s a lot to take in.

Kostas Pardalis: [00:30:33] Yeah, absolutely. So, the way that you describe it from what it sounds like, consistency, not only internally, I mean, between your Elasticsearch and your database, but especially with the warehouses that you have to work together because the whole phase that you have with them at the end is around data. Right? You need to have like the right ISBN, you have to send like the, the, the right information to them [00:31:00] back and forth. And there might be like mistakes that might come from them. So what kind of, like have you figured out any kind of like mechanism to identify these kinds of issues?

I mean, you mentioned that many times, like, the customers itself themselves, like they find the instructions. They, they, they let you know because, okay. It’s a very hard problem when it comes to search. But in terms of like the consistency between you and the rest of the vendors that you work with, and interface through data, what have you figured out so far that it works and how hard as a problem it is at the end, for a business?

Mason Stewart: [00:31:38] Yeah. So that’s a good question. So, you know, there’s kind of a couple of layers to it. So one is when we’re placing an order, which is, you know, in some ways you could consider it the most, the most important as what we’re effectively saying is you told us, you have this thing, you told us, this is the ID of the thing.

Okay. This customer wants this many of [00:32:00] this thing. And as simple as that conversation is, there are so many things that can go wrong, particularly because of the way that books are ordered and how backorders and preorders are dealt with. So you may have a situation where you say, well, look, you know, we’ll, we’ll, we’ll accept something will be on back order, but only up until a certain amount of time. So 30 days, 60 days, whatever you set that number to, and if they can’t fill the order, by that point in time, you’d want to cancel the order. Right. Because there’s just only a certain amount of time that a customer will want to have given you their money and wait for a back ordered book.

But that’s a bit different from preorders where you may have an extraordinary amount of time before the book is actually released. You may even have delays. I had a book  pre-ordered since, August of last year and, it got delayed multiple times, so it was technically pre-ordered, for a whole year.

[00:33:00] So you have those situations. And I think the thing that we’ve found so far still is that when something goes haywire with an order, it’s important enough to just send it to a human to be looked at. Right. So, like we can, we can try to recognize patterns. We can try to recognize trends and those are things that are important and that we should do.

But at the end of the day, a human is still going to be really great. It’s just looking at the order and being like, this should not have been canceled, or I know this book is in stock, something’s wrong. Oh, we’ve mangled an ISBN or the order was in some way, malformed. Or, you know, we just, it could be that on either side of us, a warehouse or us, we just, we have an outage, right.

Where someone has an outage where they just simply can’t process orders during some hopefully small timeframe and, you know, trying to discover, well, what did did they actually receive them? And we didn’t simply get the acknowledgement or, you know, all those different back and [00:34:00] forth. So there’s a surprising amount of human intervention that has to happen, even when things are working, seemingly, not AKA, but things are seemingly working well and orders are going through and, you know, we’re collecting money and people are getting their books. There’s still even then a lot of just, really making sure that, you know, those orders go through. We’re sort of seeing the same thing with the, you know, as we’re integrating into the UK is, in our fulfillment partner there, even in their documentation, says you probably won’t be able to code around every, you know, every possible exception that could be raised. You know, we really wanna encourage you to like, look at these as a human and try to understand what’s happening. So that’s you know, it’s a, it’s always going to be a challenge between, do we raise an exception because something strange has happened or, and did we stop execution? Do we stop processing or do we [00:35:00] want to assume that this may actually be okay and what level of trust do we want to put in, you know, our partners and ourselves to make sure the order goes through, even if something that seemingly mysterious happened?

I accidentally sent an order, to our UK fulfillment partner that had no shipping or billing address or name. So it was just like, I just sent them an ISBN with no information about where, what they should do with it.

And it actually like, because of some interesting things, it, the order wasn’t necessarily rejected. They actually just emailed and said, Hey we don’t know what to do with this order. Right. And so, in a sense, that’s actually pretty nice because that was an honest mistake. The trick, you know, the trick is what do we do, you know, how do we protect ourselves from sending huge amounts of orders that may not necessarily fail validation, but will require human intervention. So threading that needle is honestly, it’s a challenge, right? There’s nothing, there’s nothing like having a person [00:36:00] look at some bad data and figure out what’s going wrong, but there’s also a limit to how much of that we can do in a 24-hour span with a small staff.

Eric Dodds: [00:36:09] Mason, I mean, number one, it’s just amazing to hear about the plumbing complexity behind just ordering a book. So thank you for everyone who’s a purchased on Bookshop for all the work you’re doing on the backend to actually make that experiencing really simple to us. Stepping back a little bit. One thing we like to do on the show is just ask about the stack. So we know that it’s a, a Ruby app, but, it sounds like you have a pretty heavy duty warehouse set up to manage different jobs and I would love to know more about the infrastructure. And you said you work on some dev ops stuff too, and [00:37:00] any tooling around, you know, sort of the data pipeline flow in the stack would be interesting.

Mason Stewart: [00:37:06] Yeah, totally. So, so it’s kind of interesting because you’ve got the first version of the app out and that’s working for the United States and you know, it’s a lot of just sort of figuring stuff out on the fly and again, hats off to the folks over at HFC for even getting the app to run, and, and keep up with the, with the traffic.

But we’ve been working on the UK version of things and we’re trying to make, we’re trying to make some small improvements in the UK, on kind of a separate branch that we hopefully can merge back and to the US master branch. Cause right now we’re technically deploying two different versions of the app.

one for the UK and one for the US. The UK version, we are actually trying to run on Kubernetes, which is, you know, the Google’s distributed or not [00:38:00] distributed the Docker orchestration library. I really like Kubernetes I’ve used it before. And, it definitely has a little bit of a learning curve, but it seems like a good choice. It’s made it easy for us to deal with, even though we’re really not doing microservices, we still have some needs for, you know, having some kind of unique one-off, jobs and job queues and, you know, small services and then it might count as microservices, but that’s really not generally the architecture we’re working with is it’s generally a monolith, so Kubernetes is great.

We’re we’re on Google cloud platform. You know, I think that generally speaking, you know, because we sort of positioned ourselves as, an alternative to Amazon, it seemed like it was maybe in bad taste to host the app on AWS, on Google cloud platform. [00:39:00] Sure. We’re using Redis and Sidekick as our job queuing system.

And Redis is sort of a masterpiece at this point. I mean, it’s a really, really fine piece of software as far as dealing with, in a rails application, dealing with, you know, job queues and I think we’re, we’re going to pretty quickly end up, because we’ve, we’ve made some of these jobs very, very granular where we will have hundreds of thousands or millions of jobs running in the queue and, and not necessarily running in parallel, but we’ll have a lot of jobs in the queue.

I’ve tried to structure the new version of how fulfillment works in a bit of an event stream. So it’s not necessarily a proper event stream. you know, we’re not trying to reinvent Kafka, but there are tools like Kafka, but, [00:40:00] it’s really, really important when we are dealing with orders containing lots of items and separate shipping addresses and so many little things can both succeed and fail that we treat those things as atomic actions. And that we have a very, very clear paper trail of exactly what events happened, when and where and why and how our, what our strategy is for when the event fails. How do we want to retry it? Should we retry it?

So that’s been really exciting to kind of see that, coming along because the first version of the app did have some ideas like that baked into the app. I’m trying to formalize them so that the event stream for fulfillment processing really doesn’t know anything about the underlying data structures and the e-commerce Ruby gem Solidus.

So there’s, there’s really a very, very small layer that has knowledge of the events and has knowledge of e-commerce [00:41:00] data structure. But otherwise it’s, it’s, you know, it’s trying to be an event stream and I think we’re making some really good progress there but aside from GCP, mySQL, rails, and Redis, I don’t know that we’re really doing anything extraordinary. The hard part, the hardest parts of this app are by far dealing with things for which there are no really, there are no Ruby gems and there aren’t necessarily tools. So, we, we wrote, for example, we’ve been working on a, a wrapper around Ruby’s, standard library FTP, to add a whole bunch of modern niceties and robustness and retry-ability and things.

And, you know, you would think that that library would already exist somewhere that a Ruby gem would exist, but I just don’t know that most Ruby users are having to deal with [00:42:00] placing, you know, 20,000 FTP orders in a very short amount of time with a probably pretty slow, you know, FTP server that has probably been around for a very long time.

So making sure that we’re, we’re dealing with that responsibly and we’re not crushing the underlying FTP server, but that we also can, can really try to increase the guarantee that those things will get over the wire so yeah, the stack, I mean, the stack is actually it’s, it’s actually pretty boring.

I mean, And I think that’s a good thing for us. I would be a little worried if we were doing a lot of super crazy edge stuff here because, by keeping it pretty boring, we’re, we’re more or less kind of matching the, you know, what, what, what we’re seeing with, with various vendors where we’re not going to necessarily be seeing, anybody, any of our partners doing bleeding edge stuff.

And in a sense, that’s nice because that means things are probably not going to change and we can implement things and trust that that will work for a long time, but [00:43:00] it does kind of warrant a different approach than know everyone’s got a fancy graph, QL and, API, and, you know, it’s a, it’s a little bit of a different set of concerns, but I kinda like it.

I actually really enjoy it.

Kostas Pardalis: [00:43:15] Yeah, I think Mason, on the way that you say it I totally agree with this approach that you’re using the technology to solve the business problem and not focusing on solving technology problems that many companies have to do when they adopt, you know, like the latest, super interesting and state of the art technology and I think that, especially when we are talking about eCommerce and about dealing with the desires of people, their purchases and their money, we need to be boring. Like that’s a, that’s a way to do it.

Mason Stewart: [00:43:50] I think too you know, I should mention it now that you mentioned that. One of the things that’s exceptionally boring, but I think it’s absolutely a wonderful part of Bookshop is [00:44:00] we really don’t make algorithmic recommendations about reading or about books. So we really don’t like, we don’t look at the things you purchased and then try to make, you know, some sort of algorithmic assumption about what you’d like, when you’re on the homepage of Bookshop, it’s, it’s recommended lists from actual affiliates and users and indie bookstores and our curated lists.

And I actually really enjoy that because I think the results while sometimes, you know, you may have already read everything on there and you may, you know, it may not be the most exciting, you know, like suggested for you type results. But honestly, the results I think are excellent because humans curate them and we actually have strayed from trying to do this automatically, programmatically.

And I think it works out really well because not only do these lists, they’re attached to affiliates. So affiliates in indie bookstores will get their commissions from these, but, it’s, it’s [00:45:00] also, just great recommendations. And sometimes that can be really hard for anyone who’s ever been just unbelievably bored with the recommendations in Spotify, because you’re like, I don’t want to listen to any of this stuff. Why is this what you’re telling me to listen to? You know, I typically have a distaste for, for over sort of algorithmic recommendations, but, yeah, I think, yeah, that’s right. I think that the boring, you know, when we’re talking about people’s money and we’re talking about bookstores livelihoods and, and we’re talking about making recommendations that are, that are really actually great recommendations we’ve gone a pretty boring route and I actually think that’s been in our favor.

Kostas Pardalis: [00:45:40] Yeah, I think, I’m sorry. Go ahead. Go ahead.

Eric Dodds: [00:45:44] I’m going to say, I think, I mean, not to get too philosophical, but I think, you know, when you, when you think about the product itself, a book and recommendations the, like [00:46:00] one of the great things about books that I love is that they can expose you to new ideas, and ideas that challenge you, but a recommendation engine that continually zeroes in, on things that you’ve already read doesn’t necessarily serve, serve you well, in terms of exposing you to new things. So it’s fascinating to hear that the human curation works really well for you. Yeah. As opposed to the algorithmic recommendation.

Mason Stewart: [00:46:29] Yeah.

Kostas Pardalis: [00:46:31] Yeah, that’s a, that’s an amazing point. Actually, all the time that both of you were talking about these, I was thinking about some latest discussions that I’ve seen, where like people were talking about places like Goodreads or Reddit subreddits for reading and recommendation around books and it looks like okay, like technology, helps a lot to scale up things, right? Like having a recommendation algorithm [00:47:00] there really helps with that. Although at the end you reach a plateau where like the technology is not enough anymore to provide the right content for people, because okay. Like at the end, reading, listening to music and all these activities, they’re also like, very social events. Right? Like I read the book because I heard about something from a group that I care about from my community and all these things that you also mentioned   like at the beginning about like the importance of the bookstore as part of the community. And I think, we’re going again, it’s a bit of more philosophical point, but I think it’s quite important.

And that’s where we see like the limits of technology at this point with all this algorithm. So, having said all of that and it’s great that you’re actually trying to do that. I mean, it’s a very refreshing idea in, driving eCommerce around books, any, and this is [00:48:00] like the last question, after that we can close this amazing discussion we’ve had. Any thoughts about also building Goodreads? So many people are still using it and, complaining about Amazon pretty much has abandoned it and they are not updating it.

Mason Stewart: [00:48:15] That’s so that’s a question we get asked quite a bit and you know, I don’t, I don’t know that I have a clear answer on specific plans to do one thing or another. That would be something you could say. It’s an alternative to Goodreads per se. But I think the thing, I think the reason people ask that is that when they interact with Bookshop and they see like the engagement we have with the community, and the, and the good we’re hopefully able to do in the community, we, I think everyone sort of sees the connection that like tools like Goodreads help, they help communities. They help book clubs, they help [00:49:00] bookstores. They help people connect. And especially during a pandemic, like to connect in ways that they maybe couldn’t in person. And also maybe in ways that like might not totally make sense over Zoom.

And so I think that, I think that there’s a real desire in a lot of communities to have a way to facilitate book clubs to facilitate discovering new things and interacting with authors. And so we’re starting to do some of that with Bookshop where we, we’re, we’re hosting author events, digital author events now, and those are actually going great.

And, you know, we’re, we’re excited to have more community events and I think we’re excited to figure out ways that we can further help communities read and enjoy books and help people discover new things, whether or not that ends up looking like something that’s built into Bookshop itself or whether it feels like a more external tool.

I don’t know. But I, I do know, I know that that’s something that we really are excited about. And especially just because people keep asking, [00:50:00] like, how do I share my, you know review on Bookshop? You can’t really leave reviews at all. Yeah. There’s just no mechanism for that. And I think people want to share their thoughts and share, you know, what they got out of books and what they’re excited about reading.

So finding ways to do that with our users who are already really engaged, I think would be really exciting. But yeah, I don’t, I don’t know that we have any specific plans right now. Right now. I’m just trying to, we’re just trying to get that, things up and running in the UK for Christmas so we can hopefully help, you know, help a lot of bookstores over there with the holidays coming up. So we’ve got a lot on our plate right now, but it’s definitely something we’re we’re talking about internally.

Eric Dodds: [00:50:41] Yeah. I was going to say migrating to Kubernetes is, that sounds like enough work in itself.

Mason Stewart: [00:50:47] Yeah. That’s, that’s a, that’s surprisingly been the easy part but yeah, it is, it is, it is a definitely a lot of challenges.

So we’ve, we’ve got a good team and I’m excited. I don’t [00:51:00] have any doubt that we will be able to continue to build a great product and, and help, bookstores. But, it, the bigger this gets and the more people that buy books, the bigger the throughput, the more, the, well, the less room there is to get it wrong.

Right. So we’ve, we’ve got to really, really get it right going forward. But yeah, I’m excited.

Eric Dodds: [00:51:22] It’s been, it’s been truly enjoyable to hear about the, I mean, sort of in many ways, chaos behind the simplicity of, you know, of selling books online. So thank you for the time as always you approach things in such a thoughtful way and I think our listeners will really love just hearing them all the ways that you’re trying to solve those problems. So thanks for taking the time to join us on the show today.

Mason Stewart: [00:51:48] Yeah, absolutely. Thanks to both of y’all. I really appreciate it.

Kostas Pardalis: [00:51:52] Well, that was great. I think the conversation with Mason was more than interesting that I would ever expect to [00:52:00] be honest, Eric.

And one of the most amazing outcomes is how important boring technology can be at the end, when you are dealing with business problems at scale, like the one, of building an eCommerce site to, sell books. What do you think about this?

Eric Dodds: [00:52:21] Yeah, I think that, you know, I’ve known Mason for a long time. And I think one of the qualities that I appreciated most about him really came through in our discussion. And that is that he is very thoughtful about the way to solve problems and deliberately choosing what he described as a boring stack in order to provide a consistent stable experience is to me, it just a really mature decision.

And I think the other thing that just blew me away was I’ve purchased books from Bookshop [00:53:00] and it’s such a seamless experience. And I just had no idea how much they’re dealing with on the backend to actually make it seamless. So it was, it was very eye-opening for me and just appreciate all of the thought that’s going into it.

So, yeah, I would love to, we should absolutely catch up with Mason after the Kubernetes migration just to hear how that, hear, how that goes.

Kostas Pardalis: [00:53:28] Absolutely. I think you put it very well. I mean, what Mason was calling a boring stack. I think at the end, we should call the mature stack. And when you’re dealing with, important and difficult business problems, that’s I think that’s also as the engineers would approach these problems.

I think the approach that Mason has, it’s like a very mature, as you said, the approach on dealing with hard business-related problems and solving them using the available technology that we have today. And, yeah, I’m really looking forward [00:54:00] to talk with him again and learn more, more about his experience with Kubernetes and also, their experience with expanding the combined needs more countries and see what kind of, complexities this is going to introduce to both the technology and, and the business.

Thank you everyone for being with us today and sharing in the stories of Mason and Bookshop. Talk to you next time.