Interview with Robert Cook

Welcome to building better developers, the developer podcast, where we work on getting better step by step professionally and personally. Let's get started. Well, hello and welcome back. We are continuing our season where we're doing a bunch of interviews and we are starting a new interview. This episode and next one, we'll be speaking with Robert Cook and we're going to get again into some technical stuff. Robert is somebody that has been doing a long period of time doing software and really getting into some high, high efficiency, highly quality type code, things that are like, you know, we want to run really tight, really fast. And I think even if you are sort of a developer, if you've got somewhat of an interest in technology and solving problems, you're going to find that this will be a very interesting conversation because he goes deep in some areas. I think there's a lot of things that we can pull out and apply to a lot of the various problems that we have to solve on a daily basis. So let's go ahead and get started with our discussion with Robert Cook. All right. Today we are speaking with Robert Cook. We are going to get into a little more technical stuff this time around. We're going to make sure that we're getting into this with somebody that has a hundred thousand hours of software development under his belt, which is more than most of us, maybe more than a lot of us combined with that. As we've often had, we talked to people who've been in this for a while. We're going to talk about how things have changed. And then in particular, what, you know, as things have changed, what he sort of sees in his hourglass moving forward. And some of it will be sort of a spoiler alert early on as we talk about where he's at, what he's doing and what his focus is. And I think you'll very quickly find that this is going to be an area that we haven't spent as much time talking about over the last year or two, but is definitely an area where there's a lot of advancement. And I won't spoil that one too much because I'm going to go right over to Robert and say, hey, why don't you go ahead? Thanks for coming out. Thanks for talking with us. And why don't you give us a little bit of your background and tell us where you're coming from. Thanks, Rob. Thanks for having me on. So my background, well, I think the hundred thousand hours sums it up. That's not a lot of time for anything else when you're writing that much code. No, I have ever since it's, you know, I've kind of realized later in life that I don't know what time in my life that I wasn't into programming and into coding. Other than maybe when I was really young, I liked Legos. I liked building stuff. In fact, any hobbies I have is around building stuff. I'm almost like addicted to building. I just like building, building, building anything there is. Maybe not a key of furniture, but other than that, I might even enjoy that more than most people. But I really do like building stuff. But one thing I realized when I was a kid was that, you know, as you want to get into Lego sets and toys and building blocks, whatever it is, you have to spend more money to build something more interesting. Right. The bigger Lego sets always cost more naturally. And then when I kind of started, I knew there was something mysterious about computers that I figured I would love. But then once I got a computer and I really struggled to get into programming, because this is the early 80s when, you know, you could buy a book for $50 at your local bookstore. But other than that, it was very hard to learn. There wasn't really a lot of community or things like that. But I realized that I could write more interesting programs and better software and bigger programs. And it didn't cost more money. And so that fascinated me. What fascinated me about it was I felt as though it kind of leveled this playing field. And that's why I got into it. Because I had, I mean, I had more toys than some of the kids that their parents spent less money on. And I had less toys than some of the kids that spent more money. But when it came to computers, it was an even playing field. And so that really got me into it and got me going. And then I think as time evolves, I've really become more fascinated with it for even even different reasons, which is I've realized in most professions, and I talk to most people my age that have been doing a particular profession their whole lives, you start to kind of plateau or maybe you learn through things and kind of broaden. But with software, it's like every year, and I've been doing this for almost 40 years now, every year I can look at what I've done the year before that and say, man, I could do that way better. And I think that's just maybe I was just really, really bad at it at the beginning. I don't know. But, you know, it's it's it I still believe that it's like this craft that we're just learning almost as a society and on an individualistic level. So I guess my background to kind of deviate here a little bit, I've been in programming my whole life. I was doing any software any way I could, all the way from a kid building my accounting system for the club that I was in all the way up to working at some of the largest tier one banks. And then I started ThreeForge in 2012, which has just allowed me to execute my vision of basically building this platform that has all the different ingredients that that B2B that large corporations are looking for to rapidly build applications. So I started that in 2012. Here I am. I can't believe it's already been 10 or 11 years. But at this point, our work product is in most of the tier one banks, you know, as licensed software, and then they build hundreds of different use cases, tens of thousands of users are using our software daily. So That's not a bad start. So you've been you've been a little busy. Yes, I have. Yes. Yeah, that was I have to I have to say learning software in the 80s, when you had to eat. Yeah, you had to go buy a $50 book, but then you had to read it. And then you had to understand it. And you didn't have that community. It is a is a world of difference these days to be able to, you know, to Google and find 100,000 people, it feels like that have done the same problem and all the ways they can, they can walk you through it. It's made it free. It's a lot easier to learn as it's gone on. I don't know how much of that is because, you know, like you said, maybe you start off not knowing much. But then I think some of it is it just seems so much easier, even things that you get into areas of technology, they're maybe completely foreign. It's like, I haven't tried that at all. And the next thing you know, you've got pretty solid background of it, because you've got so many people really in an effect, you know, helping you out with it. So it does make it easier for that, that always being better kind of approach, where it's like, you know, you can always look back and say, I did this a little differently. And then you bump up a crit against somebody else has done a little bit different. And next thing you know, you're like, Oh, you know, that was a whole different, that's like, it's a different approach than I took, but actually, it's a better approach. So go back and change it up. But I do think, you know, when it comes to it, I didn't mean to cut you off. I'm sorry. No, no, go right ahead. Okay. So, you know, when I when I look back on it, I do cherish what it means to open a textbook and be able to just read almost in a linear fashion, you start at the beginning of the chapter, you read to the end of the chapter, and you end up picking up some things that maybe aren't directly in the limelight of what you're trying to solve. But going through that, you learn a lot. And I'll tell you an example that happened recently. So we have our own database, whoever we have actually, we built several databases. One of the databases we wrote recently is this columnar database, you know, and I guess I'll get a little bit technical here. But one of the things we needed to solve was a B tree and a B tree is where you have blocks and they're on disk. And then you, you know, you try to basically optimize your each node so it fills up a block size on disk. And you can find some literature online. And I actually went back to my books, Edgar Codd and CJ dates, they were kind of like the authors of the database all the way back to the 80s. And the material was actually so good and so relevant, it was more useful than what I could find online. And maybe, you know, that's just anecdotal. And maybe that isn't common case. But, you know, I do think one thing I would suggest is, it is so easy to go to Stack Overflow. I got to admit, I go to Stack Overflow a lot, I'm not going to deny it. You know, that's like a cliche thing now. You can search just about anything and find what and how to fix it by just going to Stack Overflow. But at the same time, when it comes to really building interesting stuff and differentiating stuff and things like that, there is still a place for academia and being able to just step back and really consume a body of material to help you move forward personally on a topic. Yeah, that's one of the things I've found is a challenge that's a little different now. If you go back, you know, now it's probably 30 years, something like that. Everybody had a foundation that was somewhat the same because you sort of had to go through the same, you had to do a computer science degree and you had to sort of work through, everybody sort of did the same languages and stuff like that. Now there's, I don't know, four billion different programming languages. So you've got different colleges, different courses, different focuses, and then you've got all these boot camps and other ways that people learn stuff that I think it does. I think you lose something that you lose some of the fundamentals of not just, you know, sometimes it's a language, sometimes it's actually of computing itself. It's just sort of like, oh, well, I can go find this algorithm that solves my B-tree problems and I'm done. And then you miss it. And then it's, especially when it breaks, they say, well, I don't know how that works. It's like, well, you wrote it. So, well, nope. No, I just copy and paste it offline. Exactly. And it works fine. Right, right. But I think, you know, and I also think a lot of times by implementing it yourself, it is a valuable lesson as opposed to, and I think this kind of goes without saying, but when you copy and paste, you're kind of like, well, it works. I'm not quite sure why, or maybe I get the gist of it. But if you actually sit down and write it, and even if you make some mistakes along the lines, along the way, I mean, maybe, you know, you can kind of draw some tips from the web. But the idea of actually sitting down and writing something interesting from scratch, I think is just such an awesome learning experience. And I still like doing it. Another thing I'll mention too is, and I can't say this necessarily applies for everyone, but, you know, we build a framework, which basically means our code is used in ways that we can't, quote unquote, imagine. It's just like anyone who's building a language. You know, when you build a language, the idea is it should be industry and data agnostic. It's just a language and then people can use it however they want. I mean, computer science language. And I think that by having to build generic software, you're kind of forced down this path where you have to focus on flexibility and you have to focus on performance. And performance is actually a very big deal for us. And that's one of the things that I think is really lacking when you start to go online and read stuff is usually it's just how do you get the job done? And I mean, okay, well, you know, we'll create the string, we'll trim the string, we'll add to the string, we'll divide by the string, we'll take everything after the dot, and then we'll bop, bop, bop. And string manipulation in and of itself is slow. But the answer, it's almost like who cares how hard the computer has to work to get the answer, we just want to get it done. I'm not going to say that isn't exactly, that makes a lot of sense when you're kind of trying to do side hustle type projects. Perfectly fine. But when you're building core infrastructure, or when you're building systems that are going to be used in ways that you can't imagine, you really do want to care about the performance of computer. And that's again, where I think going back and understanding how a computer works, and understanding how and why the software you're copying online, offline works the way it does, that's now very important. Chris Bounds Oh, I agree. That seems to be always that sort of the next tier that you often see that people go in, they build their software, you get version 1.0, you know, the classic thing. And then almost inevitably, that's the next thing is, oh, it's a performance release, or it's a tuning release, because it gets done. And then you realize, oh, yeah, it works. But works has got to be like in quotations, and you got asterisks around it, because it works, but it takes, you know, decades for it to run, as opposed to the two second response time you're looking for. And so it is a different, it's a different skill set. So how do you guys go about looking for or tackling that problem from a, from a, I guess, more like a staffing and a knowledge point of view of how you build out a team that makes sure that that is something that gets addressed? Right. Well, I will, I've said this on every podcast where I get the opportunity is learn math. You know, I am a big believer, and I'm not saying you have to be able to, you know, do integrals or things like that, although that helps. But having a solid understanding of the basics of math, I think is a very important thing. And a lot of my interviewing is around that when I hire people, I focus on data structures a lot. A lot of times you see the resume comes across and you mention the 4 billion different languages and databases. By the way, out of side note, I should say we, we have a data virtualization layer, we connect to different databases. And I think we probably add a new database every week. That's how often new technology, it's dumb founding how often they come out. And so that's kind of a side thing. But at the end of the day, when I see the resume with tons of languages and tons of databases and all the sort of experiences on there, I think that's interesting because it kind of gives you, it's at that point, you're just getting a sampling of how different people think about how to solve problems, if that makes sense. You know what I mean? Like if you know Python and you know JavaScript and you know Java and you know C++, now you've got an idea of how the different engineers and why those languages exist and different ways to think about it. But beyond that, I still think, you know, talking about and understanding data structures and math to me is a critical thing. And Steve Jobs, who I'm not an Apple fan, I don't own an Apple, but I really am a big fan of Steve Jobs himself. And he had this one interview back in the 90s, where he kind of talks about how, you know, the fascinating thing about developers is if you hire a good developer, he can be a hundred times more productive than a bad developer. And in fact, a bad developer can be negative productivity, in my opinion. You know, I mean, they can actually write more technical debt than good. That's a very possible outcome. And so to me, how do you find really good, really, really good developers? It's by understanding their problem-solving skills and their math skills. And I can do that very quickly through data structure type of questions. And honestly, even if they don't know how to code very well, that I feel like can be taught. You know what I mean? Like you can teach someone how to write for loops and while statements and lambdas and things like that. But understanding the basic math to me, that's a big thing. So I can't state that enough that that's what I focus on. Chris Bounds I would agree that that is, if you've got those concepts, then it makes a huge difference. That allows you to adapt to whatever the environment is, whatever the language is, whatever the database is. If you know under, and it goes back to you mentioned earlier, being essentially data agnostic and things like that. If you understand how data should work, how it should move, the difference between storing stuff in the various different ways you can do it, computational things, and how that can make things more or less complicated, then it really does. Then you get down, you're really solving the core problem. And then it just comes down to the language. So how does this language solve that problem? Exactly. How does it help? Yeah, exactly. And as a way of kind of comforting the listener, I mean, I will admit that when I first got into computers, I loved computers. And I really wasn't that interested in math and my parents and everyone's like, oh, you got to learn math? Like, yeah, whatever. I don't need to know math because I know computers. Computers can do math for me. And I wasted a lot of time on that mentality. I'm not going to say a lot of time. I mean, once I was in like high school and college and I wanted to really start doing interesting things with computers, I realized, yes, math, math really matters, especially I started getting into like 3D games and stuff like that. Math becomes very, very critical. So it's not like you have to learn math and then programming. You can learn programming and then and then learn the math and you'll be that much better off. But I think, you know, understanding the two, that's where the power really comes together. Yeah, because at some point you have to tell the computer how to solve that problem. If you don't understand, if you don't understand the math, the computer is not going to just be like, oh, hey, here's how it gets done. Exactly. You know, you may have a, you know, sometimes you'll have a, especially these days and particularly like databases, you'll have functions and stuff built in that sort of do that very complicated stuff, but you still need to have that foundation to understand what does it do when I tell it, when I'm, you know, I'm calling this function, what is it doing with this? What should I expect out of this? Because you probably run into this as well. Sometimes it doesn't do exactly what you think it's supposed to do. And then you're in trouble. You've got to figure out how do I, how do I verify it? Exactly. And I think databases are actually an excellent example of that because there is no one size fits all database. In my opinion, that's well, that's, I don't think we need to have the thousands of different types of databases that are out there, but I do think there are many types of databases that are necessary for different types of problems. And then even within a given database framework, there's still the right way and wrong way, in my opinion, to store stuff, depending on how you want to retrieve it later, what type of data you have, the different types of cardinality of that data, all those things play into that. And again, it comes back to math as to how you want to organize and sort, be able to retrieve that information quickly. And I will say that as sad as it is, I feel like when I look at systems that were written 20 or 30 years ago, there's a lot more attention paid to efficiency and math because they had to. And it's actually quite remarkable how many, I mean, some of these systems like the airline industry is showing its age, but a lot of these systems ran very, very well for a very long time because they paid attention to this. And I think a little bit of that would go a long way these days because computers are way more powerful, but it doesn't mean you can just evade having to think about performance. Yeah, I think that is a little bit of a crutch. It used to be that you could only write, your code had to fit in a certain amount of space or otherwise you'd run out of your memory and have to deal with extents and stuff like that. And you didn't have gigabytes of memory, you had like K or less of memory. And so you had to be able to fit all of those structures in. You had to understand how to move things around and how to be efficient. And now it's allowed you to just sort of, there's a lot of that stuff that doesn't come into play until you get into like, you know, like finance and banking where you're doing, it's great if you're doing, you know, a hundred thousand transactions, but if you crank that up to a billion transactions, suddenly, you know, if it's not done right, then you hit a curve. And the next thing you know, you've brought everything to its knees. And I think indexing data is probably, yeah, is one of the best examples is you look at that and you say, actually indexing the other one I've always, I've sent people to is just like, look at all the different ways to sort data. When you go back to like, you know, bubble sorts and all the different sort types that there are and understanding just walking through those kinds of problems and realizing it's like, oh, hey, we've got a, there is a difference whether I just like pick a thing and then try to find the next thing after it in a list or whether I have some sort of a thoughtful approach to how I do it. **JASON LENGSTORF** Well, if you've ever, I don't know if you're familiar with the Tim sort, but that's the sort that's used in Java now. And the guy's name is Tim. I don't remember his last name, but maybe you should have named it Tim plus his last name. Because everyone only knows his first name. No, but what's really fascinating about it, and this information is a year old, so maybe they've moved on past this, but what's so fascinating when you start to look through the code is what it does, the first thing it does is it just samples the data very quickly. And then it decides what strategy to go to. So for example, samples the data and it says, well, it's all looks already sorted. Then it's going to go down a path just assuming it's already sorted. Maybe it's already all reverse sorted. So it samples the data and then it decides what algorithm to use after that based on its sampling. So it's a very interesting thing where, but the reason I bring that up is because there's a cost to doing that sampling. And if you know ahead of time how that data is organized, you can provide those hints and then it gets the answer faster. Yeah, sorting is an excellent example of something where it can matter. And by the way, I will say when I went into finance, I was out in California for a few years in early 2000s, then I moved into finance and I thought it would be kind of a boring sort of like just moving numbers around and kind of people deposit their checks and it creates an electronic statement or something like that. But I got into high frequency trading and then I realized that this had everything I loved about computers. And I've distilled it down to the four Vs and hopefully I can get them all. Velocity of data, the variety of data, the validity of data, and the volume of data. And all four of those is like this recipe for disaster. I don't know how to put it. It's like everything has to work. And the thing that's really crazy about high frequency trading is that you're literally writing software. It's like a hackathon competition with huge dollars behind it because you're literally trying to write software that's going to solve a problem faster than the other bank. Whichever piece of software can pull in the market data, understand the position of the market, and then execute a trade faster is going to win. And now they're down to, when I kind of stopped working on high frequency trading, we were down in the 50 microseconds from the time you'd receive information to executing a trade. Now they're down to like four or five microseconds. And to me, a microsecond is how long it takes. So four microseconds is a light can travel less than a mile in that period of time. I mean, this is really short periods of time we're talking about. Maybe that's not the best analogy. But we do think about light. That's a big part of it because the information has to travel a mile. If you've got the two computers, point A, point B, that's 5 microseconds right there you've lost just in the light moving. And so kind of think about that stuff. But it's really, really cool and really fun to say you're writing the software to be as fast as possible to compete with the software across the street. Yeah, it definitely adds to that competition side, the thrill side, because then it does. It pushes you to not just solve the problem, but to solve it better. And that was always the funnest thing that getting into programming competitions or that drove me into the commercial world of software. Because it's always about not just solving it, but doing it in a way that is better, that is more useful, that is cheaper, that is all the things that will make it more likely that somebody's going to say, yep, I want your product over that other product. At the same time, I will say for me, when I kind of you go down a rabbit hole, if you will, when you start doing these super, you're just paying so much attention to performance that all of a sudden you're doing crazy things. And I will say right now, the leaders in this space, they literally are taking their software, writing it to EPROMs, like burning them into chips, that I think they're even like partnering with Intel and stuff. They burn these into the chips. And if they want to do a new software release, they literally throw out the chips and put in new ones. You know what I mean? That's the level they're down to. And I was like, at some point, I'm like, all right, this is really cool, but I want to focus on other things, which for me, I think one of the most interesting things is how do you build big, complicated software systems that have many moving parts and interact with lots of different things, and you have teams of people and you're trying to solve this big common problem? That's kind of what I've spent my time at in the last 10 years with ThreeForge. I was going to say, yeah, that sounds like that's exactly what drove you to ThreeForge. So stepping back a little bit, was that really what you initially were looking for? Was there a specific problem you were solving or was it more where you said, hey, I see where I can go with this? And that was the genesis of ThreeForge. The genesis of ThreeForge was I managed teams and projects in various capacities, all within finance, but even within finance, you have many different fields. You have high frequency trading is one of them. You have middle office, you have back office, you have regulatory reporting, you have commercial banking, investment banking, all these different things. But I found that what was happening is if I, as I went from each vertical, the problem was being solved, the problem being solved at let's say a 90% overlap, but it was always being built from the scratch, from scratch over and over again. And to me, I felt like there's definitely a space to solve that 90% once and then let people focus on the 10% outside of it. And to me, I've called it the full stack platform. So the vision I've had, kind of the grand vision has been when I think about the evolution of computers, it's been kind of layering on top layer after layer after layer. And with each layer, we get to not forget about, we still have to learn about it, but we don't have to directly think about the layer below it. A few layers I always talk about is, you know, first off, you have the hardware layer. So if you go way back, people were literally, well, it's almost going back to the high frequency training. It's like that. They're literally, you know, physically moving the wires around to build the software. Then at some point, you start to get software driven decision making, and then you get compilers, and then you get operating systems, and then you have databases, and then you get graphical operating systems. And these are each layers. And as you add these layers, you know, especially the databases, once you have databases, you don't have to think about linear reading and writing of files so much, it kind of manages that for you. And I feel like we're still, we've been for most of my career, the last 30 years, we've had databases, and we've had everything else I've talked about. But I think there's room for another layer on top of that, which says, if you're building a business application, you shouldn't need to worry about directly opening up a connection to a database, and worry about, you know, connecting and setting up your web server and connecting and setting up, blah, blah, blah, because just setting up and managing all of these pieces ends up being very, very expensive and error prone. And so the idea was to just kind of pre package and have all the things you need together, and let you focus on the business logic. So do you do that? Like, and I'm, this is where I don't have a deep knowledge of three fourths. But so is that you do it sort of like, I guess, or like, like an operating system does it, where you just abstract stuff to say, okay, you've got a data store, or you've got a computational engine, or however, you do it so that you sort of have these, these pieces that all exist, and then it's just sort of a matter of, hey, you know, throw data at that or pull data from that. So it's a, it very much simplifies it from their that end user point of view, while you've built something. And I assume then that gets back into sort of that follow up is, are you finding that you're going back into those pieces and doing a lot of tuning and things like that to get that whole thing to run, you know, better and tighter? Well, I well with software, it in 2000, when I started the company in 2012, with three forge, I had the vision I just talked about, I didn't quite know how to get there. And I've spent a lot of time studying in this net on on what does it mean to build these systems. And I actually, believe it or not, came across something that isn't isn't that well known, but it actually comes out of the military. And what they did is they they determined that what does it mean for humans to interact with machines. And they boiled it down to a few things, observe, orient, decide, act. And of course, this is a little bit of it has a military slant to it, but it works for almost everything. And so I've designed our system around this. So observe means you want to be able to see information. So that's information being pushed to you in real time. So if you're sitting in your car, if all your windows were covered with paint, and you couldn't see outside, then you wouldn't be able to observe that'd be a real problem. So observing is just being able to specifically it's about real time, it's about information being able to come at you. Then we talk about orient. These are weird terms. I'm just using their terms. Orient means now that I've observed something, I want to be able to ask questions, additional questions about my environment around me. And so I observe, I see, you know, there is a tree coming at me, I'm in the car, now I need to orient myself. Okay, if I look to the left, can I go to the left? If I look to the right, can I go to the right? Do I have enough time to break without hitting the tree? So you're orienting yourself. Observe, orient, and then you need to make a decision as to what you're going to do. That really is more of an internal thing. But we could talk about how I work on that. And then you take your action, which is you turn the steering wheel to the left, you turn your steering wheel to the right. And it's actually called the OODA cycle, because the idea is humans just go through the cycle for our whole lives. We observe something, we orient ourselves, we make a decision, we act on it, we make an observation, we orient ourselves, we make a decision, we act. So the whole platform that we built, when I started thinking, because again, my platform was now saying, okay, we built all these layers to make computers better at being computers. Now I think we're ready for a layer that makes it easy for humans to interact with computers. And so our software is focused on these four things. Observe, orient, decide. I hate to cut them off there, but that seems to be about a perfect place for us to pause, because we're going to pick right up next episode with him talking about how he takes that OODA and applies it to these tools that they're building and how that has sort of launched him moving forward and some of the problems they're solving and how that simple process allows them to simplify, essentially creating solutions. So you can put your pencils down right now, you can rest your note taking hand, but dive right in next episode because he's going to pick up right literally where we left off. That being said, for now, you can go take a walk, shake it out off a little bit, allow yourself to digest what you've just heard. And of course, just embrace your day. Go out there and have yourself though a great day, a great week, and we will talk to you next time. Thank you for listening to Building Better Developers, the Develop-a-Noor Podcast. You can subscribe on Apple Podcasts, Stitcher, Amazon, anywhere that you can find podcasts. We are there. And remember, just a little bit of effort every day ends up adding into great momentum and great success. Please check out school.develop-a-noor.com. That is where we are starting to pour a lot of our content. We've taken the lessons, the things that we've learned, all of the things that make you a better developer, and we're putting it there. We have a range of courses from free short courses up to full paid boot camps. All of these include a number of things to help you get better, including templates, quick references, and other things that make us all better developers.