Data Integration and Real-time Problem Solving

Welcome to Building Better Developers, the Developer podcast where we work on getting better step by step professionally and personally. Let's get started. Well, hello and welcome back. We are continuing a season of interviews and we're continuing and wrapping up an interview with Robert Cook, a very experienced developer who's written a lot of code, spent a lot of time working on it. And we started diving into some very deep technical kinds of things, approaches to problem solving. Last episode, we paused almost mid-breath and we're going to step right back into that where he is talking about the OODA, about that approach and how they use that within their systems to solve some problems. So let's dive right back in with our conversation with Robert. Observe means the ability to push information real time to a user, which is important for trading. So that was something we had to do anyways. Orient means being able to ask questions. So we have integrations with lots of different databases. That's what I was talking about before, because usually when we talk about orienting ourselves, you're searching for more data. And so you're querying a database. So we have that layer. And then when you're making a decision, we have some workflows around that where you can interact with other users on your team. And then you take an action, which could be as simple as clicking a button or filling out a form and submitting or something like that. So that's it. So it's been understanding that, thinking about it generically, and then all the components we build have been driven towards those four things. Now, do you do that as part of the approach then that people that build? Because you have a platform and then people are building on your platform or using it to build their solutions. Is it something where you push that to them and say, this is a... I mean, I guess that's part of the nature of it. This is the way you should approach this. Therefore, these are how the... That's maybe not necessarily language, but close to the language that they're going to be used to about how we interact with the system and actually build it and direct it. Yes. So I would say we would do both ways. I mean, a lot of times our customers will just take the platform and they're going to decide how they want to use it. They maybe will get some training on how to do this stuff. And other times they'll come to us and we'll help build it out. I'm not sure if that answered your question. But then on top of that, a lot of it is about us just making it easy to access multiple systems. Because by the way, this is a problem that isn't unique to finance, but it's almost been crippling for finance, which... And by the way, I do look at finance as kind of a canary in the coal mine, because finance has been using computers just about as long as any industry. And so finance is seeing all the issues that other industries are starting to now come in to arrive at. And I think one of the biggest ones is, as a company, when you've gone through many mergers and acquisitions and you have had many technologists with many opinions on how to build systems and design and store data, you end up with this very complicated data set that's distributed all over the place in all these different ways. And it becomes very difficult to actually, quote unquote, orient yourself, ask a question about what's going on and get a cohesive answer back. So a lot of what our platform is, is that orienting part of it, which is we'll connect to many databases, we'll help you map those systems into a more normalized set. So then you can ask a question and that answer may involve many different data sets coming from different teams. And so we've done some stuff around, we have some proprietary language around helping you ask questions across two databases or things like that. But a lot of times it's just trying to package up and make it easier to access multiple systems at once. And by the way, I think that this gives large, I think large organizations are sitting on unbelievable amounts of data. It's actually a huge asset to them, but they're struggling to make value of it because of the problem I'm talking about. Because how do you join data from two different systems? And I know this starts to sound a little bit abstract, but it's actually a big thing we focus on, which is you have data sitting in different systems that you want to be able to ask questions across that and get that back. Maybe it's to do a recon between two systems. Maybe it's just to see a particular account. What are they doing across my entire organization? Those sorts of things are not often easy for large organizations to get answers to. Yeah, I think you nailed it with it. It can be crippling. And I've seen, it's funny, yeah, it's large organizations probably hit it because they're larger and they've had more time. But I've even seen a lot of smaller organizations and younger organizations that they, same thing, you've got contractors or a big turnover or something like that. And then everybody that comes in has got a little bit different approach. And then you've got some shifts in technologies. And the next thing you know, you may have a very small team, but you've got twice as many technologies and approaches than you have team members. And trying to go back and figure out what that does, sometimes you can't even, you can ask the question. There's actually, I've been too many companies. There isn't anybody that can say, this is the comprehensive thing. This is what we do and how we do it. I could give many examples and I feel I'll just choose to relate to kind of the retail. Let's just say you use the same company to provide your internet plus your phone plus let's say you also use them for, I don't know, managing your travel points or something like that. I don't know. And it happens to be the same company, coincidentally. I've seen many times just on a personal level where I can't even say how much money have I spent in this company and they can tell me because they're saying, well, that department manages that set of billing. This department manages this set of billing. And the listeners might say, well, that's not a big deal. Maybe that company is not that organized or something like that. But this is a real, real problem because what's often happening is companies are buying up different technologies or different products and then trying to package that together and they can't and they can't even get it right for the end consumer. So now imagine internally when they're trying to actually make real business intelligence type decisions off of this data. And there I think there was this concept for a while, which was, well, let's just take all the data from everything and shove it into one spot and create a data lake or data warehouse. I found inevitably that if that actually does succeed, it succeeds only to ask the question that they sought out to answer. So for example, they said, I want to have a data warehouse because we don't even know what an individual customer is spending on average. And we just want to know that they'll build this whole data warehouse and they'll answer that one question. Then you're like, okay, I want to ask another question. Well, now we got to go change the data warehouse and get it and reconfigure all this data. So the whole so it's a challenging problem. It's one I think I could spend my whole life on, like fine tuning and thinking about it. But it is it's not unique to finance, but I do believe finance is probably their first. Yeah. And I think that goes back to like the business intelligence and data warehousing type problem is that was the first people that I saw that were using that. And it is it's funny. It's it's it is very hard to get across to somebody that you can go do that, but you have to know all the questions you're going to ask beforehand. Otherwise, you lose it because it just goes back to actually goes back to sort of the idea of math and some of the other things to understand that when you start rolling values up, details disappear. You know, you can't you can't you can't like if you sum all your numbers, you can't just magically blow that thing back out to whatever originally got you there. And so it is it's a it's a very common problem. And it's it's very hard to get past that. I need to know these three answers right now. And we know we have it in our data somewhere. So let's go solve that problem. We'll figure about other questions later. And then you find out that, oh, no, you the way you solve that problem will not allow you to solve these other answer these other questions. So new data warehouse or change stuff up or whatever you need to do with that. Right. Yep. And so it is it is true. This comes back to almost the beginning of the conversation, which, yeah, you could you can almost prove mathematically that, yeah, as soon as you start to reduce the granularity of the data, so you can roll this stuff up. Now it's hard to basically ask different types of questions, depending on how you've rolled that out. The best, you know, my the the the way we have helped our customers solve it is through this concept we call leave the data where it is. And so what happens a lot of times our customers have spent a huge amount of money to build a data warehouse, that data warehouse can answer that one question. So we say, OK, fine, that's great. You've done that. Leave that data warehouse as it is. We will tap into it as one of the data sources, maybe even a starting point. So now you can say, OK, I want to know how much money I spent last quarter. And then you get the answer back from that data warehouse, because that's what that data warehouse was designed to answer. Now you've got that answer. Now, the problem is, again, you say, how much did I spend quarter over quarter? And then you see there's this one quarter where you spent three times as much as every other quarter. Of course, now you want to find out what happened in that quarter. And that's where we come in, because now what we'll say is, OK, we'll take that information and on the fly, build queries and design queries to help you go and access the underlying systems that were used to generate that data. And obviously, you have to go into that with a bit of humility. And you know, you have to be you have to be respectful of the underlying databases. And understand how they're independently designed or individually designed. But at least you can start to tackle that problem in a more flexible, reusable way. Another way I look at it is different databases were designed and chosen for their specific. Type of data they were holding. So, you know, you might have a database which is for storing unstructured data like phone calls or something like that. You might have another database that is designed for holding time series data. So it's columnar and very organized. So you can see price changes and another one that's just a standard, you know, normalized database, relational database, where, you know, you have lots of rows or, I'm sorry, lots of columns and something like that. And these databases were chosen because of the type of data they're holding. So our opinion is keep those databases and leverage those databases. And you may need to ask questions to many databases to get the answer. Like you might need to go to a relational database to find out some information about the client. And then you might also need to go to a columnar database or an unstructured database simultaneously to find out the phone call that they have or this or that. But that's OK. You go to those three systems, you leverage that existing thing and you bring back and you provide the answer. So are you set up so that is three, four, then something sort of just is it really just sort of sits on top of their current infrastructure, whatever that happens to be? Exactly. So do you absorb any of it? You sort of leave there. Does there's sort of I guess and I guess I'm thinking from a as you evolve. So you come in and you say, all right, we're going to help you out. We get everything set up. Is there does it always leave everything in that infrastructure or is three, four set up so that over time, they can actually sort of absorb it into effectively, I guess, absorb it in the three fours so they could actually start not having to use those. So so far, we've talked about one of the very important parts of three fours, which is the orienting or being able to ask questions of your data. Where the I would say we begin begin absorbing, if I have to use your term, would be when you start getting into the real time stuff, because that's a big part of it, too, which is. There is a real time aspect to what's happening, and by real time, I mean very real time, like milliseconds or maybe at worst case seconds, which is a user or or there is some sort of exception that takes place. A connection goes down. A customer is pissed off. A a server has failed. There's some sort of anomaly. Could be as simple as someone's trying to over execute based on how much quantity they have some whatever. But that raises an event and that event needs to be pushed to the operator, to the person using your dashboard real time, like right away. And then someone takes ownership of it and then and then manages it. Maybe does. And that's where they want to now orient themselves around that. That information that's being streamed in, we we advocate that that data gets streamed into our platform. So I guess you could say that gets absorbed and that. But that's necessary so that we can push it to the user in real time. So the idea is if it's information that you need to push to the eyeballs of an operator or send an alert to their phone or something like that, whatever it is, that information. Yes, we say push that into our platform in real time. That thing gets stored inside our platform. But the idea is that's more I don't know if I use the term ephemeral, but the idea is it doesn't it's it's it that information doesn't last for years on end. That information comes in. People operate on it. Maybe you can then put it into permanent storage a week later, something like that, if it's interesting. But the idea is that, you know, it's really the real time thing. So that's and by the way, there's there's only three real parts to the product in my mind. And I'm not trying to sell it short here, but it's that ability to connect to any number of data sources and ask questions, which is what we talked about originally, the ability to stream information in real time and then the ability for operators or users to be able to take an action and for that to send data out. So an example could be, you know, an alert comes in, the operator sees it. He says, I need to find something out about their orders. The orders come up and he realizes this order should just be canceled. The customer didn't even want it. Right. Could cancel that sends a cancel message out to the infrastructure that presumably would actually cancel that order. In their database, you know what I mean? Not not in our system cancels it in their database. So really, all of this is about providing a layer that sits between the infrastructure of large companies or medium sized companies in the in the operators slash users. Those are the three things that really come down to just to repeat one more time, asking questions about your data, getting real time streaming and being able to have workflows or submit data. So your integrations are actually they're not like they're not one way integrations and read only they are they are by and maybe not the same stream, but it's the kind of stuff where we're going to pull your data so we can use it so you can view it. But then when you act on it, we're going to have some way to push back to whatever the system is. Exactly. Exactly. And by the way, I think that, you know, what I look forward to is I don't think we have many competitors really in our space. But I think that this is it's just the time savings and the ability to just integrate because all this time and money is spent building these infrastructure, building these, quote unquote, back end systems. Right. And I think that's one thing that for better or worse, I think that developers have done pretty well as we've at least in finance. You know, you've got your back end systems and then you've got your front end system. The back end system is where the intellectual property system and the front end system is really just how we interact. So we're saying, look, we will be the front end system to integrate with all your many back end systems and then give you that consolidated view. And I will say, you know, what I think where we help and this is kind of getting probably a little closer to the theme of the podcast is. I think that developers have a tendency to put way too much business logic in front ends. You know what I mean? Like, I still see it today, even with some of the most advanced things where on your on in the web browser itself, in the JavaScript are decisions, almost business decisions, validations and things like that. Now, you can do it there and also in the back end. That's fine. But to only have it living in the web server or even worse, in the web browser. That's a real problem. And so we kind of force this this bifurcation. The front end is for reviewing data and manipulating data, things like that. But any sort of validation or anything like that should live in the back end. Yeah, and that is that it's sort of that I think we we go back and forth a little bit with that and you same way pricing over the years is you get sort of a fatter back end, a fatter front end. And usually it's when it's that thinner front end that people realize it's always that that benefit of, hey, now we don't have to change stuff in 10 different places, we don't have to track in all these different places. There's like a lot of there are a lot of worry rooms there are a lot of worry related things and problems that get solved by saying we're going to have this central place where we validate and we check and we do that business logic and then we don't have to worry about versioning and we don't have to worry about it's it's pushed out and then it's in sync and all that kind of stuff. We can we can make sure that it's there. It's solid and it's tight, which is the other thing. It goes back to the performance thing. If you've got if you're doing this in one place and actually that goes back to even the mention to Apple. I mean, that was one of that was what Apple's drawing card was for years. They said, well, we we own everything. We we build the chips, we build the screens, we build it is we are all of those pieces are ours. And so we can make them all work and make them work together. And you had the flip side at for many years and I guess still to some extent you had the Windows machines that that were, you know, they were put together by all kinds of different things. And so you'd run into driver issues and things like that. And, yeah, you could get a really screaming fast machine, but there was a lot more work involved because you didn't essentially control enough of the enough of the inputs. And you get the same thing with the front end versus back end. If you're on the front end, then you're like, well, this may work for that person, but it may not work over here versus knowing that, hey, this is going to take split second to get done. And then you can move on to whatever your next calculation is. Right, right. Yeah. I think it is it is a challenge, but I think one of the risks you run with having heavyweight front ends is then people always want to upgrade the front end user experience and change the front end user experience and enhance the front end user experience. Front end developers, just like any other developer, they come and go. They leave organizations. And what I've seen almost universally is that a system, a front end system is built and if and if allowed to, validations and business logic lives in that front end. And when those developers leave, people are now terrified to change that front end, because if they break that validation, it can have really meaningful and bad implications on the back end. You know what I mean? Like if the back end isn't designed to handle null, whatever reason, and the back end isn't checking for null and only the front end is checking for null, I'm making a stupid example here. But and then the operator answers it and then the user enters in null. You know, suddenly no one can catch their flight. You know what I mean? Something like that. And so it is important, in my opinion, that the back end be somewhat self-sufficient or isolated in that it can it can. It has its own immune system and understands some things. And then the front end sits on top of that. And the front end wants to duplicate those validations or duplicate those checks. Go for it. That's that's that's fine. If you want to improve the user experience or something along those lines. But it shouldn't be the only place those things live. And I guess that's my point with kind of how we've forced the architecture or at least persuaded the architecture is that the back ends should really be doing any sort of validations, checks. They're receiving messages. Whether they happen to be coming from our front end or any other front end. All of that, at least, lives inside the server itself. Yeah, I like I like that analogy of it having its own immune system, because that really is what it is. You're not you're back in now. It isn't trusting everybody to give you nice and rosy data. It says you throw whatever crap you want at me. I'm going to make sure that I'm protected against that. And it's yeah. Otherwise, you get into that. You know, is this synced up with that? Is it does it send those? Does it allow knows is it not does it? And and that it seems like such a simple little problem. But the one that will blow people up all the time is dates. I don't know how many times I've seen stuff where front end changes the date format it sends back and suddenly the back end has no idea what it's what it's doing. And that's like you would think that would be very simple because it's just a year, a month and a day. And yet that thing can can totally mess up systems. I you know, I'm on a losing argument here, but I think the whole world should just switch to Unix timestamp epoch, that big long number. And then we wouldn't have this problem. We wouldn't have this problem. Yeah, I see a lot of systems that have done that until the point that the user says, what the heck is that? And you look at it like, oh, yeah, that was noon yesterday. How do you know that? Unix time, time, I think. Put your vote in today. Start a movement. Well, I want to I want to thank you for your time. This has been as I expected, because, yeah, when you've you've spent this many time, it's many hours developing and thinking through these problems. I know it's going to be a great conversation, but also it's a lot of I think we touch on a lot of things that just they are universal developer type problems. It's the things you're going to run into. It doesn't matter what your industry is. Honestly, it doesn't even matter how long you've been doing it. There are going to be things where you're going to you're always going to have like, you know, hey, I can take a shortcut and knock this out. But then I may end up having to come back and pay for that later and go fix it or rewrite it or, you know, all the different ways, all the different problems and the different ways you can solve them. So I really appreciate your time and just a great conversation for anybody that's that is like, man, this stuff all sounds great or Robert's an awesome dude. Love to hear more and more from him. What's the best way for people to reach out and contact you? Threeforge.com is our website. LinkedIn is where we have our social media. So if you search for LinkedIn, you know, Threeforge, follow us there. We're always doing updates of everything going on. You can always reach out to us at info at threeforge.com. And yeah, we'd love to hear any feedback. Yeah. Cool. I will make sure I have links to all of that. And anybody that's listening can reach out and check you guys out. If it's something that worked, maybe it's a good fit for their company. They say, hey, we got a ton of systems and we need to figure out how to talk to all of them at once. Then it may be a perfect fit for you guys and them. So I would thank you again for your time and hope you have a good rest of your day. Yeah, you too. Thanks, Rob. Thank you. And that will wrap it up. I hope you were entertained and enjoyed the conversation. There was a lot that I got out of it. I think there's a lot that you can get out of this as well. Even though you may not be solving the exact same kinds of problems he is, there's a lot there, a lot of basic problem solving approach techniques and then a couple of things there about, you know, how do you make something better? How do you work on performance tuning? How do you get into a situation where you need to do it just a little bit faster, but make sure that you maintain quality, things of that nature. As always, there are going to be links in the show notes. If you'd like to reach out to him, very fascinating conversation. And I would assume that you can do the same if you want to reach out and talk to him, get to know a little bit more about what he does, about his organization, about his company. That being said, we'll wrap it up and give you the rest of your day to go out there and solve all the problems on your to do list. So go out there and have yourself a great day, a great week. And we will talk to you next. Thank you for listening to Building Better Developers, the Develop a Newer podcast. You can subscribe on Apple Podcasts, Stitcher, Amazon, anywhere that you can find podcasts. We are there. And remember, just a little bit of effort every day ends up adding into great momentum and great success. Please check out school.developaneur.com That is where we are starting to pour a lot of our content. We've taken the lessons, the things that we've learned, all of the things that make you a better developer, and we're putting it there. We have a range of courses from free short courses up to full paid boot camps. All of these include a number of things to help you get better, including templates, quick references and other things that make us all better developers.