Cutting Through the AI Hype: A Practical Guide to Building Your AI Strategy (LCPS01E08)
Listen on all Podcast Platforms, like Spotify, Overcast, Apple Podcasts, iHeartRadio, Amazon Music.
Episode Summary
How do you respond when your boss or board asks “What's our AI strategy?” In this episode, Lena Reinhard talks with Daniel Paulus, VP of Engineering at Checkly, about developing a pragmatic approach to AI initiatives that creates real value while avoiding the pitfalls of hype-driven development.
Daniel shares his experience leading AI initiatives and provides actionable insights for engineering leaders, including:
How to identify genuine problems AI can solve vs. solutions looking for problems
Ways to validate and experiment with AI solutions while containing risks and costs
Practical considerations around compliance, privacy, and security
Strategies for managing team excitement and maintaining focus on core deliverables
Tips for evaluating AI technologies and building initial proofs of concept
Whether you're facing pressure to "do something with AI" or genuinely exploring how AI could benefit your organization, this episode provides a practical framework for moving forward thoughtfully rather than just chasing the latest trend. Lena and Daniel discuss how to balance innovation with pragmatism, and share specific approaches for validating AI initiatives before making major investments.
Resources
Research & Studies:
Frameworks & Tools:
Technical Resources:
GitHub Projects:
Additional Resources:
Chapters
00:00 Navigating the AI Landscape
02:48 Understanding AI Strategy
06:07 Identifying Real Use Cases for AI
08:49 Building an AI Strategy
12:05 Feasibility and Risk Assessment
14:56 Finding AI Expertise
18:08 Developing Proof of Concepts
21:04 Budgeting for AI Initiatives
23:55 Compliance and Ethical Considerations
27:11 Managing AI Risks
29:59 The Future of AI in Business
***
We want to hear from you! Email us with feedback, questions, or topic ideas; I can't promise we'll always respond, but I can promise we read every email! At pod@lenareinhard.com.
Full Episode Transcript
Intro
Lena: What's your AI strategy or what are we doing about AI? Or you even maybe have a product roadmap that just says something with AI. I know that those are situations that a lot of you are in right now and questions that many of you are getting either from your boss, your board, or product partners, and then what do you do?
And how do you cut through the hype around AI to figure out if there's something that's actually useful for your business as well as your users? I've got you. This is leadership confidential with Lena Reinhard. We'll talk on more than two hard things in technology, finding community and becoming the engineering leader you can be today, how to come up with an AI strategy.
I've spoken a lot over the last decade about the disastrous consequences of the hypes around AI, for example, and I've worked with many leaders over the last years to help them identify if there's substance to it for their business and how to assess and use it in ways that actually make sense. And that's why I'm really excited about my guest today.
I'm speaking with Daniel Paulus, VP of engineering at Checkly. Daniel has extensive experience in this field through his studies and education, as well as his work. And I also appreciate him as a very pragmatic leader, as well as engineer. Together, we'll give you ways to explore this technology for your teams and tackle the questions that you're getting, as well as the pressure from your boss, without just feeding into the hype or creating a ton of waste and demotivating your teams along the way.
So here's real talk on coming up with an AI strategy with Daniel Paulus, VP engineering at Checkly
Daniel: hello. So I'm Daniel, currently work as a VP of engineering in a 50 people series B startup in Berlin. We are a remote company, but I'm in Berlin and we build observability tooling for developers. How long have I been in the business? That's actually a good question. I finished my studies in 2010. So that's like what, 14 years now?
I did study, during my university already, I was a machine learning, enthusiast. So I, you know, studied a lot about how multi layer perceptrons work, which was hot at the time. I mean, kind of still are, but,
and, I was always fascinated by the topic. You know, I built my own, back propagation and gradient descent algorithms in Java to understand how it really works.
And, later I took online courses, like this online learning company, Udacity, which included building multiple projects with deep learning. So I used PyTorch and built my own models from scratch to get some, at least some first hand experience of how all of this stuff actually works. Cause I always like to know the technical details of how, Machine learning and artificial intelligence actually work under the hood.
and then I also spent like a year in a startup from Silicon Valley, building AI and testing. So I also have some product experience building AI solutions. And, yeah, no, today I'm here, building our AI strategy.
Lena: Perfect. the reason why I was so excited to speak with you is because you've done this for a while in different shapes. like you just outlined. and. the Topic that, I wanted to cover with you is that a lot of people get asked, yeah, what are we doing about AI? What are you doing about AI?
Either questions come from their bosses or investors or just team members even. and we're here to answer it. I guess in under 60 minutes. exactly.so I do want to start with the first question that we got from someone who wrote in because I posted a couple days ago that we're doing this and some people shared questions on LinkedIn.
and the first question they asked was, What do we mean by AI strategy? Like, isn't there a strategy overload? Like a digital strategy or product strategy and engineering strategy and so on. So why an AI strategy?
Daniel: Yeah, that's a very interesting question. And I agree, there's a lot of strategies, but if you look at so what does it actually mean to Create a strategy for AI. I would say it's simply a plan, how to integrate AI into the organization. And then you somehow need to take a look at how do I make it align with business needs?
So this can mean depending on what you're trying to do, this can mean, how do we integrate it into our internal processes? So how do we use it in engineering to maybe increase engineering efficiency or make engineers more productive. How can we use AI and other departments to make them more productive?
And then of course, if you're building a product where this is relevant, it can also mean how do we incorporate AI into the product in a way that makes sense. So these are the things you should be thinking about. And I see many companies do that. So I talked to a couple of engineering leaders and leaders from other companies and They think about this also, right? Because especially when you think about AI use internally, often the question that is asked is about compliance and security. And what's going to happen if you don't have an AI strategy is people are going to use it anyway, and you have zero control over it. So this is why you should think about an AI strategy, even if you're not planning to use it or build something yourself.
Lena: and I think that's a good prompt, to talk a little bit about the ways of thinking that go into this, like from your side and my side, because, I find it quite valuable to understand a little bit, you know, what kind of either principles or values or whatnot are people approaching this from, especially because there's so much buzz around it currently.
So I'll just start with my thinking a little bit, and then I'd love to hear from, you about yours. my thinking is that I, like, I am quite skeptical, when it comes to especially how much buzz there has been about AI over the last couple of years compared to what's felt like often not a lot of substance.
I think especially a lot of VC company, venture capital money. It's been going into the AI space for over, like for the last couple of years. there is a lot of, again, and where there's a lot of money, there's usually also a lot of, incentives to get quick profits, and to build things that are well in the end, satisfying investors.
and that doesn't always mean that those things are actually solving user problems. and to me, like having been in tech, for similarly long time as you, I've seen that kind of. horse before, just under a different name. at the same time, I do think it's interesting to look at like, what are the real use cases of it?
What is the substance? Like, how can it be useful for the business that I'm in? but I also think the way that it's currently often marketed in terms of, Hey, it's going to either, you know, reduce your workforce to 10 percent or it's going to solve all your business problems at once, and it's going to solve your, obviously all your organizational problems.
In any case, because it's going to make humans redundant, as we all know, humans are the cause of the issue of many organizations. Like, I think, just think there's a lot that's being oversold, as more than it is, but I do think, especially for engineering leaders, there is also a lot of pressure because of the business types, because your executives and your investors, they see the headlines, they say, Oh, everyone is doing AI.
Why aren't we doing it? and that's why I wanted to have this conversation to help people kind of start Okay. What's Again, what's hype, what's substance and how can I assess that for my business? So that's kind of where I'm coming from.
Daniel: Yeah, that's a very interesting point on a very interesting question and discussion that I would love to have. So the question is, AI just a hype? And I think, no, it's not. So if you want just a hype, invite some blockchain people. I'm kidding, blockchain people, you're cool. But,is there a hype around AI that maybe influences how people perceive it?
And yes, of course, right. There is a lot of hype around it and not everything is justified. and not every problem that is advertised right now could be solved by AI will be solved. I think,so what I do, I read a very interesting study before, you know, I. Approached my strategy. It was, published in August this year, and it's called five leading root causes of the failure of AI projects, and they identified all of them and included recommendations.
how to mitigate. And I think, so there are like five problems. You should totally read that study. It's very interesting. But I think the more interesting ones is one of them is. Can AI actually solve the problem that you are trying to solve, right? So this is one that means, maybe the problem is just too big.
So if I say here's an automated software engineering team that, so LLMs will automatically build entire architecture and software, then no, that's not possible right now and it's, maybe it will be someday, but it's definitely not today. So, you know, if you try to. Solve a problem that's way too big for what's available right now, you're just going to fail.
So you'll spend a year or two, tons of resources, and in the end, you're not going to be succeeding. Also often it's miscommunicated what problem needs to be solved by AI. So often you have use cases where people build stuff because it's cool, but is it actually a problem that anybody needs to be solved?
Right. So, I'm very opinionated about many things as you might,might see during this talk, but, like a lot of stuff LLMs solving all kinds of testing problems, but there's also And I usually like to say, is creating automated testing really such a problem? I mean, it takes me maybe half an hour to write an automated test for something.
And what benefit is it really? Like if I have an AI doing this, of course you can say, yes, there's the maintenance, but yeah, still, I don't know. I don't think it's justified to invest so much time into this when the problem is actually not that big, but the solution using AI is pretty difficult.
Lena: and it's expensive compared to other solutions. Yeah.
Daniel: Yes. This is especially the case when, so I think one, one thing that is very cool about LLMs, which is different than, we've had with machine learning algorithms before that is that you actually don't need to do any model training to. Change the model behavior. So just the capability of just putting stuff into prompts is really powerful because you can teach the algorithm or you can teach the network new things simply by putting strings into a prompt, which is extremely, amazing from my point of view, right?
Before that, you would have to do very expensive model training every time. Like we used, Deep learning networks to do like image recognition on UI elements. And then we always had to retrain the model. And that was very expensive. He needed like expensive data scientists, training data labelers.
It was a huge undertaking. Whereas with LLMs, you know, If you have your company notion or something, and you want it to be able to answer questions based on it, all you do is you hook up a search engine behind it and you inject the correct document into the prompt. And suddenly the LLM is able to answer questions on your own knowledge base, which is amazing.
And it's actually quite feasible, even if you don't have a huge team of machine learning engineers.
Lena: Um, and the other I did pull up that study as well, but also I'll link it in the show notes too,um, but I think the other, reasons for AI project failure was that, yeah, organizations focus more on the latest technology. I think you just alluded to that as well. or just don't have the infrastructure to really manage and deploy.
AI models once they're completed. and I think the last point was that, yeah, technologies just apply to problems that can't be solved with AI right now. so, then say, you know, my boss, is telling me, hey, I you know, I need to look at AI for my area, my team.
what's like, where do I start?
Daniel: I mean, this literally happened to me. I guess it's also a reason why we're talking. So somebody came and asked me exactly this question. So, Hey, what's Checkly doing with AI? And the answer was, well, good question.
TBD, I guess coming from the, you know, five failure reasons, and what we can learn from it is. Also how I would approach, or I would recommend to approach setting up AI strategy, right? So first I started with a customer problem. So what is something that you want to solve, right? where are customer problems?
And as those can be internal and external customers, depending on what you're trying to do. But starting and identifying like a real valuable customer problem, and then The next step, of course, you know, in product management, we would say feasibility risk. So this is the other point. So you have to understand what is actually possible with the technology today.
So if you have a problem, then you can make sure it's actually feasible to build something with a decent amount of time. And so I, myself, I sat down and I started doing what I always do. I opened open source projects because luckily they're available everywhere. Like Microsoft, for example, has auto GPT, and then there's a tons of others.
And then you can take a look at the source code and you can understand, okay, what is it actually that they're building when they built these tools? Where is the complexity? What. Do I need to know? next I, I downloaded an O'Reilly book about LLMs and I started reading the 500 pages on a flight to San Francisco to really understand, okay, what's, what are the capabilities of these, transformer based LLM networks.
And once you've done that, you sit down and you just start to build something, right? You grab one of these frameworks or one of these LLMs. Use some TypeScript and you just put in some data and see what happens just to get an understanding of, can I actually solve this problem? Um, and once you've come to this conclusion and you find a problem, that's interesting, then yeah, I mean, engage other people in your company.
Talk about them. Here's what I'm trying to do. Here's what I'm trying to solve and go from there. that's what I did. Yeah.
Lena: and how do I go about determining, like basically if AI is the appropriate solution to my problem versus something else?
Daniel: That's tough if you don't understand it yourself and you can't come to the conclusion yourself, it's kind of, I can see how it's difficult, right? Like if you get an external consultants in, you would usually hire an AI expert making money with building AI systems. So the question is, who can you trust to really tell you if the problem beats AI, even right.
Or, I mean, sometimes you have a problem where like, I don't know, what do we have, like finding spikes in metric. And time series were like, yeah, maybe you don't even need machine learning for this. You can just use some conventional algorithm, which is going to be much cheaper, achieve the same result.
And it's going to be much easier to use.
Lena: I mean, there's still like, you could run the problem by the engineers on your team.
And, you know, maybe or maybe you have an architect or a principal engineer or someone who's quite senior, say, hey, you know, here's the problem I'm trying to solve. How would you solve this? Or what, what could methods look like?
And if the only thing they're telling you is AI, then you might need to rethink that higher. But, at least, you know, are there, like, It does seem worth looking at at least. Are there other ways of solving this problem? and if so, like, what could they look like? or, you know, even if you want to hire an external person, well, maybe don't just hire An AI person, specifically an experienced engineer who can at least sketch this out for a few. you.
Daniel: Yeah, that's a good idea. I think that's a very good idea, especially asking engineers on your team, because usually you have a spectrum of people. So some of them might be extremely enthusiastic and excited, and then some maybe less so. And if you get a good,good group of people together and ask them this question, I can see how this would work.
Yeah. I also got a freelancers for my project because, for the feasibility study that we're doing. So we're building a POC it's an open source project that we're trying to kickstart. And I got, freelancers to help me. Right. So once we had Gotten this idea that you could solve this problem probably reasonably well.
I just got some budget and some freelancers to work on it. And one of them is actually an AI expert. Yeah.
Lena: that's actually something I wanted to ask you as well, because like a lot of companies don't have AI experts internally and like yours is clearly lucky because you've done this for a while, but,where can someone, you know, who's a senior leader and they just don't have AI engineers on their team, Can they find someone who has that kind of expertise and ideally also at least a little bit that if they're decent or just trying to sell something that doesn't exist
Daniel: Yeah. That's interesting. I mean, for getting some insights. so even in Berlin, there's a pretty active AI hacker scene or any hacker scene right now that, they organize hackathons and talks. So you can. find them pretty easily on LinkedIn or,
uh, I can also maybe share some share some things later.
Yeah.
right. So I talked to a few of these people. and then of course, if you go to San Francisco, like the U S right now, and you attend probably any meetup, you'll find an endless supply of, AI founders, AI experts that you can talk to at least to, Ask him a couple of questions and then find someone.
Yeah. If you want to hire someone, I guess it's probably going to be super hard because any, anyone with an AI in their title probably wants like 300, 000 plus right now.
Lena: Per day. yeah, that's fair. but yeah, at least as a starting point, that's a really good idea. so we've got to start with the problem. Honestly, I think a lot of steps we'll go through are probably useful for any kind of exploration of new technology. but, scoped out your problem.
You'd realized how you really want to like. Try and, like solve this with AI. You're not building a proof of concept. what's the next step from there?
Daniel: So the next step is, built to learn is how I put it, right? So that's what we are doing right now. we had this idea or we looked at what could you do with LLM specifically? And one thing that they're really good at is summarizing information in a way that makes it easier to digest. So we were thinking just some interesting scenarios like summarizing recent GitHub deploys or GitHub releases,summarizing alerts coming in from Observability tooling like Grafana or check the, in our case, and then.
You could have a meaningful feed of alerts and, releases, and then later maybe have an agent that could tell you, Hey, here's an alert and here are the related logs and here's what was deployed that might've caused the alert. Right. And then the idea is that you would have this information available during an incident response, maybe way faster than you could do it yourself.
So that's the idea. And that's what we're building right now. And. Yeah, minimal scope, start building it. that's what we're doing right now. And then see if it's, you know, if anyone cares about it.
Lena: and how did you budget for investing in this? cause there's, you know, someone's time, there's probably some technology cost involved. and especially if you're doing essentially proof of concept or you're experimenting a bit, I mean, that's one way already to contain the risk a little. but again, what was your approach to budgeting for this?
Daniel: Yeah. we thought about the, so the risk, of course, I mean, budget in terms of how much money you want to spend is one thing, but then also the question is in terms of engineering time, which is probably more expensive, right? And also there's opportunity costs. So we're not an AI company. We build monitoring and anything we do has an opportunity cost.
Because we don't build stuff that our customers actually want. So, and then, so to mitigate opportunity costs, this is, that's why we created an open source repository and got freelancers. Right. So the idea was to validate the somehow without too much opportunity cost.
Lena: And also distraction for the teams, like it's kind of subset of opportunity costs. It's like a bit of a different,part of the issues that can happen, it's like disrupting the road map and regular delivery and whatnot.
Daniel: Yes, because everybody gets excited if you built an AI project.
Lena: How many people do you have on your engineering teams to actually want to work on this now?
Daniel: I haven't asked, but probably all
Lena: That's probably better. Just don't ask the question. That's a good way to solve that.
Daniel: And you have to be pretty careful also, if you're the VP of engineering, right? Hey, do you want to work on this open source project?could be, maybe received very wrong. And then suddenly people do nothing else all day. Yeah. So you.
Lena: Yep, yep. Okay. it's like basically keeping it out of the teams for now.and again, having external people do it.
Daniel: I think it depends on how confident, like with most things, it depends on how confident and how you are in the business case and how big it really is for us. We are not super confident. All right. So we have this idea. It seems like it's super valuable, but as often, you know, it could turn out that nobody really needs this.
And then you know, Lose a lot of money by building something that actually doesn't solve a customer problem. I guess that's just normal product development also.but yeah, right. I mean, it's of course different if you already have a very strong business case, then you might think about staffing a team for this.
But yeah,
Lena: I know you mentioned, you know, you kind of, you know, Read the O'Reilly book. and you spoke to some people who've worked in this area. Like, what else have you done to, hear from other experts in the field, and from people who've done this for a bit longer than you have in terms of just actively working with technology right now?
Daniel: Let me think, what have I done? So I'm in a. Discord channel where, the local Berlin developer community is, a little active, that's super helpful. I have a few newsletters that I'm reading cause that's something that's pretty helpful. I think that's it. I mean, I talked to everyone I know about it, which is another reason why we are talking right now.
Right. Yeah. it's like, what are the questions you ask people who, you know, like on me, unlike me, have like more substantial expertise in the field? . Well, so, it depends who I'm talking to. So I talked to a head of an head of AI at Microsoft and there I was curious, could I, and this is a question that comes up also, it's like compliance, right? So where do you host your models? And I think Microsoft is interesting because we have all of our source code on GitHub anyway, and having a completely like, you know, like, GitHub, Microsoft ecosystem with model hosting seems interesting to me.
right. So we haven't made a final decision on how we do this yet, but especially for sending source code, I feel this is quite interesting because you trust them with your source code anyway. So what difference does it make?and also Microsoft, you know, they do a lot for security and, Like.
Huge companies use them. I think even Citibank was at GitHub universe saying that they use Microsoft for this. So I know that Citibank has a very strict compliance process.that is very interesting. At least, we're going to look at AWS too. Of course, they're also a very trusted partner for this.
Lena: so that's probably what, what I would ask these people. Right. So. Just understanding, can I get free credits for a POC? you know, what data privacy can you guarantee? And then, so those people, that's what I would ask them. due diligence kind of questions.
Daniel: Pretty much. Yeah. And then if you have AI founders or AI hackers, I mean, then you would, or I usually would discuss, is this something that's, that you think is doable with LLMs, right?
to help me understand the feasibility risk. you know, if you were to build this, how would you approach it? Have you, do you have any tips or anything that you do usually, or, you know, these kinds of questions around. feasibility,
who have I forgotten? And then if you talk to business leaders, I mean, of course, a very interesting question, we, at GitHub universe, the CEO of Netlify gave a discussion around, it was super interesting.
And there I would ask like questions like, what have you done for AI? And what was the business impact to understand, you know, what have they're built and how successful was it really like, are they making money with this or is it just something that is cool, but then nobody really uses it?
yeah.
Lena: I love that. and since you mentioned the compliance piece already, let's talk a little bit about like the risks around AI. Um, I think there's a multitude of things when there's like ethical concerns in terms of like, for example, quite vehemently opposed, like image generation done through AI because of the way that these models are fed.
I do also have some qualms with, just like text based AI that's,that's just generated using the internet and like all of it, or at least as much of it as possible. And so there's the ethical side of it, but there's also like privacy. I mean, we're based in Germany. notoriously with GDPR, quite strict privacy laws.
Um, then there's also other legal risks. You mentioned compliance, like how did you, either, you know, how did you think about this or how did, how do you recommend that others kind of go about it because it's quite a complex subject. And I think at the same time, the pressure around like making something with AI is quite large.
Daniel: Yeah, let me start with, uh, what, what I know best, because I'm also responsible for internal compliance program. so we are SOC 2 compliant company, and this is within my domain and I think they're what's most important for us because our open source project, you know, like it doesn't actually use any customer data, so we just use, you know, Staging data or our own.
So that's not critical, but for, internal use. So many people use chat GPT, obviously, and. There, I think it's important to be very active and create some guidelines around not using customer data and putting it into these algorithms, right? Because customer data needs to be protected, of course.so regardless if you're in Europe or the U.
S., if you're a trustworthy company and you want to be a trustworthy business partner, then you don't, you know, Put customer data into, you know, external services, unless, customers agree and you have done the due diligence to see that they're also compliant and, you know, there's a lot of things you have to do.
But this is something that's very important to me, right? So make sure that no, you can absolutely not use, customer emails, customer data, and put them into random AI products.
Lena: Yeah, well, I think even the same applies in a lot of cases to employee data as well.Um, so even things like, recording meeting conversations, or you get just generally employee data is in a lot of countries are highly protected, like insensitive data case to, slightly different consumer data because.
Okay, usually your employees, so there's different contractual regulations in place. but similarly, yeah, be careful with what you put in there. I
Daniel: Oh yeah. That's an interesting point that we, I see this in my own behavior. So we record team meetings with Fathom. Cause I think it's super useful for me as an engineering manager to see what. Just get a summary of a team meeting after it happened, but I do not record one on ones. And also I don't ask anyone to do it because I feel it would violate the privacy of people to have, you know, an external service record everything.
Lena: well, I think there's a, honestly, like a good takeaway that like, I think especially for, you know, engineering leaders or people in some of the leadership adjacent positions, like, It's really tempting to tinker around with some of this and it is like some of it is really exciting technology, but at the same time, there are just some things to really be careful with.
AI is just the latest iteration of it. but, where I, yeah, playing around can very quickly have. very annoying and bad consequences. Um, so just, yeah, be careful with what you're playing. Um,otherwise, in terms of the kind of risks around AI, is there anything else that you recommend, people, you know, look into or watch out for?
Daniel: Hmm. I'm an AI enthusiast and you asked me about the risks. I can't see any risks. I don't know what you're talking about.
Lena: Well, then let's move on to the next question.I mean, there, I think, you know, there, I feel like a lot of the things that we've covered so far already go in that direction. You mentioned you're working on a proof of concept. You're not currently shifting your entire business strategy towards this.
you've also limited how much teams are working on this. Like there is a whole thing where, you know, you could argue it's just standard. Business and project risk management, but I think especially in this case, it's even more important. Um, it's like a lot of things that you would apply in other cases as well to like starting a new project or expanding, experimenting with new technology, you should apply here at least as well.
Daniel: For sure. Yeah, for sure. I think it's especially, I mean, yes, it's classic project management to some degree or product management, but I think it's especially true for these very hype technologies. I mean, blockchain and AI, they're usually for some reason I mentioned in one sentence and while they're completely different technologies, actually not that much in common, maybe except running on GPUs.
But I saw the same for blockchain, right? Where people talked about applying it to things where you would say, this is just conventional software engineering. You do not need a blockchain for this. And I doubt that you actually ever read the Bitcoin white paper and really understood what this whole technology is about.
And if you had, then you wouldn't recommend to use it as a database, which it isn't.right. And all of these projects, of course, failed where you could see that people had no idea what this technology actually is for. And yeah, this is similar. We just, we discussed it before. Yeah.
Um, I think for maybe one interesting, so one interesting thing, especially with LLMs, when you talk about risks is, and this is something that I think makes this technology specifically very interesting, is the fact that, um,I mean, there's a lot of discussion around fake news and all that stuff.
Right. And I mean, LLMs are now a technology that is able to create information that is not necessarily true or because they can just make up stuff, right? I mean, this is part of how they work. It's not something that people call it a hallucination as if it were some side effect, but it's actually part of how they work.
right. they are built. basically algorithms that can generate plausible information, for whatever question you ask. Right. So you can ask it whatever, and it will create an answer and the answer will be plausible, but it doesn't mean it's true or based on some facts, yeah, and, you know, there's a lot of work, of course, to, to make them be better at actually answering stuff based on facts and, but yeah.
truthful information, but that's something that's risky, right? Even if you build these things yourself, like it can still happen that they create randomness or in inconsistent answers, misleading information. Yeah.
Lena: Yeah, well, I think, as the more skeptical person in this conversation, like I, like one topic I've given talks about for over 10 years now is, you know, algorithmic bias. and the effects that can have. And now we're seeing a lot of those effects at scale, but the topic isn't new. So I do think, if you are, you know, looking to dabble in this or apply it to your business, like do your homework.
I think, you know, a lot of it goes back to the points you mentioned earlier, like define the problem you're trying to solve. And then also really try and understand what is the technology actually capable of and what are the downsides? Like there are so many publications on this. and then when you implement things, implement them with caution.
Like, a lot of it also, of course, depends on like what problem you're actually trying to solve. but again, algorithmic biases and the issues that can cause has been around for decades now as a topic. and, We're just now seeing a lot of those things with the volume turned up to, well, 3000, I guess, but, they get do your homework and don't just blindly follow the hype or do the stuff that's cool.
because you will probably also have a legal team internally and a compliance team that you will have to run those things by. and like. Actually do that, um, and partner with them and read up on, on the risks that are associated with a specific thing that you're trying to do.
Daniel: Also, one, one thing that I might add, like, now we're talking about risks, you know, suddenly they all come to mind.
Lena: You do know them actually.
Daniel: One funny thing is security, of course, like, especially with LLMs is quite, quite a fascinating the stuff that people do. Like, I don't know if you've seen that there was a blog post last week where,they tried injecting prompts by putting them on images. Right. So you can do image recognition. And so there was a photo with two people on it.
And one held up a sign that said, this photo only contains one person. And then the chat GPT response was just, uh, you know, reading that. Telling you, yeah, there's only one person on the photo. so super interesting. Uh, all the creative security
Lena: Mm-Hmm. ways to inject prompts into this stuff and then misguide the neural net.
Daniel: Yeah.
Lena: Mm-Hmm. .. Yeah. And I think that's another thing where I especially, I mean like I know that a lot of customer support, is currently a big kind of target of ai. like technology, because for a lot of companies, it's, I guess it felt or seemed like one of the easiest areas to solve, like a business problem using it.
but that also, like, as you said, it opens doors, to the public very quickly, in terms of then also misusage. And we've seen stuff like that in the past as well. Like if you put something like that out there, people are going to push it. Past the edges, like that was with like Microsoft's Tay was a really good example, that was shut down in under 24 hours because people just got a teenage, a fun friendly teenage chatbot to say racist stuff within just like, yeah, I think 13 hours or so it was, it's like those kinds of things.
I think there is a piece, right, knowing a bit of the history of the industry and of how users will try and make your software do bad stuff if they can. It's just how software has always been. And I think that's also helpful to at least define a little bit about what the boundaries and constraints should be.
Daniel: Yep. Yeah, that's true.
Lena: so, Let's see. So you've, you know, scoped out the problem. You've gotten some insights. then what did you do after that?
Daniel: We're still working on it, so we're having first, results. And what I'm doing right now is I'm actually trying to make the business case. So we are getting it into a state where we can show it to customers. And, Of course, we, the next step for us will be also dogfooding it, trying to use it ourselves, because this product is something that we can use.
Because if it's not useful for us, then why would it be for anybody else? So those are the next steps, right? So we really want to dogfood the solution, see if it actually provides value to us, and then show it to customers, and then see if anyone is willing to pay for it.
Lena: Nice. and I know that, you know, you got the question, like, what's your AI strategy at some point? So, you know, at what point did you start giving some sort of answer back? Is that still, is that the business case you're writing right now or,
Daniel: Yes, the first answer was the presentation, right? So I explained the plan, I got some buy in from our investor board and from my CEO, from the CPO, just explained, here's what I'm trying to do. Here's the plan. here's the strategy and just get buy in and, see what the reactions were. And so far everybody was positive.
I think they, they liked the plan.
Lena: awesome. what's next for you then with this?
Daniel: Well, next, uh, get the open source project to 15, 000 GitHub stars, I guess.
Lena: But we can add a link to it in the show notes if that's going to add a star on board.
So, what support are you getting for the soul initiative internally?
Daniel: Yes. I like to talk about stuff that I'm doing and that I'm excited about. and I think it's important. of course you have to see it as a double edged sword as a engineering leader, right? like I said, initially. If you talk to engineers about this too enthusiastically for them, it can be a command or it's, or a prompt to action.
So you always have to be a little careful, who you address in what way.
Lena: It's like, I'm thinking about doing this and then three months later, suddenly whole projects. Deliberate that you didn't know about.
Daniel: Yeah. I was like, why the hell did you do that? Yeah. Because the VP engineering said aI is the next thing.
Lena: Yep.
Seen it.
Daniel: So what I did do though, is I talked to my peers and the exec team, right? So a CPO, we talked very like actively about this. So I showed him like my very first crappy POC that, Was way too expensive and didn't really work, but it already yielded some nice results.
showing early demos, you know, just to build an idea. What can we actually do here? And then is there something, and then he took this and talked to customers, you know, occasionally. And would get some feedback. And so this was for me very important, right? Getting some early signals. Yes, there is interest.
People are interested in this. Then of course I do an AI round table at a conference and I ask others, what do you do for validating your, you know, Potential ROI or business case. and, you have some very interesting people there. Well, one of them was an engineering manager from a popular search engine and she explained what they did and what challenges they had.
yeah, so that's what I do for inspire and enable.
Lena: There is also a piece where like a lot of it is really validating as much as possible, like with customers, but also with other people in the field. and, to also, I mean, again, we spoke about kind of ROI adjacent things a couple of times, but like to also keep the risks contained for your organization.like one more question that, someone asked, is the tech is new, the evolution is really fast right now. is there a way that organizations can shorten their learning curve, a bit for them themselves?
Daniel: That's very interesting. I saw, uh, uh, presentation from a PE investor and they had run this research where they looked at all their portfolio companies and looking at reasons for not adopting AI. And the number one reason was I don't have enough time to learn this, which was very interesting to me.
But I kind of get it right. Like it's, if you're used to building software, the way you do, getting good at prompt engineering is actually not trivial. Like I have spent a few times, writing prompts and. Thinking to myself, I could have done, I could have written this code and half the time I spent optimizing the prompts for it.
so yeah, it's a good question. What can you do to shorten this? I mean, one thing that we did internally that, I think was quite nice.as you can imagine, I am. very open to buying co pilot licenses to developers if they want it. And we had an internal knowledge sharing session of an hour where we just exchanged, like, what is everybody doing with, with the tools to just spark some ideas and maybe, share and help everybody get on boarded to the stuff quicker.
So that's something I think helps. On top of that, yeah, I don't know. I mean, you can do some sort of training probably. But yeah, I'm never the biggest fan of that. I think maybe having internal champions, who are really good with this, cause it's often more motivating if you have people on the team, you know, who can, solve problems that are closer to what you do every day. Like if I go to a training, there will be a very abstract You know, series of lessons that might be beneficial, but maybe not everybody can translate this to their everyday work.
So if you are lucky enough to have some AI champions within your team, maybe give them some space to, to inspire others and share what they know.
Lena: Yeah. a lot of organization, like, unless we're talking about organizations of like thousands and thousands of people, like smaller companies. I honestly don't think are largely in a position to make this a technology that's just used by everyone. I do, I would briefly like to talk about, ecosystems like CoPilot, in a second, but, I think in most cases, I think training everyone on what's currently available seems like something that's premature and is most likely just more of a distraction than anything else.
Daniel: Yeah. Yeah. It's interesting. I mean, I can maybe be more concrete. So, I mean, what is the stuff that we do with it? I believe. One thing that might explain, and this is a very personal opinion, right? So take it with a grain of salt. but I believe one thing that explains a certain barrier of communication, especially when you talk, when you see conversations happening between non technical people and engineers, like, non technical people, right?
So people who weren't able to code or. We're not able to write an SQL query before they might be extremely enthusiastic because they can now suddenly do stuff that just wasn't possible for them before. Right. So if I go to ChatGPT or copilot, and I'm like, write me a web server that can read the CSV file that I just downloaded from my boring accountant software, like, That is something that the algorithm will absolutely be able to do.
And it is going to be mind blowing for you if you're not a developer, right? Cause suddenly, Hey man, I can read, I can write software. That's cool. Of course, for me as a developer, creating a rest API and a simple front end on top might not be as amazing because I'm thinking, man, I could do this like, like five minutes. So
Lena: You can impress yourself. You don't need technology for that.
Daniel: Yeah, just use, I don't know, Ruby on Rails or something that you can do this in seconds, I guess. And for, for engineers, I think, this is less, less exciting because I can write SQL queries, right? I don't need tooling for it. Of course, sometimes there may, might be a complex query and then I maybe use ChatGPT for it.
And I'll be happy because it was faster than. If I had done it manually, but the productivity change is more like quantitative, right? So I can do some things quicker than before, which is cool. And I really like it, but it's not a different quality of stuff that I can do. I can basically do everything I could before just a little faster where for some people, this is completely different.
Yeah.
Lena: you mentioned that, you know, you're happy to buy the co pilot licenses for your team members, like how do you use it internally currently?
Daniel: so I think most use cases are what I just described. Like it's really good with boilerplate code. So one of my favorite examples is if you take a database definition, you paste it in and you go create a REST API and it will just generate all the boilerplate code and then you go now create the, I don't know, Vue JS store for it, and then we'll generate that, right. So just solving all of these problems way faster than you could before. This is mostly what people do with it. Yeah.
Lena: Yep. Fair enough. I mean, I think that also matches the first studies that have come out about kind of where it's useful versus what the limits are.but I mean, who knows where the technology is going to be six months from now.
Daniel: Yeah. And then we actually have a pretty bright product manager and I think this is kind of fun, right? Like he's writing SQL now with Copilot and it's very excited because he can do, like I said, right.
Lena: like that's where the jump in like abilities is like really great. Awesome. we're almost at time. It's like, you know, anything else you really want to say on this whole thing?
Daniel: I don't know. What should I say? We're hiring. Is that something you say in these
Lena: You can say that.
Daniel: or will you cut it? It's like, no,
Lena: no, no, no. .
Daniel: no advertising for your company, Daniel.
Lena: Well, you're, you're here by name. You get to say that, you're
Daniel: okay, cool. Yep.
Lena: Perfect. So that is the, are that, are those your closing words?
Daniel: No, I mean, I guess, if I had to close this, then yeah, I mean, I would recommend to look at new technologies with, more. German lens, right? So you can absolutely be excited about it, but then you should also be practical about it and a little more down to earth, uh, really assessing, okay, what's possible here, how can it actually benefit me and go from there and not be, you know, too enthusiastic.
Outro
Lena: I really appreciate Daniel's pragmatism and call out that again, AI is a tool to solve a problem, but it's not always the right tool for the problem at hand. I want to add a couple more recommendations for what you can do to lead through the hype and not just be swept by the hype wave. First of all, don't take all the headlines that you see at face value.
There are a lot of things going on right now. If you open Hacker News, Reddit, or just, you know, your random technology blog, everyone's talking about AI and doing it very loudly. But that doesn't always mean that they really have a substantially viable business model behind it, or honestly, that they even know what they're doing.
Educate yourself, we'll have a couple more book recommendations and other sources listed for you in the show notes for this, read those things and try and really understand them to understand not just what AI can do for you, but also what the limitations are and how to mitigate the risks and issues around it.
If you are facing questions like what's your AI strategy, really focus on what business or user problem are we trying to solve, like how short term versus long term viable is this, you can also ask questions like how will we mitigate the risks around this or how much money is this worth to explore? I also want to call out that good and boring engineering work can be really useful sometimes. Like there are a lot of tools that engineers have available at their disposal and AI is just one of them.
I hope this conversation today helps you bring a bit of pragmatism as well as some concrete things to do to combat the hype and really understand what AI can do for you versus what it can't and ultimately how you can be the leader that your business needs at this time.
Next time, we'll get real on having a micromanager boss and how to deal with that. And we'll also have an episode about neurodivergence in leadership coming up.
Leadership Confidential was created, produced, and presented by me, Lena Reinhard, theme, original composition and mixing by Esteban del Pino, production assistance, guest support, and social media by Sly Stark.
Thank you for listening. We'll hear you next time.
