Six Questions to Ask Before Investing in AI

Brian Ratté: Hey everyone, welcome to StrataEdge Insights. I'm your host, Brian Ratté, here at StrataEdge. Today we're diving into AI strategy with two guests. First, Vlad Lukic of Boston Consulting Group. Vlad is a managing director and senior partner and leads a global practice for tech and AI projects at scale. Vlad is joined by Linda Apsley. Linda is the founder of StrataEdge and also a senior advisor to BCG.

Drawing from BCG's study of 1,500 global companies, where only 5 percent are truly AI-ready and outperforming their peers, we'll explore six questions every CEO should ask before launching an AI project: targeting real P&L impact, deciding whether to build or buy, making sure the data is ready, governance, criteria for stopping a project, and competitive advantage. As always, you can find links and downloads in the show notes. Thanks for checking us out; we welcome your comments and feedback. I hope you enjoy this discussion between Linda and Vlad.

Linda Apsley: Vlad, I know you work with a lot of very senior executives at large companies on AI, and I know you recently did a study on AI. Can you tell us a little about that study, what you found, and how you think it affects the industry?

Vlad Lukic: A few interesting things came out of the study. First, it covered 1,500 companies ranging in size from a billion to over 100 billion dollars, and everything in between. We were trying to get a sense of their AI readiness. Only 5 percent of the companies are AI-ready, both in how they assess themselves and in what we assess they have in their toolkit: operating model, data readiness, and a tech stack that can support them. The companies in that group are outperforming all of their peers, and the gap is widening dramatically. That's one big finding.

Among the other observations: those leaders do fewer things, but they stick with them longer. They invest longer because they know where the value is. They focus on core workflows, because that's usually where the dollars and the competitive edge are. And they invest a lot of time and resources in reshaping processes, changing operating models, and making sure the right incentives are in place. The change-management part is where they doubled down.

Linda Apsley: That leads nicely into the first question. If one group is succeeding, we want to give the rest of the leaders questions they can ask to get into that group. That's the purpose of these questions. The first one is: what's the specific P&L impact we're targeting, and could we achieve 80 percent of it without AI? I find in my work that sometimes people have an idea and just run with it, without thinking through whether the AI solution will provide value. What's your recommendation for leaders on how to spot whether a project will deliver high value?

Vlad Lukic: This goes to basic business skills. You focus on where the value is and articulate it. Too many leaders follow the shiny object: "We'll just deploy AI, and magic will happen." It won't. You need to be very clear about where the value is and then work hard at it. To your point, every major AI initiative needs to be tied directly to the bottom line. If they can't articulate that, that's a problem in itself. The ones that do really well are ruthless in identifying those workflows and staying focused on the ones where the value is.

Linda Apsley: Here's a story you might enjoy. When I was at GEICO, we sometimes had the opportunity to work with Charlie Munger, who's well known for his sayings. One question he would always ask us was, "Would it really be cheaper to do this with technology, or to just have people do it?" When he first said that, I thought it was a really important question, because we don't often ask it. Maybe not very often, but sometimes it's true: you won't gain anything by putting the technology in place.

Vlad Lukic: I just started reading his book, and it's full of wise advice. So don't shy away from asking those basic questions. That would be one big takeaway.

Linda Apsley: The next question is: are we buying or building, and do we understand why that matters? When I was CTO, people would sometimes bring me something to purchase, often an AI solution, and I'd realize we could build it with two engineers in two weeks, yet the price tag was a million dollars. As an engineering leader, that was an important question for me. How do you think companies should evaluate build versus buy in AI?

Vlad Lukic: It has shifted a lot because of how quickly solutions are becoming available, which changes what you can build versus what you can get off the shelf. There's a new time component to it. That aside, if something will give you a competitive edge and nothing on the market does it, you should probably build it and invest the time, because it will tie to the 80 percent of P&L impact. On the other hand, if something is easily available in the market, your competitors have the same access to it. If it's commoditized and priced at an accessible point, buy it; don't build it. No one should build Microsoft Word or Excel. You shouldn't build your own spreadsheet; there are plenty of other options in that space. Don't build solutions that are readily available and are table stakes in the industry. But if something will give you a competitive advantage and you can see a reasonable path to building it, you should lean in. That's at a macro level.

Linda Apsley: That leads to the next question: do we have the data infrastructure and governance to support it, and do we have the right skills? A client I was talking to recently had tried four different companies to build a complex solution before they found one that really knew how to do it, and that had the data readily accessible. The building blocks are equally important. Have you had experience with that?

Vlad Lukic: Yes, on both sides: the vendors that support you, and what you have internally. If you don't have the data infrastructure, you're not ready to take it on. I had a client that said they had 10 years of historical data for the predictive analytics work we were going to do. We started working and realized the way they were saving the data dropped the seconds. For that particular machinery, once readings are rounded to minutes, all the data is one big blob. So they didn't have 10 years of data; they had zero years. They now need to start collecting the data. That reframed the problem from doing sophisticated analytics to starting to collect data, so they can build a foundation for analytics later. So have an honest conversation about your data readiness and infrastructure readiness, and whether you have people who can do it in-house or need a third party. That assessment is a skill each company needs.

Linda Apsley: I definitely agree, and there's also a governance piece. If you're building models, you have to be able to explain the model and show that you're following proper procedures, so people can trust the outcomes. I heard of an insurance company not long ago that built an AI model and didn't test it enough. They thought they had separated their high-value and low-value customers, and they sent out letters. Quite a few people with credit scores over 750, successful professionals, received letters saying they were no longer insurable. That kind of error can happen easily with AI because you're working with large data sets, and yet it can be catastrophic. So governance is equally important.

Vlad Lukic: Yes. In the example you shared, people shouldn't use LLMs to do deterministic analysis. You have other tools for that. An LLM can help you draft notes and so on, but a lot of people don't know the difference and use the tools for the wrong thing. So governance matters, and so does understanding the limitations of different techniques and tools. Within that governance, you also need to decide what testing you're doing, at what sample sizes. Even if you settle on a workflow with one model, when the underlying model gets upgraded, the new version produces new outcomes, so you need to look at all of it again. That constant testing needs to be part of governance.

Linda Apsley: As well as the change management.

Linda Apsley: Do you see companies that are really willing to stop projects when they're not showing value? The question behind this is: how will we know within 90 days whether this is working or failing, and what are our criteria for stopping it?

Vlad Lukic: I see companies stopping them too soon and too late. Many times they get mesmerized because something is working really well technically. The pilots work and the features are there, but it was focused on something that was never a bottleneck, and it doesn't deliver any outcome. They keep it alive, pour resources into it, and are surprised later that there's no outcome. It was never a bottleneck, so they solved a problem that wasn't a problem, and a few people have a little more time. Those shouldn't have been launched in the first place. Even then, they prolong them too long, and they use up too much management bandwidth.

On the other side, there are more examples where companies stop pilots or prototypes too soon. The team solves the technical piece but doesn't see the impact, because nobody did the rest of the work: changing incentives, changing workflows, and so on. Usually IT is driving it, and IT doesn't have the authority to change the incentives of salespeople or of people on the manufacturing floor. So they say, "People aren't using the tool we gave them," and stop the pilot because it isn't working. In reality, if they kept at it a little longer, engaged the right stakeholders, changed the incentives, and reshaped the workflow, they'd unlock all this value. Executives sometimes stop those efforts too soon because they don't look at it as a system and change all the components around it. So I see both, and the key is discipline, constant evaluation, and asking the right questions.

Linda Apsley: The way I've described one of the problems is that the technology team sometimes builds it and throws it over the wall. You might have been able to do that with some technology, but with AI the business process is so tightly tied in that you need to work together with the business, so you don't end up with IT projects that sound good but don't deliver value.

Vlad Lukic: Correct.

Linda Apsley: The last one is a little like where we started: will we gain a strategic advantage over our competitors if we implement this, or will we be at a disadvantage if we don't? As fast as AI is moving, a company that comes up with a really clever idea may not look like it has a competitive advantage right now, but it could if it becomes a trailblazer with something no one has thought of before. For me, this is an important closing question: think it through and put your money where the value will be greatest.

Vlad Lukic: Exactly. That goes to the core of running a profitable business: being able to articulate the value you're delivering, as a product or a service, and what makes you different from others. I'm not a banking person, but I've spent a lot of time over the last few weeks with people in digital banking. I was telling them I was surprised that some banks that scaled up very quickly aren't taking over more. They pointed me to one fact: those banks are very good at attracting a large number of customers who deposit small amounts of money to use for payments. As a result, they never become the customer's primary bank. They're just used for payments, so they never gather enough assets to run as a full bank, and they struggle financially because they didn't change customer behavior. The convenience is there, but they didn't persuade customers to bring the rest of their money and banking needs, because they don't offer the rest of the portfolio. So the existing banks have an advantage if they add those features; they could lock in their customers even more.

Having those conversations and focusing on what the value is and what makes it unique is where the effort should go. If you can't articulate it, you have a problem. That becomes your true north, and it ties back to the first question: where to spend the effort and where your differentiation will be.

Linda Apsley: Thank you, Vlad. That was a great summary to end our conversation. Thank you for coming by; I really appreciate your time.