Why AI initiatives stall; and how IT leaders fix it

So I'll ask you guys, give me just a a brief intro to who you are and why you're here. Yeah.

Sweet. Hi, everyone. I'm Amy Stokes. I'm an AI success engineer at OpenAI, which basically means I'm supporting our large enterprise and strategic customers with helping them adopt AI in a scalable and safe way.

Hello everyone, I'm Renal Shreedhar. I'm a worldwide specialist solution architect at AWS for Amazon Quick.

Hi everyone, I'm Justin, I'm director of Google Workspace for UK, Ireland, and also sub Saharan Africa.

Perfect, thank you so much. And I wanna hit us, hit the ground running with, we've been talking a lot and I mean definitely some of the conversations I've had over the day today, I'm very focused on sort of the technical aspects of IT pilots.

Data connectivity, the right models, but I definitely see there's less of a focus on the non technical aspects. So I'm wondering, and I wanna actually get all of your opinions on what do you see as sort of the key non technical aspects that a lot of organizations, a lot of leaders are missing when it comes to deploying sort of AI solutions? And Amy, if you wanna kick us off.

Yeah, that's great. I get to go first and then you guys have to think of something else.

Yeah, I would say the thing that I see the strongest customers that I work with do really well is really create a culture of experimentation.

Experimentation is so important. You have to create the space for people to try new things, and I think often there is the misconception that you have to have the perfect technical blueprint before you actually start building. That's actually contrary to how you really leverage the advantages of AI. Firstly, the models and the technology stack is evolving at such a fast pace. If you wait several months to design the perfect blueprint by the time you're building, it's already redundant, and so I would just advise everybody to really put the systems and the AI into the hands of people to experiment early and fast, and that's also gonna help accelerate their learning, your learning much faster too.

Would you agree, Munal?

Yes, absolutely. And I think in my experience, what I've seen is a lot of times, there is no real shared clear problem statement that people are trying to address, right? So, you know, if you got three people in the room, they've got three different idea of what the problem statement looks like, right? So for example, you might have a CTO who thinks about this as a data modernization project. You might have a COO who's looking at it as workflow automation. Or you might have CEO who's heard a competitor earnings call and they mentioned AI, right?

Yeah, just something with this AI thing.

So I think, yeah. So the challenge here is like a lot of times people don't have a commonly agreed upon clear problem statement, right? So the classic things that we all care about, right, which is like, where does it hurt, right, what are our pain points, do we all agree that there's a good starting point, right? And if yes, then let's just go ahead with it, right? So that's what I think is quite important.

Justin?

Yeah, Ivan, I think from my perspective, I just think it's almost the classic tale of enterprise readiness.

The tech is ready, but the organization is not. And I think we see that repeating, not just with AI, but with all sorts of tooling that we see through the years. We talk a lot, and there's some research done by MIT, by McPhee, about the productivity paradox, which is this repeating situation where when you apply new technology, the expected productivity doesn't always follow. There's always a lag, and I think the classic example of this is when they move production lines from steam power to electric power, and there was no productivity increase for thirty years. There was a thirty year lag, and it was only when they adapted the production line to attributes of the technology they then got that. So thirty years later, they realized, hang on, we can use these things differently, and then the production was ten times increase. So this whole concept of just putting the technology in, you have to have an organizational readiness for it, and that's what we see a lot.

Which is definitely, I definitely see this as a very iterative process. We need to get the culture right and we need to start preparing to change the ways that we work whilst maintaining.

I definitely see pilots where they're completely detached from the actual business users. We just have IT pilots opening up in a vacuum.

They're never actually getting to production because they're not useful. They didn't mix the business SMEs with the actual AI experts. It's about combining the two of them together. Getting the culture right, getting the actual what are we doing with this thing right, and then figuring out that eventually work will change because of AI. You can't just plug in the technology.

But talking about plugging in and talking about data, obviously there's a big push. We want AI to give us the best outcomes as possible. We want to connect all of the data to it. And I imagine mostly to you, Murnal, you talk a lot with organizations that have tech stack for years, fragmented data ecosystem, tools sprawled, siloed data, and they're like, I can't do anything with AI because of this. Where do you think and what do you recommend organizations should start with when they're facing that problem?

Yeah, so I mean, the way that we can think about this is there's a reason why we are where we are. Okay, so a lot of it could be just cultural. Right? There could be legacy issues that could be resulting into this, And that's not something that you're gonna change overnight.

And does that mean that you're not gonna then go ahead? Yep. You know, so I think the way to approach this is rather than starting with architecture diagrams, right?

Actually work backwards, right? So you basically do the classic, do we have a use case that we can work on? Yeah. Right?

If you don't have a use case, right, then can we all sit down, do a nice little business discovery and identify a bunch of use cases that we can work with. Right? Now, the way to identify the right use case, I like to say that, you know, go with the one that's got the highest impact, widest reach, most visibility. Right?

This could literally be a low hanging fruit. And it's okay to go after low hanging fruit, right? Just because something's low hanging doesn't mean it's not worth trying, right? And then you go at it, but don't over complicate things.

Right? So, you know, you don't have to go with thirty different data connections, you know, for your first use case. Right? Yep.

Start with three, four. Okay? Make sure that it works. Right? Keep it simple. And then roll it out there, iterate over it, get some feedback and then off you go.

Right? And once you got success in one pocket, in one use case, right, use that political mileage, right, to then go and address other use cases in the business.

Yeah, no, definitely starting small but effective.

Yeah, we've been geeking out the whole morning about this but it doesn't have to be the fanciest model, all of the data for each get ROI. Sometimes the simplest use case with the simplest model and just sort amount of data, I see this a lot with use cases that would take humans hours to triage through things, they can just get AI to do it. And on the other sort of side of the coin, do you guys think that there's like a minimum, like a baseline of where it actually is good enough to start generating ROI with data connectivity, sort of a good starting point? Do you have any points?

Yeah, I think I piggyback on what Murnal just said, that you can prototype literally in a matter of hours now, we're not talking weeks, and so within a day in a room you guys could have an application semi built and not integrated, so I would say never wait until you have a full design or full integration pattern in place before you get started.

And the other thing I think, there was perhaps some misconceptions, right, that models can't perform well if the underlying data layer is not high quality.

That might have been true two years ago, but today, the frontier models are not only good at connecting to your data, but also good at connecting what data is high quality as well, and so I think, back to my original point, experiment early. You'll learn more from where things go wrong and when you're testing than you will trying to get the perfect pattern immediately, and also, as we've seen today from the keynote, right, there are so many now progress in out of the box data connectors, MCP layers as well. Also AI can build your integrations for you really, really quickly.

So yeah, again, just crack on, get started, and you'll find the right path through experimentation.

There's definitely a move fast break things culture that we have to do. Okay, let's say we got all the data that we need connected for our use case. We got a couple of use cases identified, we have a priority list and a roadmap of what we wanna be doing, But we need to get people excited and enabled on how to be using AI.

And I imagine, Amy, you spend a lot of time with customers as part of their success team.

Why would you consider there to be the basis of a change program that's good instead of the typical, yeah, we'll get them on a webinar and they're up and running with ChatGPT?

Yeah, so I have heard some stories of like years ago when people, literally that would be it, it would be an email, hey guys, you have access to ChatGPT.

Crack on.

Off you And look, you're gonna have some like very AI savvy and AI literate people in your organizations, and that might work for, but you really need a structured program that is going to meet everybody where their needs are. And there are a couple of things that I've seen done really well in organizations.

The first is making sure that you have a really clear top down support and approach and influence. If your executive leaders are not using AI every day, you will not be successful. Yep. And so they should be aligning all of their strategic priorities and make it really clear and communicate to everybody why AI should be at the forefront of those strategic priorities.

And also communicating their successes and sharing that, because that will trickle down.

And then it comes bottoms up.

The best programs actually are where you give everybody access to experiment, and then you identify where your power users are or your champions are, and those are the people you need to go to, sit down with, and understand, okay, right, what are you using this for? Where is the value? What are you doing? Don't try and dictate what people should be using AI to do or tell them how they should be using it. Go and speak to the ones that are already driving adoption up and getting value and learn from them.

And then the final piece, once you've identified where those real sweet spots are, is then addressing where people are not using AI. And I've seen it very different across organizations, some it's very concentrated, and so you can be very focused on one domain or function. And then others, it's very sporadic. But look at your data and understand where there is low usage and then you can build more targeted enablement, targeted communications to really address and work with those people.

No, that makes a lot of sense. I've been talking and sort of mentioning a lot the building of champions programs, connecting these SMEs, power users with their AI council and their AI experts and making sure that they get all that they need to embed AI into their exact business function.

Yeah, tribal knowledge and the network effects are the most powerful ways that you will drive adoption.

Yeah, dictating change will never, never work. Munal, Justin, anything that you would add to that?

Yeah, just on the top down, right? I like to say that lead by example, not by email.

I like that. Yeah, I love that.

Because if you see senior leadership using AI in their day to day, sharing the benefits that they're getting to the rest of the organization, is just so much more powerful. Yeah.

Right, than sending an email saying, hey, now you have access to this and we're gonna keep an eye on you to see if you're using it or Yeah, mean, really building on the points that Amy and Ronal said, but I think this culture of AI fluency is really important.

And I think one of the things that we think about Google a lot is really equity of access to tools.

We see a lot of organizations where you've given some tools to some people Oh, yeah.

And some people have it, some people don't, and actually creates a lot of tension and understanding. We we might talk a little bit about the fear of AI, and I think the equity of access is important because you create a culture of fluency. So building a lot of the points that have been made, I think top down enablement and empowerment, empowering people to make mistakes, empowering people to use it is really, really important. And I think sharing examples, I mean, we will laugh one day that we were kind of thinking about this, because prompting is like a new UI, right?

It's a new way of doing that. And I remember many years ago teaching my wife how to do a search on Google. How do I search for this? No one talks about how do I search for this, but we used to say that.

Yeah. And I think we're in that world now where we're doing that with prompting, and one day we'll laugh about But it's a new interface, so I think what you have to do is give people the space to fail, experiment, be curious, but share those examples, and I think we've talked a lot about that, that tribal knowledge where, and just try things. I've learned a lot, and actually I've been educating people in Google myself I tried this and this worked, and everyone's, oh wow, I didn't even expect an outcome, and it just really creates viral moments.

Another really important point that you've just raised is also creating the space for people to try, but also creating the time.

Especially when you start learning, it does require you to put aside a couple of hours a week to actually test things out. Often, I've been my own worst nightmare, where I'm like, I could automate this if I had the time to automate it, and it could save me hours, and it's such a silly mindset, and so like, even for me, myself, where I'm literally inundated with AI tools, I have to make sure like every Friday afternoon that's no meetings, that is my playtime.

One thing I would say, and I'd say this as a tip, is that don't try and apply traditional learning models and traditional ROI models to AI, because when you're doing a traditional training course, it's typically about discrete functionality. The file menu does this, the print menu does that, and you get taught how to use it all. The problem and the opportunity with AI is just ubiquitous. It's there in context of whatever you're doing at any given time. So you can't really apply traditional training methods to what you do, so you have to create these new ways of how people are going to ingest and learn. Think that's one of the areas we see is trying to use the ways that worked for old ways of doing things in ways, new ways of doing things. You have to break that model and people default to how we do training and actually you have to do it in a very different way with AI.

No, one hundred percent.

Raju, we're talking about sort of power users and people that are excited and eager and testing.

But let's talk a little bit about the opposite as well because we're all seeing that there's anxiety in workforces about sort of all this conversation about AI, changing roles, replacing roles, which will limit what you can do with these AI pilots.

Actually, I'll point this to you, Justin. Do you have any recommendations on how you see organizations should talk to their workforce about this anxiety? I Being honest and sincere?

And what frustrates me actually is when people start using cliches like it's going to augment you, not replace you, all these sorts of I think you actually have to meet people where their fears are, and we sometimes at Google share an analogy about chess, so hopefully some people here are as old as me.

I remember back in, I think it was nineteen ninety seven, when IBM Deep Blue beat Gary Kasparov at chess using a single function AI tool, and everyone thought the world was falling in. They're like, oh my goodness that's it, it's going be robots and so on. And everyone thought that would be the end of chess and we'd have to move on to anything else. What's actually happened now is chess today is three times more popular than it was in nineteen ninety seven, and there are three times as many grandmasters today than there were in nineteen ninety seven.

And that's because people who have played chess have used AI to train themselves, to get an opportunity to go against talent that they couldn't otherwise find when they didn't have a human person. So AI in chess has raised the bar exponentially. So I think that's quite a useful example to think about, about how you can adopt AI to support you Yeah. And know those things.

So I really like that analogy, and I think what's important to say to people is it is change. It's one the biggest changes that we've come across, and that we have to face that head on. And I think we've talked a lot about it here at Asana today, whereby you're not going to do the same things. It's not going to be a measure of how many hours you work, but more about the impact that you It's not going to be so much about those things.

It's going to be more about, you know, not just being a doer, but being an orchestrator. And I think all of these things are going to be things that people have to embrace. And if they don't embrace them, it could be challenging. I think you have to accept that.

But I think people's role progress will change. But people need to see that by embracing it, the chess example, you can get to your new role profile.

Yep. Yeah. Yeah, we were talking earlier, I definitely see a world where roles will become larger in scope, skills that are sort of more generalist will be more important because you will be orchestrating agents with specialist skills that will be able to perform parts of the work for you. Whilst at the same time, skills like critical thinking will become more important so that you are, you all will be requested to analyze and review that work and orchestrate.

We will all be managing teams of AI agents rather than us developing the work ourselves. Which, I think to your point, is less about the hours or the papers or the presentations that we produce, but the quality of the outcome that we have. And on that note as well, there's definitely, I'd imagine a lot of you in the room will be feeling this, tension between, well, a workforce that will be excited, but some of them will be a bit more cynical than leadership that are super excited. They've heard it on someone's earnings calls.

They wanna get moving with it and usually IT leaders are in the middle of this and having to manage all of that. Actually, I'll start with you Amy. Do you see this? Do you have any recommendations for leaders that are trying to manage their pilots realistically and pragmatically, but they're stuck in this middle?

Yeah, yeah, I see this all the time. And I think at OpenAI we really try to approach them with empathy, especially when I work with execs, get them super excited about their technology.

They then put that on the middle management or IT managers to then go and implement.

So all of the things that we've heard today are true.

And then I would also say those teams should try to embrace that change and look at it as an opportunity to perhaps not work with the systems that they have in place today, but reimagine what that will look like, right? I think it's okay as well to be patient and push back on some of the timescales or milestones that leadership put on. I think, yes, impact is important, and you should definitely have that north star and goal in mind when you start any use case, but also look at what those small wins are or eureka moments are along the way that will help you both inspire your teams and also have really good stories to tell execs.

And again, I know I've said experimentation like a gazillion times, but in doing that you will identify things that maybe you didn't even think were possible, so don't try and granularly say, okay, we've gotta hit these specific targets by this specific date. More broadly be like, this is where we're going, let's see what happens when we try this thing So I would say that, and then the other thing, just recommendation for teams is try and reshape what the organization is. You said we're like onboarding agents.

That means the skills required are different, but that also means the team profiles are gonna be different. I no longer need twelve or fifteen people in a team for a project, I actually probably need three or four people, because with AI, one person can now be a product owner, they can be a designer, they can be an engineer, they can be a tester, so you'll also be able to get quicker wins if you like, small down? That's not the right word.

Size down your so that you can work in a more agile way.

We're gonna coin small down. Small down. Anything you would add, Justin?

I I was talking to you earlier on, we've all seen in a very compact window, people turn from skeptics to evangelists. And I do a lot of work with journalists, and it wasn't that long ago where some of my journalist media companies were saying, and they actually put it on their page, think the Financial Times said, we will never use AI. Well, know, time moves on, and actually I've seen journalists go from never over my dead body to it's now an essential part of my work in terms of verifying information, checking information, doing different things.

And when they realize that it's actually doing laborious heavy lifting but not getting in the way of what you need to do, and actually enable you to present information in different ways, not just in the written word, we've seen now that it's been massively widely adopted because it's opened people's mind to the after possible without replacing actually the core essence of the work that's being done.

No, the last three years have been a roller coaster of changing of emotions for sure.

One of the things I've personally have seen in AI pilots that is essential for adoption of AI products or AI solutions rather than becoming shelfware is the way that it is sort of designed and deployed, the way that people experience AI. Would you have any recommendations, and I'll start with you Amy again, on how these pilots should deploy and how they should design that experience?

So speaking of like the product specifically, I'm trying to think of something that I've not shared before. Okay, one thing that I think you can put in place to help sort of measure the impact and quality of a solution is making sure that you have mechanisms to be able to observe and monitor your AI agents in development and also in production. A term that we use a lot at OpenAI is evals, short for evaluations.

And the very simplest way it basically means make sure you know what good looks like before you actually start building. And so when you think about what does a good experience mean, think of a customer contact center, That might mean, Okay, well, I need the AI agent to sound friendly. Okay, well, does friendly look like? Go and get some great examples of what friendly is.

I need to make sure that at the end of the call, they close off with this specific phrase, right? It could be very deterministic like that. It could be, I need to make sure that we've solved the problem, So you build these evaluation sets so that you know exactly what good looks like, and that actually helps throughout the development period to know if the agents are going to perform at the quality and accuracy levels that you need to before production. And that's also incredibly important for when you go live, because you can then continue to measure and observe the experience and the performance of these AI agents.

And so my recommendation before anybody starts building more complex agents is start with evals, understand what good looks like, and then build that data set over time.

That's really interesting because it's making me think a lot of conversations I've had over the last six months where people are using tools like Asana's AI Studio to automate a particular process and I hear this a lot of, our process is so unstructured, it can't be automated by AI and I'm like, well then the problem is a little bit different. It's not an AI problem, this is gonna be hard for a human nonetheless.

We need to determine what good looks like for AI or for a human so we can move forward.

That is so true and I think we were speaking about this before which is sometimes it might be the case that you don't wanna just recreate or reinvent what already exists. Yeah. And actually sit down with the business and ask them like ideal state, remove all of the tools that you have today, what outcome are you gonna drive? Okay, and can we actually reinvent a new agent without trying to stitch this crazy process together make it more optimal and better suited for what their needs are.

It's fine to go back to the drawing board. Oh yeah. As long as we know what we're trying to.

Throw things in the bin, start Yeah, yeah, yeah, exactly.

Anything you would add, Justin or Marnell? I'll just add, I mean, just building on this, right? And that is, let's put AI aside for a moment. Yeah. Okay?

And let's start with humans, okay? So, this is a litmus test, Okay? So you ask someone, you have all the time you need, go and get this either task done, right?

Get me this answer.

Okay?

Now, if they can't come back with it, let's be honest, magically things are not gonna improve with AI. Yep. Okay?

Second is, you know, if they come back with outcomes or outputs that you just simply can't trust, that's even scarier. Because I feel that with AI, you're just gonna get there faster, and there's that halo of intelligence around it, where you're just gonna trust it somehow. Yeah. Okay?

So, that's where going back to the drawing board and then figuring out, right, what really needs to be solved. It's not necessarily an AI problem. It's the people and the process. Yeah.

Bit that we have to fix first, right, before AI can come in and help, right? Otherwise, we're just gonna make it faster and jump into it not knowing what we're signing up for.

Yeah. The AI is an accelerator. It will accelerate nice, tidy, good processes, but it will also accelerate untidy, chaotic processes, not to anyone's benefit.

And yeah, I definitely could be talking for hours, we haven't even gone through half of the things I wanted us to discuss but we are running out of time. I do have one last question for the three of you, very actionable.

For all of you in the room, if there's anything that they could do in the next, well starting today to the next ninety days in order to be successful with their AI projects and pilots, what would you recommend they do? And I'll start with you Justin.

This part's gonna sound like a sales pitch, I honestly would say roll out Google Workspace because I'm sorry for saying it, but what I mean is that it's kind of AI with a very homogenous contained container, so it's very good within its domain, right, on the data that it stands because it really understands that. So it's like a very benign platform that you can roll out at scale to get people fluent. It's about equity of access and getting people fluent in AI. I think because if you get your people ready, when you then go onto the bigger projects that could be using Claude, OpenAI, or Amazon, or Gemini, connected to all your tech stack and your legacy debt, you've created a workforce that's prepared to meet that moment.

But giving people that core platforming and common language around AI and skill set something that we see is going very well. So we kind of work in a sort of two speed process where we want to get people enabled at scale, but then we want to go deep and do things that involve some really exciting workflows, integrations, and automations, and so on. So I think it's about really raising the tide of your employees through a common language and fluency.

Yeah, no, would definitely agree that finding Apologies for saying that.

That connected context layer is by far the key thing.

Getting people using AI wherever they are already during their work is is fundamental to I mean, let's be honest, this is a technology that people are choosing to use in their private lives, right?

So, you know, everyone says, oh, chat GPT this, I chat GPT that. So, there is a barrier that's already been broken Yeah. That I think where a lot of technologies come in and people say, over my dead body, but the fact is people are using these technologies, so I think that is something you can leverage to bring the organization forward, that this is something that people are embracing.

Yeah, yeah, one hundred percent. Runeel?

Yeah, so I'll actually build on that.

I would say, why don't you start with AI, right? So pick your AI tool, the one that you like or the one that you use day in, day out, and just have a nice little therapy session talking about, you know, what are the type of things I could be doing with AI in my business or in my life or in my team, right? So just have that brainstorming session with an AI tool of your choice and have that conversation. It's gonna come back with a bunch of responses.

Fine tune, select the ones that you think could be a good starting point And then, just go ahead, don't think too much, don't over complicate things, right? You've got ninety days. Go ahead, roll it out. You've got ninety days, right?

And then, come back and see what value it brought to you.

The type of metrics that you could be looking at here is, if you're thinking about individual productivity, right, so if you're thinking about an individual or an individual contributor in a business, right, think about it from the dimension of time, right? So this is how much can I do more with my time or how much fun I could have with my time and so on? If you're, let's say, mid manager, then think about people, right? So how much more can I get out of the team that I have, right?

How much more productive my team can be? Number of projects we could be working on? You get the idea. Think about people.

And then if you're a top exec, think about it in terms of money.

Does that help me bring in more revenue, help me with my bottom line, reduce risk in the business?

So you could just apply this as a yardstick and then go about picking one. Perfect, thank you. And lastly, Abe.

Yeah, loved both of your answers and just to piggyback on Justin's, it's such a good point that there's a reason that both Google and think OpenAI have been so successful in the enterprise is because people were already using it in their day to day personal life, and definitely leverage that, and as you said, don't have a pilot of a few people testing these technologies.

Give it to everyone because you'll learn so much more.

But just to leave on a very pragmatic note, the other thing that I recommend is don't have a siloed AI transformation team that's working to dictate and define what the AI transformation is gonna be for the business.

You need to be going and speaking to each of the business lines and really understanding what their challenges and problems are, and if you are not already, become best friends with your CISO head of security, because in many of the deployments that I've done, it's not been the technology, it's not been the ideas, they are in surplus, but you get to a certain checkpoint, governance is important, controls are important, make sure you understand what those are and work with your security teams to redefine what they might need to be for agentic rollouts. So go take your CISO out for a beer.

I love that. And with that, we're out of time. I'm sure you'll have plenty of ideas and things you wanna go and apply when you get back to your offices. I'll be around, so come and find me if you have any questions or wanna kick out for a bit more, but in the meanwhile, Amy, Rinal, Justin, thank you so much for your time.

Today's been great to chat. We had a lot more chats as well, we've been gigging out for a few hours now. But yeah, thank you so much everyone and hopefully enjoy the rest of the conference.