What matters next in a human-agent world

Hello, humans.

Oh, it's so exciting to see your human faces here in the audience. You are a gorgeous bunch. I'm so excited. How's this day been for you? Fantastic.

Fantastic. Well, I'm excited because we get to finish out this day by talking about humans and agents working together. In order to do that, I wanna do that with a little help from a fish, a form, and a finger. That's an unlikely trio I know, but bear with me and we'll get there.

First though, I think in order to talk about what agents can and cannot do, we probably ought to talk about humans and what we can do.

So I think one pretty fundamental place to start is just by asking, what do we even mean when we say humans? What makes us human? After a day spent thinking about agents, let's take it back for just a moment and think about ourselves. Be a little self centered.

I would like to invite you to think about one word that for you really feels like it captures what it means to be human?

Yeah. You're starting to do it already and I was gonna say once you have it, start shouting it out at me. I heard love and I heard joy. What else is there?

Meaning. What? Meaning? Someone say meaning. Are you stealing my thunder?

Joy. Choice. Okay. Empathy. Empathy. Compassion. What was the last thing? Imperfect. Imperfect. Soul. Soul.

What else? Feelings.

Emotions. Emotions.

I missed that one. Ethics. Ethics. Okay.

What else? Did we get them all?

Humor. Alright.

Creativity. These are great. One more. Connection. Connection. I love it. These are fantastic. So I like these answers very very much.

You came up with a great set.

Let's give yourselves a round of applause actually. Yes. It's the end of the day and you're doing well. Alright.

What I love about this is I have asked this question in so many rooms around the world And what's fascinating is no matter where I am in the world, no matter what language people speak primarily, or you know, if it's the Southern half of the world or the Western half of the world or whatever, people say the same things. People say the same exact answers. And that's really fascinating. It's not that unusual when you think about it.

Right? Because we're all humans and we all have sort of the same understanding of what it means to be human. But it's cool to know that we do all have this kind of same understanding of what it means to be human.

And when we boil it down, like for me, I've heard these answers enough that I start to think of them in sort of two buckets of answers. And you will have heard that too from the answers you heard yelled out. So the first bucket I hear is like creativity and problem solving and that sort of thing. Right?

You heard that sort of rational answer? And I think that's a good set of answers. But I think if we think about it and we think about non human animals, we can sort of understand that maybe that's not a uniquely human trait. We've seen ravens using tools and monkeys using tools.

There's even a BBC nature documentary where a fish throws a mollusk against a rock to crack it open so fish use tools?

That's bonkers.

But so okay, we're not the only tool using species. We're not the only ones who can be creative in our problem solving. But what about machines?

What about machines? Like, if you ask a machine to do something, is it being creative when it solves your problem for you?

Like if you put in the prompt, what does it say, modern art in the style of Picasso depicting humans at work in offices, and this is what you get back.

I mean, right? You may not want to hang it above your sofa, but these are not a bad approximation of what that generally would look like. So what's the difference between that and Picasso?

Right? So hold that thought for just a moment because the other set of answers that I hear very often and heard here a lot are like empathy, love, compassion, feelings. Like, this is a very loving room and I like that about you. That's awesome.

But also think about that. Is that a uniquely human trait? Is that something that we only have? Like, don't think it probably is, I think most of us would agree that it isn't. In fact, I wanna prove it to you. Like, raise your hand if you have a pet at home.

Raise your hand if your pet loves you.

Yeah. I know. I know. Your pet does love you, and I'm not taking that away from you. We're gonna all agree that your pet loves you. So we're gonna agree that that is perhaps not a uniquely human trait to love.

But what about machines?

Like can machines depict something that feels a little bit like empathy or compassion? Like what about a therapy chatbot?

Right? If you're interacting with... I just love the name Woebot. They don't pay me to use this or anything. I just think it's such a great name.

But there's a whole category of conversational chatbots that are programmed to do therapy with you basically. So you might put in something like I've been feeling anxious lately and it might say how long have you been feeling anxious? You might say three weeks. Been going on?

Did something change three weeks ago? And you might go like work got really hard. My mom's been sick. You're off and running and you're sort of unpacking your own issues in a way that's not that different from what you would do in like talk therapy.

So what's the difference between this and a human therapist? And what's the difference between Dolly and Picasso? And if what you're thinking in your mind is something like, I don't know, it feels like there's something about the intention behind it or the purpose.

That would be what I would suggest is the correct answer here because it gets down to the heart of what it is that someone means when they are doing these actions.

That's what I think is at the core of what it means to be human, is that humans crave meaning. We are meaning makers and we are meaning seekers, and we thrive on meaning, and we look for meaning everywhere even where it doesn't exist. That's why conspiracy theories happen.

But meaning, I would argue, is at the core of human experience.

And what's important to understand about that is when I say that, it sounds abstract but it actually has many levels of abstraction, but reality. Right? We understand that there's this whole semantic layer of how we communicate with one another, and that's part of meaning, but we also know that there's purpose and patterns and status and significance and truth and all these middle layers, and then there's the big macro cosmic existential what's it all about and why are we here layer. And we mean that too when we say meaning. And all of these things sound like they're conflicting with one another, but actually it turns out across languages, across cultures, are words for almost every one of these. So as a universal human experience, recognize that there is this multidimensional layered understanding of meaning, and all of it means the same kind of underlying thing, is what matters. Meaning is always about what matters.

What matters to us, what matters to someone else, and what matters enough to shape what it is we do next.

And I love I love this question about what is the fundamental human condition because nobody ever says like, you know, processing capacity.

Right? Nobody's ever like, humanity, the ability to summarize a PDF.

Right? Is that a good characteristic? Is it useful? Sure. But is it a defining characteristic? Probably not. Right?

So as we tend to think about what we're gonna use agents for and when we go back to work and we think about designing for humans, a lot of times we we think about how we're going to create these nuanced experiences, but then we reduce and simplify down to like a drop down menu. And we have to.

But today is about a new possibility. It's about thinking about how we can work together across humans and agents in that shared context, and work that can move across teams, and it's genuinely exciting.

But I just want to spend our last half hour together on a question that sort of lives underneath of that. It's not just like what can these systems do, what can agents do, but what are the conditions that we are creating in order for them to succeed and for people to thrive? Because they're not necessarily the same thing.

Right? And in order to explore that question with you a little bit, I'm gonna need to take you underwater with me and I'm going to need to...

You're like, oh no, this is a work innovation summit. What's this with fish? She keeps talking about fish.

But here's the thing, in Jonathan Haidt's book, The Happiness Hypothesis, he asked us to consider if you design the mind of a fish. If you design the mind of a fish, would you make that fish respond equally to an opportunity and a threat?

Like say there is something that might be food and something else that might be prey that might eat the fish. Would you have that fish respond with the same level of urgency? Probably not.

If there's... Because if you miss lunch, right, there's probably going to be more lunch.

But if you miss the predator, you are the lunch.

So those are very different consequences. So you would want there to be a distinction in the way that the fish brain processes these things.

But the thing that's interesting about that is that there's no designer behind this. Natural selection doesn't have a designer. It's design without a designer.

Nobody... I sat down and created a fish requirement specification. It's not like must support swimming, should support snacks, predator avoidance priority one. Those are not the things that happen. It's about the fit for the environment. Right? And that's fascinating. It's purposeful looking behavior without anyone having to specify the whole pattern.

It actually gives us a really useful way to thinking about agentic AI. You see I snuck it in there.

Because an agent is not a fish, clearly. And updating a workflow is not biological evolution. And not every agent learns continuously from what happens to it. But whether we're changing prompts or evaluating outputs or updating models or revising workflows, we keep making selections.

Like the response worked, let's use more of that. This result scored well, repeat it. The behavior got approved, keep going.

Over time, those choices can produce a pattern that nobody designed in full.

So the interesting question is not really like is this system evolving? The question is what is doing the selecting?

And in our organizations, the answer to that question includes our goals and it includes our incentives, and our data, and our measures of success, and it includes the things that we're not bothering to measure.

And you know this already because, you know, you work with humans. But the thing is that if you tell a team that the number that matters is how quickly they close tickets, they are going to close tickets quickly. Right? If you say that the number of meetings matter, oh you are going to get meetings. You are going to get meetings about the number of meetings.

And we all know this. Right? But the thing is that what we also know is that getting better at the score is not the same as getting better at the purpose.

Correct?

So if what we know about this is that if we are trying to accomplish helping someone get back to work, how many tickets we close might be evidence that we are moving toward that purpose. It might be evidence we consider, but it is not the whole purpose.

And the system can become exquisitely adapted to the measures that we we've articulated while actually drifting away from what the measures were meant to serve.

It's not because it's become malicious. It's just because we made the wrong thing look like success. So here's the thing. There's a lot of human information that lives in other.

But there's a lot of situations where we create what what are known as binary gates. Right? Where we force an articulation or force a choice and and we answer a question that allows us to click next on the form.

Right? That's not a new problem. That's not created by AI. We've been simplifying people and solutions and situations for a very long time.

So we think about high priority or low priority. We think about qualified or unqualified.

Are people engaged? Are they disengaged? Are they on track? Are they off track? These are all things that we've reduced to a yes or a no in a form. And sometimes we need those distinctions. Work requires decisions, and at some point we have to, you know, send the message or not and approve the request or not.

But a yes or no question does not make the world a yes or no place.

There's a lot of nuance that gets lost.

I'm not proposing that every expense report becomes like a philosophical retreat. Like, I get that. We're we're not necessarily asking for that level of of review, but there's a difference between making a decision and deciding that we now understand the whole situation.

Imagine a project that has been marked late. It's red. It's glaring in your system. The system knows it's late. But why is it late?

Is it late because someone forgot?

Is it late because, you know, the client changed the brief?

Or is it late because the team discovered an accessibility problem and decided not to ship it to thousands of people?

Like those are three very different kinds of late.

And one might need a reminder, one might need a new agreement, and one might deserve a thank you.

But if the only fact that survives that process is late, the agent may recommend the wrong intervention with admirable efficiency.

Right? And then another agent reads that recommendation and another one builds a plan on it. The shared context can make a team much more capable, but shared context also raises the stakes of what we put into the context.

It's part of why the conversation about work infrastructure matters.

And it's part of how we need to take this forward when we think about how we can articulate what the questions we need to ask are. What is the status based on? When was it last true? What changed? What did we leave out?

What we know is that late, because we found a problem worth fixing, is different from late. Right?

Customer declined this offer last month is different from cuff... Customer never wants this ever.

Those nuances are small, they sound small, but they're where good judgment lives.

And good judgment is key because context is actually not just more data, it's what keeps data from meaning the wrong thing.

So we need that context.

There's another wrinkle. Sometimes our simplified version of reality can become the environment that the agents live in and that simplification can start reinforcing itself. Like in a hypothetical example, a customer might get labeled through the process of, you know, kind of reducing their behavior to how they respond in certain environments unlikely to renew. And then the system routes that customer into a lower touch service profile and experience that has fewer conversations and less attention and less help, and then the customer leaves.

And then the dashboard says, hey, we predicted it.

But did we? Did we predict it? Or did we make it happen? Did we make it come true?

You know, imagine trying to explain that to the customer like, well, we knew you were gonna leave. Like, didn't you? Did you? Because I left after you stopped answering my my responses, my questions.

Like, yeah. Very predictive of us. Right? I just think we need to be very mindful about that.

A system can be consistent and still consistently wrong about what matters.

This is where the designing without a designer gets a little more complicated cause nobody is designing the whole pattern. Someone does choose the labels and someone does define success and someone does decide what consequences will appear in the report.

The behavior can be emergent, but we're still responsible for it.

Right? Now, it's at this point in the conversation that usually someone raises their hand and says, yeah, that's why we need a human in the loop. And I don't disagree with you, but I have a few follow-up questions.

Which human in which loop?

With how much time? And what are they allowed to change?

Those are important follow-up questions to this because there's a version of human oversight that looks a little something like this. A recommendation appears, and the person has 47 other things to do, and the recommendation looks plausible, so they click approve.

Right? That is a finger in the loop. That is not a human in the loop. And it's definitely not judgment in the loop. Right? A person can be present without being meaningfully powerful.

If I can approve or reject a recommendation but I cannot question the categories or the goal or the missing information, then my oversight has a ceiling.

If I can question the answer but I cannot question the question, that is not meaningful judgment in the loop.

Meaningful oversight has to include the power to say, we may just be solving the wrong problem here.

So if solutions can emerge, if we can see emergent behavior and emergent solutions then so must the oversights that help us, or sorry the insights, the human insights must be able to be emergent too.

That we can recognize when things need to change.

If agentic processes take shape... Let work take shape iteratively, then the human insight has to be able to take shape iteratively too. I am stripping over my words here. So I hope that makes sense to you.

We have to be honest about the incentives. If we tell people use your judgment but when then we penalize them every time their judgment slows down the processes, What are we really asking them to do? We're asking them to approve faster.

We're asking them to be a faster finger in the loop.

And a finger can't question the goal or the categories or what's missing.

And the whole point is if we want humans to be in the loop, then we have to have the option to inject meaning back into the loop.

And that means the opportunity is to put human attention where it matters most. Right? Where consequences are significant. Where uncertainty is high.

Where a decision is gonna be difficult to reverse. One of the things I often advise my executive clients when they wanna know where to start with AI is start with something you're not gonna regret.

So if we know that we have got the opportunity to human oversight where things are gonna be difficult to reverse, absolutely. That's where we ought to put it.

Automate the routine where it makes sense. But who gets to decide what good means is another question that often comes up in this and it's an important question. Because within a business some of that is clear. We have authority, we have leaders and we have teams and responsibilities, but decisions also touch people who didn't choose our objectives.

Right? Customers, employees, applicants, or communities who are downstream of the solutions we offer and the products we create. So being able to optimize something doesn't automatically give us the right to define it for everyone else.

And that's a really hard thing for us to grapple with, but fairness is a useful example of this for us to see how this plays out. Because treat everyone the same sounds like it's a fair thing, and sometimes it is, but if you sometimes it preserves a disadvantage that was already there. Like, think about asking two people to use the same staircase when one needs a ramp. Those are identical instructions but very different access.

If you bring it back to the workplace, if a measure of responsiveness assumes that everyone can respond at every hour, what is that missing? It's missing caregiving or disability or time zones or fact that a colleague is inconveniently a person with a life. If we turn that into a durable label that exists within our systems like less committed, then it's not that we've discovered a lack of commitment.

It may be that we've encoded a narrow idea of what commitment looks like.

So it's really important for us to understand that fairness is not always the same input. Sometimes it's a better understanding of the circumstances.

And that doesn't mean that one person can define fairness once and write it into a system and declare the conversation over because reasonable people are going to disagree about trade offs. History does matter here and different experiences are going to reveal different sort of blind spots to this. And some of the questions belong within public institutions that need to be argued for at scale. They need to be in professional standards or in law, not just in a company's settings panel.

So within our own though, when we bring that back, can start by making sure that affected people have more than a feedback box.

That they can challenge a decision and that their challenge can actually potentially change a rule.

Because we don't wanna be merely collecting their frustration to improve the interface. That's the last thing we wanna be doing. We wanna make sure that people who are affected by a system have some power to change the terms, not just explain the damage.

That's a bigger responsibility than just accuracy. It's the responsibility of making decisions that remain answerable to people.

I wanna take this from a fish to a drop down to the foundations of legitimate decision making. I mean, it's just a regular Wednesday afternoon. Right? That's what we do here in the Work Innovation Summit. But let's bring us back to something that you can use and you can apply. So if you'll think of just one workflow, not your entire AI strategy, but maybe think about one workflow that you might like to revisit and talk about with your team and how you're going to make it more human and more ready for this human agent collaboration.

Where agents are doing or could soon do meaningful work. What I would suggest that you do is ask three questions about purpose, about context, and about permissions. So as far as purpose, I would say, ask what human outcome are we trying to improve? Not just what task are we trying to complete. Right? If the task is resolving an IT ticket, the human outcome might helping a colleague get back to useful work.

Or if the task is preparing a client update, the human outcome might be helping that client understand the decision and helping the team to make it and trust it.

Right? Those are purposeful human outcomes that you can articulate. But a useful test here would be like if our metric improved but people's experience got worse, would we notice?

And if the answer is no, we need another signal.

For context, what does the agent need to know to act appropriately? And what parts of it, of what it knows, should remain open to revision?

Cause everybody has this spreadsheet column that has existed forever that nobody knows why it's there, but nobody dares to delete. Right? We all know this experience.

I just wanna make sure that we don't give that column the power to manage people.

You know what I'm saying?

What we can do is keep that information handy. Keep the source, keep the date, keep the uncertainty, keep the reason an exception exists. That's all fine. But don't let a temporary workaround harden into an eternal truth.

And as far as permission, what can the agent do and what requires human judgment?

Right? Who has the permission to challenge the system itself? And that last question is one I feel like we often forget because we very often define who can approve the output, but we don't always define who can say this objective no longer seems to serve us.

Right? And who is going to be heard when they say that?

So purpose, context, and permission then make those answers visible enough that the team can use them.

This isn't gonna fix everything. I think it's just a promise. It's a practical starting point for for better human agent collaboration. It's not a promise that that nothing's gonna go wrong. It's just a way to notice, to learn, and to respond when it does.

And and it matters for value and for ROI too because if an agent saves time in one step but creates confusion and distrust and rework in the next one, then we have not necessarily made the work any better. Right? We've moved the cost somewhere that our dashboard isn't looking.

So we need to be having this conversation about holistic systems. The goal is not about making one task faster, it's to make the whole outcome better.

I have one more capability that I would invite us to consider the value of and that is reconsideration.

Right? We can get better at noticing when our assumptions stop working. We can preserve options.

We can build processes that let new information change the decision.

And that's not indecision. It's not about endless debate. It's the ability to recognize that a rule which once helped us is now getting in the way.

In a changing environment, it's really not a defect. It's part of being ready for the future. I talk a lot about future readiness throughout my work. Future readiness is really not about predictions. I know this is... I am called a futurist in my work a lot and I think it's such a misleading label because I don't give a hoot about predictions, I care about preparedness.

What I want is for you to be concerned with preparedness, for you to be thinking about how you can be more resilient, for you to be thinking about how you can be prepared in advance for not having every answer in advance, but being able to respond when a better question arrives.

It's a very key difference.

And I think it's also really important that we understand that a healthy organization can remember and revise. It can say this is what we believed then and here is what we have learned since that.

It can keep the history without becoming trapped by it. And it can build human checkpoints that aren't just speed bumps. They are places where meaning can reenter the work.

It looks like sometimes we are saying like, hey we are measuring the wrong thing or that person didn't have the opportunity. We assume that they did or wait maybe this looks efficient because we are not counting the consequences. I have lived through that example. Like, we have plenty of these kinds of opportunities and they not... They are not necessarily failures of collaboration or of deliberation. They are they are opportunities for humans to bring the most meaningful contribution and that brings us right back to us, to humans. Because at the beginning of this discussion, I asked you to consider what it is that makes us human and we talked about that and you know what we're talking about there. What I would love to invite you to consider is that our future depends not on defining every task as something only humans can do because that's inevitably a moving target.

It's not a very generous way to understand ourselves either.

We are so much more than the work machines have not learned to do yet.

And I think we need to sit with the realization that the more useful question is what responsibilities we want to keep even as machines get better and better and capable of more. Like the responsibility to define what matters.

To notice whose experience is missing from consideration.

And to question an objective that's producing the wrong consequences.

And to change our minds when the evidence and the people effective give us the reason.

Those are not leftovers from what machines can do. That's actual leadership.

And it's something only really humans can do. But again, this is gonna bring me right back around to the fish. Like, what? What is it with the fish?

The thing about the fish is that fish evolve, and natural selection doesn't owe the fish an explanation for why.

But systems that shape human opportunities should be held to a different standard.

We can build agents that take on more of the work. We can give them better context, clearer permissions, more useful ways to coordinate. We can free people from work that is repetitive, fragmented, and exhausting. That is totally worth doing and important work.

And we can do it without pretending that faster automatically means wiser.

We can do it without letting autonomy become opacity.

And we can do it without turning a rough model of a person into the last word about that person.

And we would have to do that by asking some questions like what is getting rewarded?

What is getting remembered?

Who gets heard?

And who can change the rules?

Those are the questions I would invite you to carry out of this room and back to your teams and to constantly, constantly, constantly bring back into the conversation.

Because it's not that we have to design every single move, every single action that eventually emerges, that it's it's going to be incredible that design without a designer happens. It's extraordinary and it must not mean that nobody takes responsibility for the outcomes. We don't have to design everything, but we do have to take responsibility for the outcome.

So what matters next is not just that we know what our agents can do, it's what we make possible for people and what we refuse to stop caring about along the way.

Thank you so much for having me here with you today. It's been an honor.