Mastering AI Tokenomics | WP Intelligence Briefing transcript

Good morning and welcome to WP Intelligence weekly Friday briefing. I'm Yunhi Kim, deputy editor of WP Intelligence. Today we're going to dive into a critical but often overlooked question about artificial intelligence. What does AI actually cost to use? Many companies and executives still cannot confidently answer who owns the AI bill or what it will cost 3 months from now or a year from now. And yet they continue to spend heavily to adopt AI because of competition.

The cost of AI measured in tokens can rise with every query, document, image, and task. In this live briefing, we'll examine AI tokconomics and explore how businesses and governments can get more value from AI without breaking the budget. Before we begin, a quick note about WP intelligence. Our coverage is built for executives and policy makers who need insight to help them see around corners of industry and policy trends.

Our subscribers receive concise intelligence reports with actionable recommendations across four verticals, AI and tech, global security, energy and climate and healthcare. We also have deep dive data graphics as well as exclusive interviews and live briefings like this where we share um insights with policy makers and business leaders. So today we have three expert guests to help us understand the economics of using AI.

Dan Gallagher is WP Intelligence lead AI and tech analyst based in San Francisco. He covers the companies, markets, and technologies driving the AI economy. Welcome, Dan. >> Thanks for having me. Good to be here. >> James Milin is CEO of Work Helix, which helps organizations identify where AI can create the most value and and measure the return of their AI investments. Based in San Francisco, James has also worked at Google and Amazon Web Services.

Great to have you, James. >> Yi, pleasure to be here. Thank you. >> And Dan Diazio is global consulting AI leader at EY where he supports clients with AI enabled business transformation and strategic direction. Thank you all for being here today. >> Great to be here. >> So, here's here's how we're going to organize our conversation for the next hour. First, we'll explain what's driving token prices and why AI tokconomics should be on every executive's radar.

Second, we'll dive into how to better budget for AI and determine whether you're getting a real return on your AI spending. And third, we'll examine getting the most out of AI agents. I want to also thank you to our audience for the questions that you've sent in prior to this briefing. We've used your questions to shape this conversation today. And um just a reminder that you can still ask questions through our chat function today.

So if you have any questions, drop them in and we'll take relevant questions as we go along. So let's get started. Dan Gallagher, in your recent report titled AI tokconomics using the technology without busting the budget, you say that token maxing is short-lived. explain why that is and what it means. >> Well, sure. Um, so token maxing uh was a was this bit of a trend that emerged earlier this year because you know so for context token uh you know most AI services charge by tokens at least those are served to the enterprise.

Consumers mostly pay a monthly fee to claude or chat. Um you know for basically unlimited usage aside from a few exceptions but but for business use cases most deal enterprise deals are priced based on consumption and that's measured in tokens. Um what happened earlier this year, it was in March at uh Nvidia's GTC conference is that uh Jensen Wong, Nvidia's CEO, um you know, got up and essentially said, you know, engineers should be really using generating a lot of tokens, using a lot of tokens in a in a podcast a couple days later drew this comparison that he'd be deeply concerned that if an engineer that was making, let's say, $500,000 a year in salary, um he'd be deeply worried if they weren't also using about $250,000 your tokens.

Um, and so that message from somebody who's obviously a very influential voice in AI uh really kind of generated this, you know, um, uptick in companies and workers like trying to maximize their token use. Um, and the for context, this is when, you know, companies are trying to be really aggressive in how they adopt AI. Um, workers who might be concerned of like losing their jobs if they don't show that they're aggressively using AI. uh that create another thing.

So you had some companies actually recreating creating leaderboards to measure token usage. Um and what happened is that really quickly it became apparent that token usage by itself doesn't necessarily create economic value for a business especially if you have this and if that's a point of measure you incentivize the wrong kind of behavior. um because you might have you know people throwing all kinds of stuff into into AI models just to boost their token count but it might not be relevant to the business or helpful or lead to a business outcome.

Um so within it was even just a few months of that of that you know essentially like from March to I think May or June um you kind of had this token maxing fade out because all of a sudden a lot of companies were realizing seeing the bills that were hitting them. Uh we had a case where um Uber talked about blowing through their entire token budget um even before the half the year was done and that was an annual budget.

Um and you had companies like uh Meta and Amazon even and these are companies that have a lot of resources, a lot of cash, you know, essentially sending out memos to their employees saying they slow down. Not all not all AI use, not all token use is actually useful. So you that that died down pretty quickly to the point where companies are realizing that okay if you're we want to use AI use it aggressively but it needs to produce actual outcomes and not just run up the bill >> right and also we see that companies are not slowing down their spending on AI we actually kind of have a chart from your report um can you can we pull it up and um Dan why don't you explain what our viewers ers are seeing right now. >> Well, this what we're seeing here is this kind of shows the um the drive AI AI companies are going to have to produce, you know, essentially the revenue they need to produce because so much money is being invested in building up AI services.

This is the this is the total of like capital spending um you know in the US. It's it's going to exceed a trillion dollars. um uh at this year I believe um or next depending on how you measure that um the capital spending for things like data centers chips you know the things that like essentially are powering AI there's so much money flowing into that you can see from the chart how much has gone up you know just over the past you know two years um and what this creates is that um the companies powering AI the companies like Anthropic and OpenAI and their cloud partners the companies like Amazon and Google and Microsoft and Oracle and these um really have a strong interest in you know generating revenue and returns from this business because so much money is being poured into it in the front end.

So um they have a drive to like you know so the more companies essentially the more tokens a company uses the more revenue they're going to be the more the more they're going to be paying. Um, and so we're kind of what we're seeing now is like why the AI industry really needs to try to maximize its returns really quickly because um, that is a lot of money going in. >> And Dan, you also point out in your recent report that a lot of companies are actually using frontier models to the point where it's actually overkill.

And you talked about the Uber example, but um what is your recommendation in terms of AI usage if companies aren't using frontier models? >> Well, I think I think you're seeing that like not all AI uses AI uses within a certain company need the same model. Um you know, so there's a lot of employees, let's say that you are using AI to help them draft emails, you know, to colleagues or or customers. you know, you don't need it.

In most cases, you don't really need a top tier uh model, frontier model to do that. That's overkill. Um you might be able to get by with a um less powerful model, maybe from the same companies. Um if you use company, if you use services already like, you know, chat, if you have subscriptions like Chad or Claude, um I think what you'll notice is that the default setting is usually at like a mid-tier model anyway. um and that you have to kind of select one of the more powerful ones where you'll face some like maybe usage limits.

And the whole point of that is that is and it helps send home the message that like okay for certain use cases you want those most most powerful models. Um but there's a lot of use cases within a company where you just don't need that. And so finding essentially matching the right model with the right use case is the real trick now. And that's the way you maxim essentially make your cost the most efficient. >> That makes sense.

I want to turn to Dan Diazio from EY. Dan, you work with many clients to help transform their operations into more AI enabled organizations. What's the biggest mistake companies are making right now with their AI usage? And do you agree that token maxing is dead? Uh so first great to be here. Uh I would I would say I don't really see token maxing um as an indicator for performance or for success and and as a derivative of value be their prevailing theme anymore.

Um to the point that Dan mentioned you know at that time when token maxing was really picking off most enterprises were focused on driving adoption. You know, like the idea was that the organizations that can empower their workers and adopt AI prevailing across the organization were those that were going to be successful, but they were paying a subscription fee. When that turned over to consumption, I think lis the latest study that we had, 88% of employees are using AI at work, but only one in 10 companies can point to where that is driving material value inside the P&L.

So the shift with this new line item called tokens um you know token spend that increased the more you were adopting these tools real questions started to be asked by CFOs and boards of what is the return on investment that we're getting from using AI that offsets this new this new line item of spend that we didn't have six months ago. And I'd say one of the challenges that many organizations are facing today is that most of their business cases didn't include this new token consumption cost in the formulation of where they were going to get value.

And there's needed to be a revisit of the scorecard if you like of where AI is driving value across organizations looking at the total cost of AI which is actually not just tokens but there are other costs you know we've mapped it out at EY to say there are six costs currently and there's one potential new cost that might be coming down the line that that really compute the fully loaded total cost and that's really important to make sure that companies are able to answer that big question around return on investment.

As Dan had said, I think uh consumption is no longer it never really was the indicator for value, but now companies are having to sharpen their their their economic review of their AI programs to to make sure they're investing in the right things and to control some of the spend that isn't necessarily lined up to be able to create strategic value. But to your point, that's not necessarily cutting off these budgets. I think it's a redirecting and a better discipline of where companies and how companies should be investing that money. >> And James, I know that work Helix um you help organizations like Autodesk and NASDAQ um take a more of a taskbased approach to measuring ROI from AI rather than token usage.

Can you talk about that approach and why you think that's more important? >> Absolutely. We're discussing this really important breakthrough where instead of just measuring usage, we can get at the substance of what the work is. And fundamentally the way we think about it is jobs the right unit of analysis are tasks because jobs are bundles of tasks. And if a job is a bundles of task, that means an entire organization and the society we function in is this big bundle of work units of tasks.

And each of them are not created equally from a value and cost perspective. How do we know that? If we look at the balance sheet of an Autodesk or a NASDAQ, we can see that the wages aren't distributed equally. People pay companies pay different wages for a software engineer or someone in HR and we can look at that and say the cost of these tasks aren't equal. So to give a specific example, if we have someone in who does sales, some of their tasks might be highly suitable and high value for AI, for instance, summarizing a document.

But if the task is um to go write a 20page brief, that could be a higher value task. So fundamentally, we we think the way to take this is to take the jobs, break them down into bundles of tasks, look at the wage and value of each of those tasks, and tie that to token consumption. >> And um Dan, let's talk Dan Gallagher, let's talk a little bit about the outlook for token prices. Do you do you expect prices to keep falling or could demand for computing power, energy, and advanced chips put a floor under those declines?

Uh I I think it's a little bit complex because I think overall we've seen token costs go down like the uh some data I cited my report from from ramp uh found like from in you know and I when I wrote my report I had some pricing data as of August and it it had fallen essentially by half from the same period a year ago. Um this is the average token cost. So that's a blend of averages, but what we have now is an ecosystem where there's a a big range of of models to choose from.

And some of those some of those are at the high end. They're expensive to use. And how how they and what's going to happen is I think over time as models get developed um the most advanced ones are always going to be more expensive than the like mid-tier ones, let's say. Um, so you're going to have you'll you'll have a mix of things and the the complexity comes from the fact that token costs may be coming down, but as companies are having more employees use more AI, you have this thing, you know, I think we referred to it in past briefings, uh, the Jevans paradox, the economic principle that the as something gets cheaper, usage of it jumps.

And so as as token prices overall get a little bit cheaper and then more companies are incentivizing more more employees to use AI and hopefully use it wisely and not just like run up token costs. But um but if you but if you have a company where like you know let's say 20% of the workers were um using AI in the first half of this year and you that number comes up to 40% um in the second half because the company's done a better job at deploying rolling out capabilities.

You're just going to see that go up. a company's AI bill is going to go up um even if the token unit costs are coming down. So I think you have both those impacts going on there. So you'll see I think you'll see yeah token prices will probably keep coming down on average. Um but that doesn't mean AI bills are going to go down same way. >> We we actually have an audience question and the question is are there ways to use formulas to manage token limits? happy to discuss that.

Um, you know, so it's the classic measurement from an economics perspective of is total factor productivity where we take the total cost of uh capital and divide it by the labor costs. And right now we have a different way to calculate labor costs which is to actually look at the task people are doing. And when I think of AWS and cloud services, one of the big breakthroughs was not just the amazing work the technologists had done.

It was the pricing model. And what we've seen is historically in order to purchase those types of services, you'd have to work through finance and go get a multi-million dollar purchase order and review it and write a business case to go buy it. But now with these AI services, any developer can put in a credit card and all of a sudden start get a massive bill. So the way to really budget for that is to start at the work and say if we have a finance department, what are the outputs that we're expecting and how do we currently price those?

If we can say this finance department, they do things like close books, they do bank reconciliation, let's price that work and then see what we'd be willing to pay to augment that from an AI perspective. So it's starting fundamentally at the work being done, pricing what we pay for that and then comparing it. And we just see this as an incredibly complimementaryary technology to a lot of the work people are doing today.

And just like Dan was saying in Javon's paradox, there's no shortage of an expansion of labor opportunity. I've never met an a CEO of a large organization that says once this work's done, we'll have no more work. It's just shifting a lot more of this work from um what humans are doing that may be monotonous and helping and bringing in these AI systems to help uh complete it. >> Dan Dazio, do you have something to add? >> Yeah, absolutely.

So, you know, often I get into a room with uh a number of executives and we'll put a sentence on the board and we'll ask how many tokens are in this sentence and um and the the the range in some of the answers that we get is quite variable because we built this pricing model on something that's not very intuitive. I mean a a token is a a word fragment in a way and often one word is approximated to about 1.3 tokens uh because these models are all built based on uh based on language.

So uh you know I think I think even just starting at what a token is is a helpful exercise inside many organizations because they've been become fluent in talking about token limits but often understanding exactly what a token is is not is not so intuitive. Um to to supplement what James said, um I would say that in practice I see organizations taking a less than formulatic uh formulaic approach towards setting limits and shifting this like this idea that we're going to use AI to replace work instead to uh think about what we can do to augment and uh augment an employee to be able to do something that necessarily wasn't possible in the past.

And and that often means that you look inside the company to find where those big areas of growth like I often go through this thought exercise. If we could add a thousand people to to drive revenue tomorrow, where would we put them? And that might be an opportunity to figure out where and how we can power some people with AI to be able to do something entirely different. But what that means often uh let's take a pharmaceutical example for instance that might mean that the R&D teams are equipped with the frontier models to you know and with large budgets because a discovery or an enhancement to accelerate um uh a new a new uh compound or a new protein could be extremely valuable versus uh versus other departments. finance department, the human resources department where they probably don't need a frontier model and they're given a less than frontier model with a smaller budget to be able to do their work like that.

That seems to be where I see many enterprises uh today not setting one budget for the organization but starting to allocate that to where there are areas of high computation and also areas of potential growth for the company. How do you actually kind of judge that though? Like some people are having a hard time figuring out for for a particular task this model would be best or for another task this model would be better because there are so many AI models coming out so quickly.

I think it's very difficult for business leaders to judge. Any tips on that >> today? this. >> Sure. Yeah. One of the analogies I I think about today is it's as if we're going to the gas station and I have a car that doesn't require premium gas and all of a sudden I'm putting in premium gas in my car except this is about a hundred times the cost. These frontier tokens are far more expensive. And what we can do to build on on that example of within R&D is actually even break the types of folks in R&D.

We might have quantitative research scientists who are sending queries to crack a next big computational chemistry problem. Those folks should absolutely be having access to the frontier tokens where needed. Um within that same R&D department, there could be tests questions such as I need to do a procurement task. That task shouldn't be assigned just as an expensive token. I think that's why you're seeing such a takeoff in these model router capabilities.

But the model routers today, even the ones that have jaw-dropping acquisition prices, are are still relatively rudimentary where they're just saying, is this API available or here's an overall budget. So, there's a huge need to actually get more granular and look at the contents of each query and figure out where it should be routed from a business value perspective. We have another audience question and the question is how are leading organizations forecasting the value of their AI use cases.

Um, I'll jump in real quick and I'll let the others above in one thing. Um, but it was interesting when I was doing the reporting for my for my for my report. There there was a case where, um, Airbnb talked about, um, in their latest earnings call uh, made a point of talking about how they were able to really improve internal operations and put out new features with the help of AI. and their their CTO um put up a blog post uh I think a day or two after um where he kind of made a specific mention that instead of like measuring based on like consumption or use they were making a measurement based on what they were shipping.

So was your AI use leading to something that was a product or feature that was actually getting out the door? Um, and I think that's a way where I see more businesses trying to look at like how they're going to measure the effectiveness of what they do because at the end of the day, um, this the AI use needs to needs to result in an actual business outcome instead of just use for its own sake. Um, so I think that's at least one example of a of a way one company is trying to u look at like how how its AI use is viewing.

And I'll throw this to Jins and Dan for uh further other examples. Yeah, I would I would say that um the last couple of years many organizations have gone and looked for where inside their organization AI can be helpful and that often might mean taking a process identifying where inside that process maybe step two or step seven um where AI can really speed that up or make something a lot better and take work out of that process.

But then the work often flows that it's a person engaging, doing some work, passing something to an agent, getting something back, and then proceeding forward. And it's not really reinventing the work. It's mostly just speeding up aspects of the work. And I think in that area it's become increasingly difficult to be able to I mean there's of course ex exceptions to this but it's becoming increasingly difficult to be able to move some of the P&L you know to have such material impacts on the flow of the work that it that it starts to hit the financial statements.

So we see an alternative approach being taken now something that we often refer to as moving from bolting on AI to building AI into the workflow which is to start to have agents. You know maybe the work go starts from a person and then an agent does some work and passes to another agent and passes to another agent and they complete a bunch of the task and in a lot of that sense you move away from the current way of working and invent an entirely new flow that didn't that didn't exist in the past.

When you start look for looking for value instead of just looking for what the AI can do, this is essentially moving from a technology implementation to a business transformation. And then you start to think about entirely different outcomes that can happen with inside functions or different cost footprints that it takes to be able to run functions. like the the AI value story it in in my observation has shifted from trying to get value out of uh trying to get value that you can take to the P&L of hundreds of initiatives and moving to a couple of strategic things that will substantially change the company and make them more AI native in their operation and that that becomes an exercise of transformation measurement as opposed to just looking at where and how you can get the benefits measured out of your AI adoption.

So just on that point, um, Dan Diazio, if you were advising a company at the beginning of its AI journey, how would you determine which tasks are worth spending the money and which ones to drop? >> Yeah. I think what we often do is say that companies simultaneously need to be able to take two motions on this journey. One is to be able to go broad across the organization and empower the workforce so they can innovate from the bottoms up.

And this means selecting the right tools and the right token budgets for each of those departments with the way that James and I were referring to earlier figuring out what are those most value creative ways uh where they should get deeper budgets. But there's this broad initiative that we're going to empower the workforce and we're going to allow people to to be able to think about their work differently. And this is really important because everybody has a phone.

Everybody uses AI in their personal lives or most people use it in their personal lives. And to say they can't use it at work sends a really direct message to employees on the culture and how much they're going to be a part of the organization's future. So there's there's one part which is going broad but the other area that we advise is to pick a couple of areas that you can strategically go deep on and you shift from thinking about things inside the the the org chart of today and look for value streams of where and how the company creates uh revenue in a lot of ways and start to identify those areas that you can address more in a top- down way to reinvent the flow of the work that often looks like some sort of um um like a a share a total shareholder return exercise or looking at the financials and starting to map out the way private equity often would inside of a company where the opportunities are for growth inside the business.

Um, and that top down exercise is where we're starting to see companies put and get a lot more benefit in in being able to articulate a story to their investors and to their to their uh to their managers. >> James, do you have thoughts on this? >> Sure. I think there's a lot of focus on the P&L that people are talking about looking for immediate impact. uh my co-founder Eric Bolson who runs the digital economy lab at Stanford has been talking about how this whole field has gone really mismeasured for a long time.

Here's a specific example. If we think about Wikipedia, which a lot of people would ascribe has been valuable to them in different periods of their life, the GDP contribution of Wikipedia is almost zero. It's a free product, nearly a free product. No one's paying for it. So if we're just looking as a business and saying where's the P&L impact, we're missing an interstitial measure and this gets to this fundamental view of something called the J curve in economics where traditionally when new technology comes in it actually may decrease productivity in certain ways.

And if we're just measuring and staring at the P&L, we're looking at the wrong tachometer. It doesn't mean that AI can't of course drive those impacts, but that is a lagging indicator. Leading indicator we think is high quality and high quantity usage of AI. So if I'm using Frontier tokens to solve some pressing chemistry problem and I can actually demonstrate that. Fantastic. And what's interesting is when we look inside of these massive Fortune 500 company or and organizations, we've looked at millions of queries, we see this consistent power law distribution.

What does that mean? It it's not a bell curve. If we say what's the value everyone's creating, it's not that there's this mean here. What happens is about 1% of the users are driving 20% of the total volume and the top 10% are driving 68% of the value and about 23% of employees are never even touching the sanctioned AI tool. So there's a massive disconnect. If you sit in a corporation and look around at a room of a hundred people, even if they have the exact same job title, their AI usage is wildly different.

So that is even the big gap within a particular department. Another comment to pick up on is this idea of where do we identify these opportunities? Uh open AI actually had that same question when they were creating GPT core and said how is this going to impact the labor economy? So they reached out to our team and we published a piece of work that's now in science and the title is generative pre-trained transformers GPTs are general purpose technologies.

We think of AI as electricity, as a general purpose technology that can impact almost any part of a company. So the hard part then is how do you really quantitatively and rigorously identify where are those pockets of value? And we think the right way to do that is take all the jobs, look at the tasks, and ask a simple question for each task. And the question is, can this task with a human be done at least twice as fast without a reduction in quality?

And we've published that work and shown and this is a few years ago some of the most exposed sectors of the economy and now that we have these big uh foundational model labs publishing reports on where the usage is so far that's lined up uh pretty accurately so far. >> I want to turn to um Dan Gallagher with AI bills kind of soaring um to what extent are open weight models helpful there in lowering the bill? Well, uh, well, I mean, helpful in the sense that they're being used a lot, uh, by companies who are trying to figure out how to make, um, how to essentially use something that's powerful and cost less.

Uh, because those models do cost less. Uh, they're open weight, they can be tweaked. Um, I think and I think there's some dynamics of these are a lot of times coming from Chinese companies that might be enjoying some state support and not have the same necessarily cost and return pressures US companies might be facing right now. So I think you know the data I've seen has shown that there's a lot of usage among US companies of Chinese models specifically for that reason because they are cheaper.

Um now how that's going to play out longer term is a good question because um I think among US companies you might they're going to be um especially as companies like OpenAI and Anthropic look to go public and look to like um you know show good usage numbers. You know that's going to be I think maybe a little bit deflationary on some of their cost bills. I think they'll try to get as much as they can for their their most advanced models.

Um but at the same time they don't want to price themselves out of competition with the Chinese models. Um and I do want to go back to one quick point here too. I think um there's anch dynamic at play relative to what James was just talking about is companies are trying to figure out how to you know essentially um make make their spending the most efficient. Um, one point of pressure that I'm that I I wonder about now is that um, if you have a diversity even within the same a group of the same employees in a company where you know some people are using AI super aggressively have figured it out and others haven't at all and you have a you know obviously a spectrum in between.

Um, how do you incentivize the more hesitant people to start using it? they might be worried that are they going to, you know, essentially find their way out of a job because they've they've figured out how to use AI and AI can replace them or is there there's some other fear. Um I wonder if there's room in AI budgets, there needs to be also some room for essentially experimentation because a lot a lot of these models when you apply them to your specific job, there's not necessarily a manual out there on how to do that.

There's going to be trial and error figuring things out which is going to going to cost something. And I'm wondering how companies are thinking about that and b, you know, trying to strike that balance between, you know, driving more getting getting more people using it um and letting people experiment and sometimes fail in those experiments while at the same time keeping control of the budget. >> Yeah. James, do you have um any kind of concrete examples working with Fortune 500 companies on that front?

Yeah, my my take is it's less of a budget problem and more of a training problem. Uh, and just like we've seen with other ways of technology. Uh, if you buy a license from a frontier model, it doesn't come with an instruction manual. Doesn't say here's how to use it. That hard work has to be done by these massive corporations who are very busy doing many things. And the most successful firms we've seen are not the ones that say, "I'm going to bang my fist on the table and demand that more AI is being used." You know, when I was at Amazon and Google, no one was saying, "James, did you use the internet today?" It's just not a helpful view.

What is incredibly helpful are the organizations who are saying, "Let's find the person who is in that 1% of an actual power user who's doing high quality and quantity work. Let's understand what they're doing." And they might say, instead of manually writing a territory plan as a sales rep for a pharmaceutical that used to take me three days, I now feed in all this information. I make a project. I manually review the plan and I'm done in an hour.

Great. Can you run a workshop on that? Can you run a training on it? The tough thing is identifying these power users. It's frequently not the person who's the loudest and saying, "I've really figured out AI and I know how to do this." It's you have to actually look and understand the work products they're completing. See if those are high quality. If they are, great. Let's have that person dissipate. Um, you know, in previous waves like the industrial revolution, it took decades to actually get this dissipation.

So, even these models, which are the least intelligent models we will ever have access to in our lifetime, if there was no further progress, there might be decades still before this gets adopted. capabilities are the the scientific capabilities of these AI models are far in excess of what they're needed to do to complete so much corporate work today and to augment that work. And and actually in our analysis, we found about half of jobs can have uh about 20% of jobs have half their tasks that could be completed at least twice as fast with an AI system in a human today. >> It's very interesting.

I want to shift gears to talk a little bit about financial governance. And um this question is for Dan Diazio. How do you impose financial discipline without slowing experimentation? Um I think what I see organizations if they take this approach of going broad and going deep and the going broad starts to do a lot of that training um that James was previously referring to um you know what I've seen many companies start to do is start to look at setting some sort of limits in place where they're not necessarily blocking usage but they're requireing hiring exceptions to use above a certain quota or a certain budget with some of these tools.

And that that kind of helps do some of what James is referring to of identifying who are some of those people that are really driving um a lot of innovation inside the company. Because if you have a list of your top 10 users in a um top 10 consumers inside of a inside of a business function, my guess is in that list there's a mix of people that are probably using the wrong tool or the wrong model for the job at hand that with a little bit of training, they'll probably not show up on that list anymore.

And then there's some people that are doing some really incredibly valuable work and you don't want to slow them down. So this is where if you have a uh a budget that you put in place uh you know for different departments inside the organization and then you have a governance process that when you start to bump up against that budget then you go for approval and there's usually a quick discussion or uh a form you fill out to be able to show what you've been doing with the tool that should get you an additional aotment.

So, so you start to put some of those training opportunities uh better in place. I I also see many companies starting to look at their holistic, you know, as they look at their business cases that they've started to drive and they factor in the total cost, they're starting to look for a lot of alternatives in ways of driving down their total spend. If you leave it to a person to say, would you rather use a more intelligent model or a less intelligent model, most people are going to pick the more intelligent model.

So what Dan Gallagher was referring to earlier is maybe we ask a system to make that decision for them by putting a router in between the models that they select. In many cases we see companies starting to in you know introduce open-source or open weights models and fine-tune that model with their own intelligence to be able to take the cost down even lower. We've seen companies that are very highlevel users of AI start to think about how they might be able to bring some of the infrastructure on site to be able to run it and get some arbitrage uh in that in that capacity as well.

So I think there's a variety of different commercial strategies and technical strategies that companies are deploying to be able to manage their budgets. But I think categorically it's not about stopping the spend. It's about just making sure that you're learning from what is effective spend and and funneling more of the tone kit consumption to the stuff that is going to help do work better, add value, create growth as opposed to uh just maybe maybe some of the of the training opportunities that are often driving a lot of these budgets today.

And James, do you have any thoughts on what a good financial governor governance looks like? >> It's a it's a really important question and to me it goes back to the outcomes. From an economics review, it's always what are the inputs? What are the outputs we care about? And if we can define the outputs very clearly, then we can start to figure out what the right budget allocation is. Uh there's been all these tests when a new frontier model comes out where they give it an exam and say, "How well does it do on the LSAT or the GMAT that tell us about how effective it is at work?" Almost nothing.

Those aren't jobs that people have at Fortune 500 companies are to take the LSAT or the GMAT. What they do is they have tasks that need to be accomplished. Uh, one of the only randomized control studies I've seen was this great piece of work uh called Gen AI at work at a Fortune 500 company. They took a call center and they did a randomized control trial where they decided who got access to a Genai co-pilot. This wasn't replacing the call center employees.

It was giving the call center employees a chatbot that they could work with to help answer and they we measured the KPIs and looked and said what was the change in customer satisfaction? What was the change in case resolution per hour? Uh what was the decrease in expletives that people sent because they were frustrated. We saw people who used the chatbot actually had less people typing in all caps and being angry with them.

But the two most shocking things we saw is there was a massive decrease in churn. So one of the biggest KPIs that a call center cared about was how many people would quit the call center because they're frustrated with the work or on a different job. and the folks who were junior and immediately got access to these AI co-pilots, their churn rate dropped off a cliff because they were happier with the role. So to me, that's a great example of if you're set up a scheme to measure the right KPIs that really matter for that function and start to correlate it and ideally causally tie it to AI usage, now you can really effectively allocate budget.

So while the technology is different, I think it's the same playbook that we've seen for other disruptive technologies, which is each function has a set of KPIs that they're measuring. Let's tie that to usage. And because there's a whole breakthrough in in econometrics called the credibility revolution, companies can actually causally tie that and figure out what's happening. And that to me is the most effective way to do this um is to get that measurement in place.

Can can I add can I add one more bit to it just from an organizational perspective? um we looked into his you know we we went back and looked to see where these new costs start to dry start to appear on on companies P&Ls and about a hundred years ago in the automotive industry most of the hiring was done by foremen on the line where they would go pick who was going to work at that point in time um and they would lay people off as they needed and hire new people but during some of the labor unions and uh the the depression all that hiring and firing cost a lot of money and they realized they were hiring far too many people for the number of jobs they had.

So the organization looked at it and said actually these are new costs that we're incurring because we're needing to pay for people after we've after we fired them. And as a result that gave the the groundwork for the modern human resources department and a similar scenario happened in the 70s where foreign exchange rates were pegged to the dollar. It was a task by an accountant. um when that became unpegged the treasury management function was too important to leave to the accountants and that became a new department called treasury.

What we are starting to see with some companies is they are now starting to create an office for AI governance. It's elevating governance or FinOps from where it often sits today into something that blends the finance department, the technology department, the strategy department, and the risk department to figure out is there a central way that we can start to better invest capital, manage costs, and build discipline around capturing return.

So we're not just measuring is AI producing things but we're measuring is are we taking different decisions and taking actions because of the way and what AI is outputting. So so we are starting to see this um you know AI went started central it started to federate pretty broadly and it seems like it's coming back to a little bit more central coordination on um on some of those areas of big capital and uh and operational expense.

Is that a new title that we're going to see within organizations? AI governance officer? >> I I I mean I don't there's no uh you know there's no title for that yet. Like we have heard uh companies starting to talk about uh you know leaders of token economics or uh AI economics leaders. Uh but I think just the idea of AI governance becoming something that is no longer just controlling access to who has uh who has what tools but now um making sure we're putting the right discipline in place like I imagine that will become a job title in the future and maybe even a new department in the future. >> Dan Galler, you something?

I'm just waiting for the first C AIO to come up and how how we say that. That should be that should be fun. And I totally believe in profanity as a KPI. >> I'm I'm going to go ahead and uh disagree, Dan, with with you uh a little bit, >> which is no company I work with has uh an electricity department or an internet department. And tr traditionally I I see that sometimes when people send up set up centers of excellence it can actually become massive gatekeeping and really take this power and say only we understand it and you can come to us and ask us for a use case.

Of course, there's effective PE departments that do this, but typically that is a huge risk and if this is truly a general purpose technology, which is what I believe, every person and every function is going to be using this to power their work. So my view is that the traditional IT function is going to be setting up their those guard rails and making sure everyone has access to it and the usage complies with corporate policies.

But I I really uh do get I do try to push back on companies that are trying to just set up centralized uh authorities that'll say whether or not people are allowed to use this technology. Um so I'm cautious of that. >> Yeah. And and James, just to respond, I I think I I actually agree with you that if this is a gatekeeping function of who gets access to what and it's a control function, then that that's probably not um that's probably not helping to move the company with this transformation.

But if you're being strategic that we're going to disproportionately put capital into this part of the business because we think this is an opportunity to allow us to compete like many organizations feel like they're susceptible to a company that is going to be born over the next couple of years that is going to have an operating system and a way of working that is fundamentally different than the way they operate today.

So in a lot of ways I think you have to enable everybody but you companies need to remain competitive in this position of allocating capital to areas where they can they can they can compete in new ways and um I totally agree there's not departments of electricity but there are departments of human resources. So, you know, it's and and I don't want to equate AI with uh with humans by any means, but I think it's really about disproportionately investing capital into the areas that allow the company to be most competitive. >> Not >> I want to jump in with um an audience question.

Um what about security? How are companies handling security while using LLM and servers and prompts? >> Well, I'll I'll jump in and then James and Dan can uh can add to it. Um my previous report I did and a previous briefing call kind of made clear that like right now like security is chasing the opportunity. Um a lot of companies are really aggressively adopting AI and especially agents first and then figuring out oh security how to secure them.

Um and I think this is the case whether a company's using like a US frontier model or or a Chinese model. So, it's it's a it's a definite it's a vital time because um adopting AI can certainly make a company a lot more competitive. There's we've talked a lot about that already about all the ways that um that AI can help a company do its work better, help employees do their work better, but there it brings new security risks to it that um I think are still being figured out. um especially on the agent side.

Um and James Dan may have some more uh more viewpoints on that too. >> There's a a critical distinction when we talk about non US models. There's non- US models that are served via subscription and those if you're are are not ones that corporations should be sending financial documents or uh you know producing IP and putting it into because the corporate protections aren't there. There's open source and open weight models which are ones that can be downloaded and run locally within the four walls of your perimeter.

Those still uh if if they're produced by non- US manufacturers require care to making sure that there's not access for Xfiltration. However, they're incredibly powerful. It's also notable that these even the US foundational model providers have openweight offerings that can be used. So there's a massive cost arbitrage opportunity and we were speaking about this earlier openweight open source models should completely be on the diet for companies that are trying to uh control their AI budgets.

But that distinction is critical when we're talking about non- US models. >> One more question from the audience. Companies have budgets based on historical performance. What what should companies allow in terms of decreases? Um, sorry. What decre what should decrease to allow AI to exist in the budget? >> Dan Dazio, do you want to take that one? Yeah, this is the tricky question that we're engaging with many of uh of our clients on and is why I uh I actually agree with a lot of what James is saying that this this is uh you know just looking at financial return is not necessarily fully quantifying what the impact is for society and for these companies.

Um but in many cases figuring out how you cover for that that cost item is is a really is a financial exercise. what what I often uh what I believe is that many companies technology budgets as a percentage of their total revenue is increasing and will continue to re uh increase over the course of uh of the next couple of years like I think average it's around 4% of total revenue and and my guess is is that um starts to significantly increase uh over the next couple of years.

Uh so so what what is important for organizations is as that cost is starting to increase they're they're looking at not just ways that they offset that cost with revenue but they're trying to put this into ways where they can grow the business. One of the most common areas that we're engaging with our clients on is how do they grow the business by a significant uh growth rate without necessarily having to uh hire a workforce that is at the same run rate as what that growth is.

And that's just allowing them to become a little bit more uh fluent in the way they use technology. >> It's a question of where is the company creating value and where are they looking to reduce costs here. Here here's an example. Let's say we have a pharmaceutical company. If they have a sales team that has an incredible demand for their product that is a and it's a high margin product that's a great place to use AI and and actually I would say I wouldn't want to start reducing costs there.

I'd want to aggressively invest right there. But if I have an off a back office function for instance that's doing purchasing and that's not value accreative to my organization that may be a place where I want to use that same budget and figure out how to do 10x the output number of POS I can cut number of contracts I can review with the same budget. So you we're always thinking about where's the elasticity where are high areas that if every dollar I invest I get a$150 back.

Those are areas where I'd actually want to drastically increase the AI budget even outside of what may be uh historical and spend way more versus areas where I know that uh I need my cost to level out. >> Well, we like to look ahead in every every briefing. So, I'll turn to Dan Gallagher. Um Dan, what are you working next for our subscribers? >> Uh well, uh it'll come as no surprise. uh it's on uh AI and uh specifically we're going to we're uh taking a dive into this question of um I mean it's it's the dominant question I think right now is you know do we slow or pace AI development or keep it keep it at high speed because of China all the complexities in that and I'm working with some colleagues to like shed some light on that >> very important and newsy topic um Dan Dasio what are you keeping an eye on when it comes to AI next? >> Um, well, there's not much AI news as of late.

Um, it's uh it's been a pretty it's been a pretty crazy couple of weeks. Um I I would say the the big thing that uh that we've been focusing on, we have a a piece of uh a paper that we're we're going to put out that looks at this shift towards uh seeing AI as a tool for taking cost out of the organization to pushing AI more towards growth inside of organizations and what it takes to be able to build that mindset that um your people with this technology can go do a lot more as opposed to the technology replacing what your people do.

And we've done a bunch of uh studying on that. Um the other area that that I find uh extremely uh fascinating is as companies are starting to move more from the model of people to agent to person to uh people doing work and agents starting to pass work off to each other before the work is returned back to a person. There's a lot of uh discipline that the technology se that the technology and the AI labs have gotten really good at.

Um you know building rubrics, evaluations, really good control planes, uh teaching some of these models with fine-tuning or reinforcement learning to be able to behave optimally inside an organization and to have the appropriate guard rails. And that's an area I'm starting to see a lot more enterprises start to think about and start to build discipline around. So that's a I I think that will start to unlock new capabilities in a very safe and responsible way. >> And James, what about you?

I >> think the we've mostly focused on human usage of AI. The topic that I'm keeping extremely close eye on is computer usage of AI. We have looked across a handful of Fortune 500 companies comparing human usage of AI versus a computer using AI and it's it's gone to about 10x. So every token that a human is putting in, we're seeing about 10. That means that as an agentic coding fleet starts to go off, each of those modules is sending off its own requests.

And that I think the sticker shock and the the headline news about folks getting jaw-dropping bills as computers are starting to use AI and consuming that budget is going to be in the headlines again soon. >> We'll definitely be keeping an eye on that. Um, thank you so much. I think we're out of time. So, we've we've covered a lot of ground, but wanted to thank Dan Gallagher, James Milin, and Dan Diazio for joining us today.

Before we wrap, a quick reminder that you can request a WP Intelligence Platform demo by clicking on the link on your screen and we'll also send our live audience a link to the recording of today's briefing shortly. Please do take a moment to complete the short post briefing survey. Your feedback directly shapes what we cover next. And please join us next Friday, September 25th, when my colleague Louisa Savage will sit down with Dan Gallagher and Benjamin Guggenheim to discuss AI safety.

Have a wonderful weekend and we'll see you next week.