Chips, Power & Inference: How to Build the Physical AI Layer | SambaNova CEO transcript

When we started the company in 2017, we thought that AI would be this big transformative technology and that feel across the board. We're all starting to realize it's actually bigger than we even thought. The amount of infrastructure support to actually meet the needs of that demand is still catching up to it. >> Sometimes the most powerful forces in the world go unseen. A critical layer of intelligence driving possibilities.

This is Micron memory and storage. Intelligence [music] that uncovers new cures, models changing climates, and build smarter, safer cities. Every breakthrough in AI is driven by data and data lives in Micron memory and storage. >> Welcome to episode one of the Washington Post's Building America podcast. The Building America series explores how America builds better for the future. Over the next two episodes, we're examining how AI is as much a physical story as it is a digital one.

[music] Chips are a key part of the architecture of AI. New disruptors are entering the space quickly and SambaNova hopes to be one of those disruptors. Today, we're joined by Rodrigo Liang who is the CEO of SambaNova. >> [music] >> Rodrigo, thanks for being here. >> Yeah, thanks for having us. >> It's a pleasure to chat and I just want to dive straight into it. You all have been around now for a startup for a little while, right?

You're not exactly spring chickens, but tell me when you when you all got going, what what was the problem you were trying to solve? What was the the impetus for for your company? >> Well, when we started this company in 2017, we're thinking about the transition that the world was going to go through with AI and being something as transformative as it's now proving to be. We just thought that it was going to be something that we needed to actually create more efficient chips and more efficient infrastructure in order to scale.

So, really the issue that we're seeing now is how do you actually take this amazing technology uh that uh many people have now demonstrated capabilities what these models could do, but how do you take it to scale? But, how do you take it to production? How do you take it to the entire world so it's democratized so everybody has access to it at a cost and a access level that is available to everybody. And so, what really what someone of us are targeting is trying to take us into the next phase of the AI evolution, which is pervasiveness.

It's driving this across entire planet and um and making it available so that every country, every individual across the planet can have access to it. >> And I want to dig a little more in in into the the nitty-gritty. One one of the goals with this series of episodes about the chips economy is to try to put into to plain language or into, you know, getting down to what exactly this means for for people. And so, with that, we hear the term inference a lot.

And could you talk a little bit about inference and sort of when you all look at the the overall ecosystem, what what your company is doing uh in terms of uh inference and and its place in uh how we're developing AI and how we're building out these these structures that you're talking about. >> Yeah, we we we talk about the fact that, you know, inference is here. Inference has broken the market open. These are all the things that people talk about.

And and ultimately what it is is if you look at the world of AI, there are two main pieces to it, the training part portion of it and the inference portion of it. The training portion of it is where we're creating these models. You look at it, you have ChatGPT model with GPT-5 from OpenAI or the Claude model or Opus model from Anthropic or the Gemini model from Google. And you know, people are inventing and creating that model.

The process for creating, inventing this model is training. Right? It's It's called training. You take the data that exists, you run these algorithms on these incredibly uh large, sophisticated computers, and then you actually create this model that is able to then produce the outputs that we are now starting to see when you chat to ChatGPT or you talk to Anthropic, right? And so, that's training. Now, when we are using that model, it's called inference.

And why is this so pervasive? It's because that's what the rest of the planet does. Right? The OpenAIs, the Anthropic, and those few company handful of companies have the investment that they're going to make to train and create the models, but the rest of us, all 8 billion people on this planet, we use it. We use it in to in order to get responses back. And so, that entire process of using a model is called inference.

And that's why it becomes so pervasive because now it's not the purview of a small number of companies. It's the rest of the world, and everybody is using it for different aspects of their lives. >> And why why do you need a different physical architecture for inference than you would for training? What in sort of layman's terms, what what why do you need a different kind of data center or a different kind of of build to to run these different uses? >> Yeah, if you look at training, what you're doing is you've aggregated the world's data, and you put it all together in one place.

Then you have this algorithm that you then deployed across thousands of chips. And then you are actually then, you know, in some ways, you know, similarly to us going to school, you're training it. You're presenting this data to that model over and over and over again over weeks, months, uh quarters, right? Where the model is getting smarter and smarter and smarter as it learns from that data on how to respond to various types of things.

That could be text, language, it could be videos, images. You know, so you're teaching it. And so, that whole process is a model you know, what I say, a monolithic process. You put it all together one place. This is where you hear about these gigawatt data centers that people are building, right? Because you've got all of these chips that you're putting together in these racks into one large data center, and then you actually operate a single model over and over and over again for a long time because you're teaching the singular model, singular algorithm how to respond to all these different things.

Now, once the output is done, what you and I are doing is we're talking to it for very specific things that we care about. You know, I want to ask about, you know, how to plan a trip to Europe for 4 days. Or I want to, you know, figure out kind of how to you know, how to file, you know, my taxes. You know, what whatever that question is, it's very diverse. And so, what you're seeing now is people needing to take that model, the output of that trained model, and you want to deploy it in a variety of different places.

You know, so, banks want to privately in their own environment because they they have security and privacy issues that they have to comply to. You know, you have large-scale uh um hyperscalers that want to do within their own ecosystem. And then you have uh you know, all sorts of retailers or whoever, they've got distributed you know, data centers across the world because their customers are distributed. And so, you no longer are tied to this singular monolithic location where all the infrastructure has to be.

Far more distributed, far more diverse in the use case, and then also narrower, right? Because you have different people focusing on finance versus healthcare versus different you know, so so you got a use case that's much more distributed, and therefore your deployments are also going to be distributed and smaller >> And at the same time you know, you all have been around for nearly a decade and you've been thinking about this for a long time.

Most people when GPT launched a few years ago, you know, the last 3-4 years they've really started to think about AI as the LLMs became bigger. But this isn't a a new engineering or technical problem. A lot of smart people have been thinking about this for a long time. Uh and at the same time a lot of people want to lower costs. They're worried about their dependence on one company, Nvidia, which has become a multi-trillion-dollar firm because of uh the the rise of of AI over the past few years.

I'm curious, why is there such uh a scarcity of computing power? Or why why is compute such a scarce resource? Despite the hundreds of billions of dollars, you know, maybe trillions being thrown at it, why now so many years into this is it such a hot commodity and that everyone is fighting over? >> I I you know, when we started the company in 2017, uh we thought that AI would be this big transformative technology. And I think across the board we're all starting to realize it's actually bigger than we even thought.

Right? And so if you think about the supply chain problems that we have today, you think about the data center shortages that we have today, you think about the energy crisis that we've got coming, right? We're putting you know, we countries that want to deploy data centers they don't have enough power, right? And so we're trying to figure out how to actually even you know, source the energy you know, to deploy. Let alone talking about then the sustainability things that come with deploying those type of data centers, right?

So So you think about all of those things that we're all talking about today in support of building these AI centers, it's all tied to the fact that the capabilities of this technology far outpace what we even thought and the amount of infrastructure support to actually meet the needs of that demand is still catching up to it. Right? And so uh you know, it had we known, right, had we known this, you'd have all the suppliers built far bigger factories for wafers and more memory, DRAM memory, and more, you know, capacitor for uh um um data center space.

But, as it's kind of grown, then you look at OpenAI and Anthropic and Gemini, all these different amazing companies that are building these models, I think what they're finding is that the demand and the interest in the AI has coming faster at a greater scale than we even thought back in 2017. And now that you're seeing the world move into inferencing, using it, right, then the aperture for number of people interacting with these models expanded by orders of magnitude compared to when we were just training, which was primarily tied to computer scientists inventing and building models.

Now, every person on this planet is starting to engage with these models. >> And I want to talk a little more about chips and not not just data centers and chips innovation, but the last last question on data centers, there's the practical technical question of how do you build more, how do you have enough energy capacity, you don't strain the grid putting them online, right? But, it's also a political question, and you're seeing in municipalities, lots of bans or moratoriums, and now at the state level, there does seem to be a genuine pushback from a lot of people against data centers, and there's debates about how organic it is versus how much it's being encouraged, but how do you all and I know you're you're a a you run a business, you're not a politician, but certainly that has to factor into your thinking, the the trends and how people are are thinking about data centers.

I mean, what do you think is behind that, and and how are you all trying to respond and make sure that you're able to still, you know, operate your business and and build out to meet this tremendous demand despite kind of this growing opposition we're seeing across across the country? >> Yeah, what what we see what we see is as we talk about scale and as we talk about making this democratized and you have access across the planet on this.

We do think that this is an incredibly important aspect of what SambaNova fits, right? Because we deliver the most efficient racks when it comes to inferencing. Right? And that efficiency allows you two things. One, I don't need a brand new liquid cool gigawatt data center to deploy. We can use existing data centers. We call them brownfield data centers, right? Existing data centers that are already running some sort of computing, we can replace the computing with SambaNova racks instantaneously bringing you up to state-of-the-art AI without having to create a new data center, without having to build new buildings, without having to bring new sources of power within that ecosystem.

We're able to just leverage what exists. And so that's one of the really, really important things about SambaNova, which as you deploy, you do not need to actually build new data centers. Your air cooled your air cooled infrastructure into existing data centers that are proliferated already across the planet. Let's reuse. Let's just reuse it, right? That's one. The second part of it, as they we talk about with grids and kind of energy constraints, one of the things that we want to do is actually just deploy infrastructure that produces the tokens.

You know, tokens is really the output of this inference, right? They have produces the outputs that people care about at a much lower power profile. And so if you look at an average GPU rack, it is consuming 130 to 140 kilowatts per rack. Right? That's that that's kind of equivalent of a home for a year, right? Running. And so SambaNova is at 10 kilowatts today. And so at a fraction of the power, you're able to operate these you know, these systems that produce the same if not more tokens out of these uh same racks at a fraction of the cost, which then helps.

And so So, that's really what we're focusing. We think that scaling is about efficiency. Scaling is about dropping the power, dropping the cost, dropping the time it takes to build data centers, and allowing everybody to be part of this AI economy. >> And you uh you know, as you just alluded to, a big part of getting more efficiencies out is perpetual improvement and innovating when it comes to chips, getting more out of them.

The United States is the world leader, uh but there are other countries, of course China, that would like to surpass the United States. What does the US need to do to keep its edge? Uh it's I know it's a it's a theme I could speak for the opinion section at the post. It's something we write a lot about is that you just can't take for granted that the United States will be the world leader in any given industry. As As someone in the in the thick of it, what what worries you or or what makes you optimistic about how the US can remain globally competitive when it comes specifically to innovating when with chips? >> Yeah, look, you know, someone over We are American-based company out of Silicon Valley, and the idea is that we are basing the company on came out of research that was coming out of Stanford University.

And so, we've taken that research out of Stanford, we brought in some brilliant minds that allow us to actually take the technology, commercialize it. And then And then what we did was we initially immediately deployed it into some of the top institutions with the American government. And so, the American government is incredibly supportive of American institutions like ours that allow us to actually then build a a relationship with those institutions like the Department of Energy, like the National Labs, and we're deploying at various different places in Texas Accelerate Compute.

You know, the government is able to help us hone in hone in on how is our product competitive in the market, how's it building good good use cases that people want, and the US government continues to be an incredibly large customer of ours. And so, I think two things that I'm you know that that that I I uh fundamentally believe is the United States that the the educational system of the United States is still one of those that attracts some of the best talent across the world, and we bring some of the best minds here that allow us to actually continue to invent.

And then the ability to actually drive that technology into yet the largest market on this planet and be able to then hone in our craft and build technology that we know that if it's good for the US market, it's good for the planet. It helps helps us actually deliver this value into a global stage, which continues to be one that's very competitive, as you said. >> As an entrepreneur and a builder, uh how do you see the narrative around AI and concerns about jobs in the future?

I know you you know, there's like we were talking about data centers a moment ago. A lot of the opposition to them comes from rising energy prices or maybe misconceptions about water usage, but it does seem like at root, uh a lot of people in the space, you know, Sam Altman has more recently walked it back, but a lot of people in in in the AI space were making some pretty bold pronouncements about what they might do to jobs, and that seems to have a affected perceptions of of the technology.

How do you think about AI? Is it uh is it an automator? Is it an augmentor? Is it both? What What is the What is the you know, the labor market look like as the buildout continues and adoption increases in the coming years? >> You know, like look, you know, can we be more efficient as we build our technology infrastructure? Yes. Can we actually be smarter about how, you know, we use it? Absolutely. And so, those are things that we in the in our field, we continue to strive to continue to improve and get better.

And again, at scale, we will continue to actually drive more efficiency more uh um a better value out of the tech that we're building. But that said, history teaches us a lot. You know, I've been in this business I'm building high-performance processors for over 30 years, and the same type of uh uh conversations we had and debates we had when the internet was just coming [clears throat] in in the '90s. So, I didn't So, as the internet was coming in, we felt like it's going to replace a lot of jobs, a lot of the industries that we have, whether that's the the Blockbuster the videos of the world or whether that's the you know, a CD at the music industry, whether that's the press.

All of those things were going to change. In fact, it did change. It did change. Yet, you know, it also create incredible opportunities for new businesses that we never even imagined when the technology was first introduced. You think about the ride sharing economy. You look at the cybersecurity industry. You look at all these things that got created that we never had prior to the internet. I think you're starting to see a lot of the similar things happening in the AI industry that yes, you know, some things are getting replaced because AI can't help and accelerate, but it's creating an entire industry of things that allows us to actually create new opportunities.

And here's the other thing that I think if you look at kind of the capabilities of AI for a number of industries, it is actually lowering the barrier of entry for people to participate in that industry. Right? That you can actually come in just like you know, with the internet enabled people to actually be a merchant out of your garage and selling things from your house where before you had to go to a store and create a storefront, here AI is allowing you to actually be part of an economy even though you may not have all the support staff to create all the documents, all the other assets, you can now actually leverage technology to be part of an economy at a much lower entry point than otherwise it would.

And so, I'm actually very bullish about think you know, for the people who really want to participate, they can actually think, you know, about this technology as an additive way to build your business, but it's going to create new economies, new industries, new opportunities that I don't think we even have thought about yet and I think it's going to continue to actually evolve and continue to expand. >> And you said you all are always looking for efficiencies.

I mean, how have you deployed AI in your business? What are some use cases or you know, how has it affected your your business? Not just in the meta sense that you're you you all are building out, but like the day-to-day activities of your company. You know, how has it affected your workforce using it yourself? >> Yeah, look, we use AI across the board with everything that we do. You know, obviously we're an AI-first company and we use it for a variety of different things including kind of how we actually use it in our marketing assets and in some of our legal work, but what is today one of the most valuable most valuable um use cases for artificial intelligence is code generation.

Right? And so this is going straight into the heart of engineering that we actually use AI to help our engineers code. You know, and you look at these models that are out there whether that's a um um Anthropic model or an OpenAI model or all these open-source models that are out there. You look at MiniMax or a Gemma 4 model from from Google. Fantastic models out there that can help you actually generate code. Right? And in an industry where it's incredibly competitive to find great engineers to help us develop these products that we're actually using AI ourselves to actually accelerate the product development and generate the code that's necessary to deploy.

And so it's in in in everything we do and my expectation is over the course of the next 6 12 months we're going to see even accelerated use cases within our own company and across the industry. >> You have a very unique perspective. You're born in Taiwan, raised in Brazil, came to the United States for your education and now to to be an entrepreneur. So you you've had all these, you know, international experiences and a really unique perspective.

But when you look at the United States and the the builders and innovators we have such a a rich history 250 years of of it really who do you particularly admire and and what is it about you know that particular builder or or innovator that you know sort of inspires you? >> Yeah, there's so many people out there, you know, but but I keep going back to you just watching have the opportunity to watch and live and kind of do this, you know, Steve Jobs and what he did into to an entire planet, you know, in terms of technology.

He took a technology a cell phone technology that effectively was already pervasive. Everyone had one and frankly something that you know, many of us didn't think that much of and turn it into the most valuable product that all of us on the planet have to have at all times, right? And you think about kind of we don't leave the house without this device because it's no longer just a means of communicating voice. It's something that actually our entire lives are highly highly dependent on and so I look at something innovation like that.

That's not always just about creating something that you know, that that never existed, but taking something that we've already used by the time he started the cell phone business was already quite you know, pervasive and turned it into something that none of us could live without by integrating all the real use cases understanding what we all wanted we all needed into something was significantly significantly better significantly more valuable and now years later, we're still completely dependent on it.

And so I do think that you know, it's it's not always about having just a brand new brand new technology that no one's ever seen. It's about creating solutions that we all want and we all need and we all depend on. >> And I know this is perhaps an apples to oranges comparison cuz that's consumer electronics and AI is a category of technology, but do you think that question remains unanswered with AI? Is there this use case or this even more widespread adoption or this obvious applicability that someone needs to popularize or make more usable that will really change.

Do you think that is a question out there that's unanswered that there's a Steve Jobs somewhere who needs to sort of reveal it to to or is that already happening and is that what we're seeing right now? >> I think we're early stages of this though. I think if you look at kind of what we have with the models, what we have with the data centers, what we have with the chips, these are all the pieces like we have the cell towers, you know, what we have here the internet.

We have these these are key components that we are actually making more robust and more available and yet there's a next chapter. There's a next chapter that allows us to actually put these things together that actually make this an everyday necessity for all of us. And so and you know, today you and I were, you know, intrigued. We're intrigued by the technology. A lot of people use it. You know, the kids love chatting to it.

Some of the business starting to create agents to it. But how do you get to the point where basically it's our hands every single day, right? And or in in businesses quote unquote hands, right? Virtual hands every day because it's a necessity in our business. I think there's a chapter still to be written and a lot of innovators out there looking to figure out what that is. And so that's the exciting part of our industry that I think there's still a lot of opportunities for that Steve Jobs moment. >> Rodrigo, thank you so much for coming on today.

This was a fascinating conversation. >> Great. Thanks for having me.