I went to Microsoft Build (as an iOS developer) transcript

So, we are at Microsoft Build right now. We are back in SF. >> [music] >> And all of this computer Yeah, this is where the AI is running by. >> The official forecast was trending more towards hitting [music] Taiwan versus the northern Philippines. >> This entire industry is moving so ridiculously fast. A lot of you guys do watch my channel because it seems like I'm kind of on the cutting edge of some of this stuff. Like I am always experimenting with the latest models, the latest new agentic frameworks, but to be honest, I am also very behind on this stuff.

Like it's really hard to keep up. I'm excited to talk to the people who are on the cutting edge of this stuff to see what am I missing. If you've been following along, I have been experimenting with local models on the channel. The calorie tracking app that I launched originally was using local models. I'm also following Gemma 4, which was just released, which is an open model from Google. A lot of the stuff in AI is happening on Windows, and I'm not that familiar.

So, I am excited to talk to people who build on top of Windows. I'm excited to talk to some people from Microsoft, some of the startups, and see what am I missing here as a Mac user. And then obviously, we're going to be here with Logitech. So, thank you to them for flying us out, getting these ridiculously expensive tickets, and getting us access to this stuff so we can share it with you guys. Logitech, I think is one of the few hardware partners here.

I haven't really talked about it on the channel that much, but I have a suspicion that I will eventually be getting into hardware. Probably talk to some people from Logitech about what's happening with hardware and AI. I think that's why I'm very excited about this one cuz there's a ton for me to learn here and then pass on to you guys. We'll see. We'll see what kind of conversations we can have. >> Stay tuned. >> Stay tuned.

[music] Stay tuned for day one. >> [music] >> We tried getting into the uh keynote session with Satya. But it's already 9:00 a.m., completely full. Even this overflow is already like completely full, too. Yeah, we're just going to have to I think we just have to watch it on the live stream or something to be honest on our laptop. We're going to walk around, check it out, maybe get ready for some of the breakout sessions that are happening right now, too.

What do you think? >> Yeah. >> Cool. >> Let's do it. >> Okay, so it turns out this was the first time Microsoft Build was hosted in SF and it was a much smaller venue than usual. We kind of struggled because most of the keynotes and talks barely had seating and most of the time people were just standing. We learned the hard way that we had to show up really early for the sessions we cared about and there was one talk I made sure to get a seat for.

A talk on building apps with local AI models. The big takeaway for me was it feels like Windows is really ahead on local AI and I say that as someone who builds primarily for the Apple ecosystem. Apple does have their foundation models and I've used them in the past, but for most of my use cases, it just isn't there yet. that presented was running a booth, so we stopped by and it turns out one of them actually watches the channel and recognized me. >> to meet you.

I'm part of the >> Oh, hey. Hi, hi. Chris. >> I think I've seen your videos on my >> Oh, nice. Oh, I appreciate it. Yeah, thanks so much for watching. >> cloud code video. >> Yeah, yeah, yeah. >> Your app that you're building about nutrition app. >> Yeah, yeah, yeah. Oh, that's Oh, amazing. Yeah, thank you so much. Yeah, yeah, I appreciate you watching, too. That's You work at Microsoft or >> Yeah, I'm part of the Microsoft Play team. >> Oh, that's amazing.

Okay, well >> Nice to meet you all. >> You can be on the channel. We're going >> Absolutely. >> We're doing a video here. Shout out to Greg. He and his team were so kind and spent a lot of time with us. They showed us some pretty incredible demos that completely changed my perception of local models. The first one I want to highlight and honestly the first real magic moment of the trip was real-time video upscaling. This is a big deal to me as a builder because to do this well, you usually need to do it on a server and pay for some sort of expensive upscaling model.

But now, you can just do it completely free on the device and I'm pretty sure this was just a standard Windows laptop and here's what it looked like. >> First, I do want to say this may not show up the best through the camera screen because we're seeing some live upscaling, right? So, it's going to be compressed and probably upscaled by YouTube again. So, what this phone does, uh the way we do video transmission is it actually compresses it on the sender side because we want to optimize for bandwidth, latency, lag, right?

Uh so, we're going to compress this down to some super tiny resolution. We're going to stream it over the web to this device. And so, here on this device is where I'll leave this here for a moment. Uh this device is where we're going to be doing uh the upscaling back to the original resolution. Uh and then that's where video super resolution comes in on top and can actually upscale it past whatever that original resolution was.

And so, when you keep that video compressed through this whole transmission process, again you save on bandwidth uh which helps with lag, cogs, uh CDN service costs, etc. So, that effectively you're helping people stream better video quality for less cost on your resources, on your org, etc. >> I see. Okay, okay. That that is so interesting cuz I assumed, looking at the demo, okay, this is happening on the server or something like through >> All local, man. >> Okay, okay. >> So, no token cost, nothing, right? >> The next demo that changed my perception was real-time transcription.

If you've been following the channel for a while, you probably remember that I built an app called Lily. It's an app that used local models to help you record meetings on device. It failed because the accuracy and latency just weren't good enough at the time. It was literally slowing my whole computer down every time I ran it. But the part that really surprised me was that the device running this was an older Windows laptop that didn't even have much RAM.

You can see it barely touching the CPU. Normally local AI demos run on really powerful machines, and this was running on a hardware that's a few years old. As someone who's tried this exact thing and given up, this genuinely impressed me. And it made me realize maybe I should revisit building Lily. Now, aside from how smooth all this ran, the other thing that surprised me was how easy Microsoft made this for developers.

I'm not sponsored by Microsoft and have no incentive to say any of this, but I think they're beating out Apple here. They showed me that you could add something like image upscaling in just a few lines of code, but the part that really impressed me was how much they embraced third-party models. >> Now, if you don't want to use our model for any reason, you can bring your own. You can go to Hugging Face, download any model, and integrate with the WinML stack as long as it meets one of the supported standards.

And then from there, you interface directly with the model, but we handle the rest through the machine learning layer. You can also roll your own, totally an option. We also have the foundry model catalog, as well as several other model catalogs provided by Microsoft or third party. When the model selection's is happening, too, is it taking into account the hardware that's being Oh, okay. >> what I'm saying is you call one video super resolution API.

So, when that API is actually running on the machine, when it's executing, it'll detect the best available hardware, call up and download the appropriate model. >> One last thing I wanted to highlight. Because all of this stuff is so lightweight and efficient, they talked about a technique they're starting to see more of called model stacking. Running multiple local models at the same time or chaining them. Again, this is something you usually run on the cloud or pay some sort of third-party for, but now it's completely free.

And as a builder who's bootstrapping, this is huge for me. >> yeah, local models are also getting better every year. So, over the years, the next couple years, we anticipate that those models will just get, you know, better and better over time, more efficient as well. So, you know, those APIs are just, you know, we're just getting started here. >> One of the benefits of designing these models to be uh so small is that it actually allows for model stacking, right?

So, if we look at the video stack, for example, uh super resolution upscaling is just one of those things you can do for video. Luming here is one of our model researchers, actually. And so, his team has also been working on something called camera stabilization. And so, uh that model's still in production. It's one of the ones that we're working on right now. But as we look towards releasing that, you'll actually be able to start stacking uh video super resolution and camera stabilization so that you can stabilize the video and then upscale it. >> Cuz they're small enough you can run >> multiple models at the same time, right?

That's the whole point here. >> reason why we need to keep that as this size. >> Exactly. Uh DT is one of my partners here, where she works on audio APIs. So, you can uh stack models like uh live translation and live captioning to >> uh we have a speech recognition API. Yes. That does That does speech to text. It does it in batch mode, so you can, you know, do one-time audio, uh or you could do it in um a streaming mode, like if you tried to have a call or meeting type of scenario.

They run well on NPU. They also run on CPU. So, we got We got a lot of reach. Yeah, it's shipping really soon. >> Again, on the video side of things, you can stack camera stabilization, followed by video super resolution. Uh we're also working towards uh compression models that help with latency and uh frames per second. Now, you compare that with our audio models such as live translation and live captioning. So, at the end of the day, you have your videos auto translating, it's being captioned so that you have subtitling easy and naturally for your video on top of the improved camera quality or perceived camera quality to users at the end of the day.

Yeah, this is Windows AI APIs and unmetered intelligence unlocked. >> Will is talking tomorrow. >> You got to go watch his talk. >> But if I got a really technical question, that's my goal. >> Yeah, yeah, yeah. >> Chris, will okay, please. >> Let me call up a member of the audience. >> Yeah, actually, matter of fact, Chris, you're better suited for this question. You're an expert at this. >> I got you, dude. >> You want us to give you some softball questions? >> I feel like people care about privacy too, but not as much as cost.

I feel like cost is the most the one thing they care about. You know, that's why I cuz like at first I'm like, "Oh, who cares? I'll use the best." But it's like, "Nah, dude, like you actually want to lower your total cost." >> Yeah, they do. They do. They do. >> And then I think local's going to be the next I think that's going to be the next You should talk to people there. They're like actually like so good. Like they know what they're talking about. >> Dude, I love all like researchers too on like the local side.

It's like so good. >> Next, we checked out some demos from Microsoft Research. They're basically the group that takes the cutting-edge research that Microsoft is working on and shows how it can actually be applied in the real world. We saw a lot of cool stuff here. They have this model called Trellis where you could give it a single image and it generates a full 3D model out of it. But the most interesting thing I saw was something called Microsoft Aurora.

Here's the best way I can describe it. You're probably familiar with foundation models like Sonnet and Opus that generate text, but a foundation model like Aurora forecasts Earth systems like weather and air pollution. This was genuinely my first encounter with a foundation model like this and the conversation was fascinating. Unfortunately, because of the nature of what they work on, we couldn't record any of it, but their team was super nice and said they'd send a few emails and see if we could come back the next day for a proper recorded conversation about Aurora.

And I'll just tell you now, stick around because they did end up making it happen. That conversation is coming up later in the video and it's probably the most interesting conversation we ended up having the whole trip. We just took a break. We are going to head to Logitech now. Again, they are the reason we are able to attend a conference like this. I specifically asked if there's somebody that we can talk to that works between software and hardware so we can pick their brain on what that's like, what they're excited about.

They actually were able to pull someone in so we're going to talk to someone on their SDK team and have a conversation. >> Hopefully I don't disappoint you. >> it's okay. It's okay. We might not use it just yet. >> Yeah, yeah, of course. >> We talked to Paul who works on the Logitech SDK, which is the layer that lets people build on top of their hardware. And Paul told me something that I think is genuinely a big deal.

The entire audience for hardware SDKs is changing because of AI coding. I'll get into what he means by that, but first the fun stuff because we spent way too long at this booth. If you've been following the channel, you know that I use Logitech's MX line, the MX Master 4 and the MX Mechanical Keyboard. And they also have this creative console where you can map buttons to customize actions. Paul mapped his to the terminal, so one button switches models in Claude Code and another one cleans up all his terminal windows.

And Paul claims that someone else did this, but when we got there the mouse was mapped so every button played a very specific music video. Whoever used it next got a nice surprise. And they even have specific mappings for apps like VS Code that they worked with Microsoft to build. We spent like an hour here, it was way too much fun. Now here's the thing that Paul said that stuck with me. Their SDK used to be aimed at big third-party developers and Logitech would come to conferences like this to convince companies like Adobe to build on top of their hardware.

And he said because of things like Claude Code and AI coding in general, things have changed. He said at this point everyone is basically a developer. So now they want to reach people like me and you. So they've done stuff like published an LLMs.txt page. It's a document you hand directly to a coding agent like Claude Code and it teaches it how to use their SDK. So they're writing documents knowing that people are going to give it to their agents.

And he said that every hardware company with an SDK is starting to think this way. And for builders like me, that's genuinely exciting because it means hardware is about to get way more accessible. If you use Logitech stuff and I know a lot of you do, I highly recommend pointing Claude Code at their SDK and telling it to just build a customization. [music] A lot of the stuff I showed here was built in like 5 minutes with one prompt.

Huge shout out to Paul for spending way too much time with us. And again, shout out to Logitech for making this entire video possible. >> So much for taking the time. It's lovely to meet you guys. Thanks. >> This was actually really helpful. I had no idea that >> Yeah, no. It's genuinely as I said like about about I joined I'm here less than a year. And when I came in the job was like, "Oh, like how do we get more developers on?" So we were like, "Right, let's do Node.js because the the feeling was the feeling was oh Node.js is that easier to use.

But like in the last few months it's like we need stuff focus on languages and just focus on AI consumption. >> Yes. Okay, that is so brilliant. >> I love that camera by the way. >> Oh, do you do you have this >> No, I have the older is it I had an A6000? Yeah, old school. >> Yeah, yeah. We just got this. >> [music] >> Yeah. This is such a >> [music] >> On the one thing you're running like fully running on your Uh what is the most frustrating part? to really dig into to find those problems out? >> So all of this computer >> Yes, this is where the AI inference [music] is running by the >> And we're working closely with them, right? >> Yeah, yeah. >> And you can feel >> Okay, remember when we visited Microsoft Research, but we weren't allowed to record the conversation?

Well, a huge shout out to Heather for making this possible. He sent some emails and we got to sit down with Kenji Toketa, who is the director of research incubations to talk about Microsoft Aurora. I'll let Kenji explain what Aurora is, but first let me show you a couple demos he walked us through so you can see it in action. But the quick version, Aurora can forecast things like weather as accurately as the systems we use today except those systems need a massive amount of computing power to run while Aurora doesn't. >> So this is what you're saying is like an Aurora forecast of temperature compared to like the regular forecast that you would run on supercomputers.

So you need to remember like this forecast would have taken, you know, an hour or two on thousands of CPUs and this is just taking a couple of minutes. >> Wow. >> And then this is one of showing Typhoon Doksuri which was in 2023. Um, you know, unfortunately it caused a lot of damage. Um, but what you'll see here actually is a track. This blue IV tracks was the true track of what happened. Um, PGDW is like the, you know, the existing hurricane forecast and then Aurora.

And what you can see here actually is you'll see the Aurora's tracking the blue line, which is what actually happened, but the official Philippines forecast actually, you know, was getting it slightly wrong. And actually with Aurora it computed that the um, hurricane would hit on day four, whereas the official forecast said it would, you know, hit a day later. And then also what you see here is the official forecast was trending more towards hitting Taiwan versus the northern Philippines.

It's a very extreme example, but we actually have tried this on lots of different hurricanes and Aurora does, you know, pretty well. And so it's, you know, really important use case. So this is actually showing a fine-tuned model for air pollution. It's nitrogen oxides. What you can see is like fossil fuel plants kick out a lot of uh, nitrogen oxides. So you can see here in the day and during the night the variation in nitrogen dioxide, which is like fossil fuel emissions basically.

This one here is actually a sandstorm that happened um, in the Middle East. What's interesting with sandstorm again is a fine-tuned model is you're not just predicting winds. You actually have to predict like how the sand gets picked up, moved a few miles, and then it gets dumped. And it's a lot of super complicated physics. Somehow Aurora has managed to kind of learn that physics and it can do this kind of multi-day forecast.

Normally you can only predict about 24 hours ahead. And this just shows the architecture of Aurora where you kind of feed it the initial data. It translates that into this latent space. So it takes all of the variables, puts it into this latent space. It runs this actually swim transformer unit. It's actually kind of a computer vision model. Um, because this is like a bunch of slices like pictures. And then it it then advances it like six hours.

And then the decoder then maps it back onto the globe. And so this is the loop that you run. And so, it's it's it's very interesting just in terms of um, Yeah, how this works as an architecture. But at the core of it is kind of using a lot of computer vision algorithms to predict the weather, which is kind of interesting. >> So, that's what this thing can do. And the more he showed us, the more I realized this was the most extreme version of the pattern I've been seeing all week.

Tasks that used to require massive AI infrastructures to run requiring way less compute. So, here's Kenji. >> Aurora is the world's first foundation model of the Earth's system. So, that includes the atmosphere, but also the oceans, and also the land. Uh it was developed by a fantastic team of researchers in our Microsoft Research AI for Science Lab. And that lab really is thinking about how do we develop AI models that learn the language of nature?

So, going beyond the language of humans, everything from atoms and molecules all the way up through proteins and to the Earth. So, it's really fascinating just in terms of how we can use AI to understand nature. All the different possibilities that has. >> Most of my audience, they're all builders. What are some of the use cases that a builder or someone who works at a company could potentially use Aurora, fine-tune?

Do you have any examples that you're allowed to talk about? >> There are certain industries where weather is really really crucial. One of those is the energy industry. You can think about renewable energy and wind farms and solar power. You need to understand the weather. Is it cloudy? Is it not cloudy? Is it windy? We've been working with a company in Switzerland called BKW. Again, they've been really exploring Aurora for different scenarios.

So, that's really great to be working with energy companies like uh BKW. There's a really uh amazing project we're doing with a startup right here in San Francisco called Terra Dot. And they're a company that's a carbon removal company. So, they actually uh and they work with us on how do we take CO2 out of the atmosphere. It turns out that rocks absorb CO2, but really slowly. But if you crush the rocks, okay, you can really speed up the carbon removal.

So, they've actually extended Aurora and added an encoder and a decoder so that they can model the absorption of CO2 in like a particular field in Brazil and do that forwards. It's been a work we've been doing with them with again the I for good lab and it's a really interesting extension way beyond weather forecasting. Finance industry is very interesting, understanding risk due to changing climate. You know, even sort of retailers wanting to understand, you know, next season or like food production, agriculture obviously is huge.

Our partners at the University of Cambridge are also working to see how this Aurora model can be used for instance to support African farmers where weather forecasting was just not available to them like it is, you know, here in the US. >> To do weather forecasting today, right now, how intensive is that? Cuz for some reason I kind of think, oh, anyone can do weather forecast. You could Google it or look at it and it just happens for some reason.

But you were telling me that it does require actually a lot of computing to be able to do that. Could you just talk a little bit about that? >> Yeah, so weather forecasting today, you know, is done on massive supercomputers and basically you take the laws of physics of how the atmosphere behaves and then you model that, you know, time step at a time, you know, every few seconds or minutes. Very, very computing intensive.

So to do like a 10-day weather forecast for instance, which is what you see on TV, it may take like an hour or two on say 10,000 CPUs. Okay, 10,000 CPUs. Yeah, yeah. Okay, so that's what then, you know, the the national weather services in every country run that every day and they provide these forecasts that help everybody. Um with Aurora, you spend a lot of time pre-training but to actually run that forecast, it takes like a couple of minutes on a single GPU to do the same forecast and it's as accurate or even more accurate thousands of times faster and it's starting to change how we think about using weather forecast, weather intelligence for these different applications. >> I was very surprised to hear that Aurora was open source.

What was the reasoning behind Microsoft doing that? Because I feel like this was probably cost a lot of money train and then put out there. What was the reasoning to actually open source a model like that? >> So particularly when we're starting out with research and we want to publish the work, one of the important things is, you know, when we put it out and it gets peer reviewed, other people can actually look at what we've done and reproduce that.

So, this reproducibility of research is important. It builds trust so that people again could just move the whole field forwards together. >> Could you talk about if someone wanted to fine-tune something? So, I'm trying to make this as like applicable as possible. How much power do you need or or what how much data do you need to be able to start doing fine-tuning on this and actually using it? >> So, it does depend on the problem that you're doing.

But, you do obviously the more data, you know, is good. So, you might need say 10, 20, 30 years of historical data, which could be, you know, multiple terabytes of data, okay? And then in order to do the fine-tuning, you know, you have to run that on, you know, a [music] set of GPUs. So, it does depend what you're doing. If you're doing something quite specific and small, then you might, you know, need one or two GPUs.

But, if you're kind of doing a global fine-tune with lots of data, you know, then you might need, you know, a few, a dozen GPUs to run for a few days or, you know, couple of weeks or something like that. So, it does obviously vary on the problem size. You have to train it until it converges to some good results as well. So, and yeah, you keep training it and testing it and evaluating it. >> So, it sounds like it is very problem-specific.

So, it's not just like a blanket "Well, you need this many GPUs to do this." >> So, it is problem-specific partly because of the flexibility of the foundation model. >> Is there anything that you want other people to know just about the research you guys are doing or something that you feel is not uh mentioned enough or not covered or like "We need more very good engineers working on these specific problems." Is there anything that you want to kind of almost plug here on like "We need more people working on these problems." >> Sure, yeah.

I mean, I think one of the things, you know, that we do need are people looking and working together as a community. So, you know, with Aurora that is open source. So, people can contribute to that. And that means that, you know, we can kind of lift the tide of AI research, particularly in areas like science uh and weather and climate, which affects everyone. And we just want to make sure that the use of AI has a really positive benefits to society. >> I really appreciate you taking the time to >> No, thank you for coming back.

You know, like it's it's it's you know, it's just it's we really enjoy what we do and again if we do it, you know, we get up in the morning because it's it's exciting science but also just the applications. >> No, I I think the conversation we had yesterday was probably the most exciting one for me. Oh well, I I feel like it just unlocked a full area that I was like, why is no one talking about this? >> Yeah, and Aurora is just one of our projects today at AI for Science, you know, and then at Microsoft Research AI for Science.

So you have the protein modeling >> Yeah, yeah. >> modeling like atomic structure of molecules. No, no, no, thanks a lot. Really appreciate it and uh yeah. Yeah, definitely. I'd say there's a bunch of other stuff we do. >> Sadly, there were so many things that didn't make it into this video. A lot of incredible conversations I couldn't record. But this conference genuinely meant a lot to me and I wanted to give it a proper video because this was probably the most out of my element I have been in a long time.

And I think you guys could tell. I learned a ton about local models, about models way outside the LLMs we usually talk about like Aurora. I had really in-depth conversations with people from AMD and Nvidia and got to ask some things like, what does hardware optimization even look like? But the thing I really wish I could have conveyed to you guys is how passionate and smart every single person I talked to was. If you watched my SF vlog, you know I talked a lot about leaving my bubble.

But honestly, even that was still a bubble. My whole world has been solo dev, app developers, and more recently startup founders. But this week I was talking to researchers and people building hardware at massive companies operating at a scale and with access to things that me and most startups will never touch. And it really opened my eyes that I need to talk to more people like this. I always assume most big companies move way too slowly and to be honest, they kind of do still.

But the people I met were working borderline startup hours and moving at such an insane pace just because of how excited they are and how fast the industry's moving. This was a very different experience from any conference I've been to. I've got a lot more trips planned and after the support you guys gave on the last SF vlog, which seriously, thank you guys for that. I want to keep taking guys along. So expect more in real life content on this channel.

A huge thank you to Logitech for making this possible. Thank you to everyone at Microsoft who sat down and let us film and a big shout out to Brian for filming again. Hope you guys like the video and I'll see you guys in the next one. >> Is it obvious this is a closet or anything? Can you see the See your clothes. >> [laughter] >> All right, [snorts] so we are at Microsoft Build, right? We have um the founder of Open the Closet. >> [laughter] >> Hey, come out.

Come out here. Is that you? Yeah, we [snorts] can't >> [laughter] >> Here, where where can we film?