Robot Demos Are Easy. Reliability Is Hard — Jason Ma, Dyna Robotics transcript

[music] >> Thanks for the kind introduction. So, I'm Jason, co-founder of Diana. Today, I'll talk about how we're developing high-performance and very robust generalist robotics policies. Yeah. So, let's jump into it. So, today's talk will focus on, you know, how we're bringing robots into commercial grade and what we are doing to make these models, like I said, very high-performance and robust. So, just a brief introduction to our company.

So, our mission is to build very robust foundation model autonomy in the real world. So, we want to train models and these days also hardware to have a single platform that can do many economically useful physical tasks in the real world, like the ones we're showing here. So, the company the company was founded in September 2024. We are a series A company, have raised about $120 million, and we And our thesis is that to actually bring robots into the real world, being at commercial grade doing useful tasks, the company needs to combine doing frontier research with a lot of commercial deployments, so we can build what we consider a research and deployment flywheel.

Uh which is that the research and hardware we do in-house, the R&D informs the kind of tasks, the kind of workflows in the real world that we can commercialize. So, we actively try to deploy. So, these days we have more than five deployment sites doing a bunch of different tasks, which I will talk about in a bit. And then by building product and by deploying robots, it does several things. One is that it can help us gather high-quality deployment data, and it can also tell us what our models and what our hardware is not good at yet.

So, it helps us sharpen our research focus to figure out what is the right problem to work on in robotics. Because if you're familiar with the robotics field, there are too many problems. There are too many different fields you can spend your effort on. And by deploying and by building a product, we know exactly the right kind of problems that we need to focus to actually make a robotics not just a demo or videos you see on YouTube, but rather in the real world impacting millions of people's life.

So, before I jump into what we specifically work on at covariant, just a very quick introduction on the kind of models we're training for robotic manipulation. So, at a high level, uh we do data collection where you know, there's a lot of data being collected. So, here's a video of me uh manipulating a robotic hardware to, you know, fold a t-shirt. And once you collect enough of this kind of data, you can put all of them in a large neural network.

And a neural network essentially takes in a, you know, representation of the world, in this case just camera feed of what the, you know, a table looks like, and the outputs robot's, you know, joint positions or torque to actually control the robot to do the task that you have collected data on. Right? So, you know, this is the high level of how you robot models in the real world function. So, how are we actually scaling this up to actually create models that's generalizable and it can do a lot of different tasks.

So, at covariant we focus on what we call the pre-training data pyramid for the real world, where we gather a lot of diverse off-robot data, just because robotics data is very scarce. So, we have data captured from humans wearing cameras, uh from public data sets, and these days we're also considering some simulation data. But if you only have off-robot data, it's not actually enough to get robots to do very precise actions.

So, on the robot themselves, we also collect diverse tasks collected in many different scenarios, on many different types of tasks, like industrial tasks, household, uh in the laundromat, and uh you know, hotels. And then finally, because we're also deploying robots, we can collect very high-quality deployment data, which helps the model to close train and test distribution gap. Because when you're developing robots, you know, most of the time your robots are in your facility, in your laboratories.

But if you're deploying robots, it's in a very different environment. So, we found that by combining these three data sources, we can actually train large-scale foundation models that work very well in the real world. And so far, we have more than 200,000 hours of data in our training pipeline. And the model architecture roughly follows a high-level reasoning model with a low-level world action model that can actually output dexterous actions at fine-grained high frequency, right?

So, this is the kind of architecture we have because in the real world, if you think about a robot doing physical tasks, it needs to have a semantic understanding of the world, but also needs to understand physical interaction at a fine-grained level to be actually able to, you know, recover from mistakes and do very precise actions to complete the task. So, once you combine, you know, this architecture with a lot of data, what happens is that the model can be rapidly fine-tuned to do a bunch of different tasks in the real world.

Yeah, so here are just a gallery of the kind of benchmark tasks we're doing in the office. So, it spans from, you know, like cleaning trash, opening a box, to, you know, things like folding towels, folding t-shirts, and also just a bunch of other tasks our researchers have thought about. And what's really interesting is that with a good pre-trained model, even without any, you know, for some of these tasks I'm showing you here, they're not in the pre-training data at all.

But with a good pre-training, you can actually post-train the model to do these kind of highly dexterous tool use tasks with very little amount of data. So, both of the videos you're seeing here have only been trained on with less than 1 hour of task-specific data. But you can see that the robot can repetitively do these tasks over and over without failure. And I think this is a stepping stone towards actual commercial grade real world deployment because in the real world physical tasks need to be done by humans and by robots over and over.

And if your models are not reliable enough, then yes, you can shoot these kind of pretty demos, but it's still very far away from actual deployment. So that actually brings us to what I consider the current status quo for training large-scale foundation models for manipulation, which is that the generous models that can do many many tasks like the ones I've been showing you. Even though they make pretty videos, but what I would tell you is that the success rate is actually not super high.

You know, when these models are actually deploying, they're about 80 to 90% success rate. But if you're stuck at 80 to 90% success rate, then you know, the chance that you can do the same task 10 times in a row is actually less than 0.1%. And on on the flip side, in the history of robotics, we have had very specialized robots and a very specialized machine learning pipelines to do singular tasks like pick and place. But this kind of pipeline, which is what I consider specialist models, aren't aren't very scalable, meaning that you can't take the same pipeline to just do a new task and do an arbitrary task very fast.

So our mission is to resolve the status quo and you know, this kind of like dichotomy by training general purpose models that can both do many many tasks and also be reliable enough for commercial deployments. So how do we actually do that? In today's talk, I'll briefly talk about some of our progress on mastering very complex tasks very reliably and also taking the same skills to be performant not only in the environments in the laboratory environments that we're training, but also being able to deploy to arbitrary customer sites without fine-tuning or without fine-tuning adaptation.

Okay, so let me get to this. Right, So, the first question we want to answer is can we take, you know, the general pre-training and post-training recipe that we had, but then turn these models into models that can be almost close to 100% robust on uh any task. So, this is the first research result we published last year. Uh for detail, you can check out our blog post. But, at a high level, you know, if we want to make a model 100% robust on many, many tasks, I think it's very good to first narrow down on a commercial use case that's reasonably hard, which allows you to make research progress, but also has commercial value.

So, when we first started the company uh last year, what we discovered is that if you go to any, you know, uh restaurant, you know, uh fancy restaurant or dim sum places in the US, you'll see nicely folded napkin on the table, right, for you, right? If you go to a Cheesecake Factory, you'll see those napkins. And what happens is that in the back office of the restaurant, there's usually uh workers or, you know, restaurant employees that's folding these napkins one by one by hand.

And that's a very mundane process, and a lot of the customers we have talked to are looking into robots that can actually do the same task. So, this is one of the earliest case study we did on how to train models to be very robust. So, our research result is a model called the Dyna-1, which is a generalist robot foundation model that's fine-tuned to do napkin folding. They can actually achieve 99.4% success rate over a 24-hour span.

So, here is a time lapse of the model, you know, doing the task. And uh you know, it's sped up about a thousand times, so you can see the clock in the back running very quickly to show the progress. And uh my favorite part of the video is when the clock, you know, hits about like right now, right? Like 7:00 a.m. in the morning, so the lights, you know, actually come out in the outside. So, the environments are actually, you know, shifting over time due to the lighting as the robot folds the napkin, but the model's robust enough, and it just keeps going.

So, how do we actually get to a model that can do this? And uh first of all, you know, this uh video we put out is not a one-time occurrence. The model can do this many, many times uh in the office, you know, so these are four distinct trials of 24-hour runs. So, before I dive into the technical detail, just to highlight how difficult the task is. So, you see that in the napkin folding task, you start out with a stack of napkin on the side, and what the robot has to do is like be very precise about picking out exactly one napkin from the stack, and then fold it, and then put it into a bin.

And then a lot of the failure cases just come from the fact that these, you know, parallel jaw grippers we're deploying, you know, the gripper may not be precise enough to be able to pick out exactly one napkin. And in these situations, the model has to learn how to recover from the mistakes of pulling out extra napkins. And then secondly, the in commercial environment, different from a lab demo where the researchers like me are thinking of task success, there's actually very well-defined success criteria for these tasks.

So, on the right, you see the difference between what we consider a grade five fold, which is a fold quality that the restaurant would accept versus a grade three, which is something that's below the acceptance criteria. And what you see is a barely like 1-in difference in how, you know, low the you know, the second fold seam of the napkin is. So, how do we actually get to a model that works really well? So, our internal attempt was kind of like the pre-training many, many hours of data and then post-training recipe that I told you about in the beginning.

And doing this roughly gets you about 80% success rate. And what happens is that the model, you know, is doing fine in the beginning, but as soon as it makes a mistake, it'll typically go out of distribution, get stuck, and unable to recover. So, we have to do something more than the standard pre-training and post-training idea that's very popular in robotics and also in other fields. So what we did is that we developed what we consider reward models for complex long horizon manipulation tasks.

So these are models that can, you know, look at a robot video and accurately score its progress towards solving a task. All right, so let me just play these videos again. So here's the same model, you know, being able to score how well the robot is doing these long horizon complex tasks as it's, you know, going from you know, starting of the task to finish. So you see that as the robot's completing a task, it's able to go from zero to one.

And if you squint at these videos enough, you also see that whenever the robot is actually making some mistakes, there will be like slight dips in the reward model. And that actually becomes a very important insight into how to make these models very robust. So here's what happens when you run such model during like a autonomous run out of the robot. So you see that when the robot is doing fine folding napkins, the progress estimation is roughly monotonic going up, right?

Because the robots are not messing up. But what's really interesting is is that let me just for fast forward a bit. Whenever the model starts to make mistakes, so here it is. This is the ninth napkin is folding. So you see that when the model is like making mistake, that's when the progress estimation, you know, starts to like, you know, show non-monotonic sign indicating that the robot is messing up. Right? And this is very important because once your model is like good enough in the 90% uh range, then it's very inefficient for humans to manually oversee the robot to detect its failure and then try to recover.

But once we have this reward model, we can actually do what I consider uh scalable supervision. So you can just have the robot trying to fold napkins and then run this reward model in the background. So whenever a model does make a mistake, uh we as researchers or operators can immediately know the kind of video case that the model is struggling on and then do very targeted data collection and error recovery data for the model, then we can fine-tune the model again.

So, the overall pipeline looks like a human-in-the-loop active learning process where we can use the reward model to help us catch the the kind of mistake the model is bad at and then do targeted collection to make the model better and iterate. And what we found is that once you iterate on this uh couple cycles, then you start to get a model that's extremely robust and can recover from all kinds of errors and finally bringing us closer to a commercial grade robots.

So, here is just uh some of the uh you know uh highlights, I guess, during the 24-hour trial. So, what you saw there was the robot accidentally picked out more than one napkins and the model is able to, you know, separate the napkins and uh you know, here it's kind of doing that and uh be able to continue progressing. And what we found very interesting is that uh napkin folding is a deformable object manipulation task, right?

So, there is almost infinitely many possible states or configuration that a napkin can get to. So, it's impossible to exhaustively collect data for all the error cases. But once we had done the active learning many, many times, we saw the model able to generalize to new ways of recovery from the mistakes made and continue to make progress. And that contributed to its ability to be able to uh fold that uh 99.4% success rate.

So, here's uh what I found the most impressive bit from the trial. So, typically, if you look at robot videos, you know, they only show you the successful cases. But here, I wanted to highlight that even where a model accidentally pulled the entire napkins stack over, it's able to demonstrate this kind of error recovery behavior. That was very surprising to us when we were developing the model and it contribute to how it's able to just continuously run, right?

So, here he made a big mess, but he's able to just do all kind of like very impressive like pulling you know, stretching behavior to recover from his mistake and then continuously going. Yeah, so you know, after developing such technology, we actually are successful at deploying our models at many many restaurants in the US. So, here's a real restaurant deployment of our robot folding napkins for the customers in their back office.

And in addition to folding napkin, because the recipe is quite generalizable, now we also have models doing bunch of commercial tasks. So, here's at a real laundromat in Sacramento, and this time we fine-tune our model to do towel folding for the customer. So, you can see you know, there's a restaurant I guess a laundromat worker coming here to fill the bin over and over, and the robot just kept folding stacks of towels to serve the customer.

So, now let's talk about we have a recipe to master very complex tasks. But, the caveat here is that in all the videos I've shown you so far, we have also collected data at the exact customer site where location that the robot is deployed. But, if you think about scaling robots to any task or to any customer site, then it'll be much better or more ideal if the models can readily generalize the environments they haven't seen before.

So, this is what we have worked on in the I guess the end of last year where we figured out a data recipe to collect a lot of diverse tasks to allow the robots to be able to deploy at a new site without any additional data while maintaining the task performance. So, here's a demo we did at Coral 2025. So, Coral is the premier academic conference on robot learning, and it's held in Korea last year. So, you know, bringing a robot from the US to Korea was its whole challenge that I can talk about offline.

But, the recipe we discovered was able to, you know, we just brought the robot to our, you know, exhibition booth and just dropped it there, and the robot can start folding many, many different t-shirts. And you see that the robot is facing the, you know, the conference attendees. So, many, many times there were people just deliberately trying to mess with the robot, use the t-shirt to cover up the robot's camera, then, you know, all kind of fancy stuff that you see at academic conferences.

But, the model is able to continuously fold t-shirts over and over for 3 days straight at the conference just to demonstrate the ability to, you know, solve this task at environments it's never seen before, bringing it, you know, much closer to the kind of, you know, ideal, you know, go-to-market that you want to have for robotics company, which is just putting the robot to a site and it just starts working. And recently, we have also ventured into many different tasks.

Like, we have a a partnership with Red Bull where we're opening Red Bull cans at, you know, Red Bull events. So, here's a video of our uh robot. Again, uh I guess this video won't play properly. So, but you get the point. We brought the robot to a Red Bull event and it's able to uh open uh these Red Bull drinks for, again, conference attendees over and over without failure, even though in this particular deployment, it's at a music festival, so the lighting's always changing in the background, but the model can continue.

Yeah, but due to yeah, I guess the video will not play. So, just to summarize, uh our mission at AINA is to be able to build general-purpose models and robots that's both competent at many tasks, but also being able to focus and attain really high performance on commercial tasks. And I have demonstrated some of our recent progress on mastering complex tasks and also generalizing the skills to environments is the robots have never seen before.

So, we are a series A stage company actively growing and hiring and if you're also interested in partnering with us for deployments and other things, feel free to reach out and then talk to me after the talk and thank you for listening. >> Awesome. Thank you, Jason. Really appreciate. If you guys haven't seen the data robot in life in real life, have you check it out? I will say see as very impressive performance. Do we have any question or want to ask?

Okay, cool. >> Thank you for sharing the presentation. I wanted to ask if you are already doing kind of like providing developer kits with robots, SDKs, frameworks, etc. for your commercial partners? >> Yeah, so we haven't been working on develop developer kit, but you know, at the current moment we're building our own hardware stack. So, maybe at some point it's on our road map, but not as of not as of now. Yeah. >> I'm curious if you have ventured into education like teaching kids elementary maths or any such use case you tried. >> Yeah, so we haven't looked into education use case, right?

But I think in the future, you know, once our full stack robotics pipelines mature, once we have our own hardware, our data collection, you know, toolkit, I think it's possible and I think it'll be very interesting to venture into education use cases because I also think robotics will only get bigger in the future. So, I think it'll be really really interesting to get children and young kids into the field and the learning how to actually train models to do tasks. >> Thank you for the presentation, really good.

Uh my question is like is there do you guys have a plan for uh kind of taking this technology direct to consumer? Like I saw the use case for the laundry folding laundry. A lot of people don't like folding laundry. I think this is a great use case. So, is that something that you guys thinking about? And obviously there's a cost and all those aspects, but what's what's your plan in the long term on this technology? >> Yeah, so our long-term plan is to develop, you know, like a robot model plus hardware platform that can be deployed anywhere for anyone.

So, that would include, you know, like going directly to consumers. And you know, our models today are able to just fold, you know, uh many kinds of garments. But uh in terms of like go to market strategy, our current focus is on enterprise use cases because I think the distribution channel is like I think it's easier to uh work with uh enterprise customers. Like we can deploy right away and iterate at customer sites.

But for consumer product, I think uh I would imagine that it has to be very very polished and it's less tolerant for mistakes. And there is a lot of privacy safety concerns that we think uh we are trying to not get into at the current moment to uh unblock us from deployments because we think that the bottleneck for AI robot is getting AI and hardware co-working together very very well. And I think commercial environments provide ideal testing ground uh for companies and for the entire field at this stage. >> Thank you, Jason.

Uh my name is Ahmed and I work in a in voice AI in in the voice space. And at least for consumer robots, uh where I believe humans will want to issue voice commands to robots, uh can you talk about how to integrate uh traditional voice stacks, you know, which are intent-driven and uh uh different kinds of models from uh uh robotic, you know, VLA models. >> Yeah, that's a great question. So, we think uh robot human interactions really important, and it at the end of the day, right?

We want to develop models that are teachable. So, it'd be really nice if humans can speak to it, and the robot does the task. So, the way I think about it, so I'm not I don't have a speech or audio background, so what I can think of is like there is very mature speech-to-text models, right? So, you can use that running the on-board to translate human intent or human speech into text very quickly, and then, you know, that's where our reasoning model and our world model comes into play, because both of them are able to interpret text commands.

So, that will allow the model to translate instruction into actions. But, I think in the overall stack, the bottleneck is still on the robot foundation models, because models today our models and other people's model are not ready to just execute any arbitrary commands. So, I think there's still a gap in terms of low-level robotic manipulation to get there. But, I think that's the eventual goal that we want to get to.

Just a model that can be steerable, that can be taught by anyone. Yeah. Okay. Thank you. >> Um how do you think about, I guess, the split of intelligence between the general foundation models that you all are training, I guess, something like a task-specific model for something like laundry folding, and then kind of some of the on-site specific training that needs to happen for a specific deployment in a particular environment? >> Yeah, I think both are very very important, right?

So, the kind of recipe that we have is, you know, we have pre-training, post-training, then we have some of the sort of active learning, right? I I think uh if you think about, you know, humans, right? You know, we have like basic level of like semantic and physical understanding that allows us to adapt any physical task very fast, and I think the best way to even build commercial grade robots today is by doing that.

Because by doing a lot of generalist model training have a general understanding of physical world that's very useful. And what we have seen, so this is something I didn't talk about in the talk is that when the model was recovering from all kind of errors that you saw, a lot of that also just came from interpolating different error recovery behavior that is gathered from doing other tasks, right? Just because, you know, the model has seen thousands of tasks we had some data to recover from all kinds of mistakes is able to just execute that kind of like uh on demand on a new task it hasn't seen before.

But if you're only specializing a model from the get-go on one task, then it's actually missing out on a lot of the physical knowledge that allows the model to be more robust than if you only train on one task. And that's what we have seen consistently. That's why, you know, in the beginning of the talk I also emphasize on having that general pre-training backbone and then doing post-training and active learning on top of that to get to actual commercial grade usability, not on just one task, but also the same recipe that can be repeated across many many tasks. >> Awesome.

Thank you, Jason. Really appreciate. Um that's a wrap for uh data session. Uh if you have more questions, feel free to catch Jason after the session as well. Uh but yeah, this wrap our morning session. Uh this afternoon we have Unity, Skydio, Zoox, uh Waymo, DeepMind. Um so we'll catch you and see you AI as well. So we'll see you in the afternoon. Thank you, everyone. >> [music]