OpenAI just released Codex Voice (It's basically Jarvis) transcript

Open AI just released real-time voice directly inside codex and this is the closest thing I've seen to Jarvis and this is actually just a voice agent that connects to all of my tools, can do basically all of my work, build and edit applications and fully control my computer. And arguably the coolest part about this new feature is you can use real-time voice on the chat GPT app to control codex on your computer and I'm going to show you exactly how to do that today.

In this video I'm going to be using codex with real time voice for the first time. I'm going to test codex real-time for email and other productivity workflows. I'm also going to use the in-app codex browser and excalidraw with my voice. Then I'm going to build an iOS app just by speaking and then finally I'm going to test out real-time voice on the chat GPT app and control codex from my phone. Let's go into codex. At the bottom right you will see this button right here which is start new voice chat and this is what I call the new Jarvis feature which is different than dictating, right?

Jarvis is real-time voice. Let me show you. Hey buddy, can you hear me? >> Yeah, loud and clear. What's up? >> Yo, I need you to go to my email right now. I want you to find any complaints for customer support. I want you to draft a response to all of them and I want you to open them inside the codex browser. >> Sure, I'll take care of that. >> And as you can see here, right? And notice here I can actually um mute the microphone so now I can speak to you as I use it but it can use all of the skills that I've set up when using codex and I've set up so many skills, right?

I have all of these skills that I've been setting up for the last 6 months and now I can use them in real-time voice which is just absurd. Amazing. Yeah, just make sure these are open in the Codex browser, please. >> Got it. I'll use the Codex browser. >> There we go. Look, it opens up all of them up at the same time. Here we go. And boom. Look at that. >> Okay, those 12 complaint briefs are open in the Codex browser, and I saved drafts for 11. >> Okay, one thing that's really cool about this is I can close the side panel, and I can just enter this full-screen mode.

And are are you still there, Codex? >> Yep, I'm right here. >> Very cool. Um can you just actually turn all of these into Don't use any names, just create a new Google Doc for these, and open it up in a new [clears throat] tab in the browser, and and um just so I get a good summary of the issues going on uh with our product. >> Sure, I'll take care of that. >> And I love that I can just see this right here. It's creating file, thinking, getting documents, and I don't even need to have it open.

I can just kind of have this real-time voice just kind of like hovering over everything. It does take a bit to get used to, but I think I will. >> Done. I created a Doc that groups the issues without any names, and it's open in a new Codex browser tab. Take a look. >> Very cool. Can you actually make this document public, and then I want you to text it to Emily Lambert for me? >> Okay, checking that now. I'm setting the Doc to public and sending the link. >> This is pretty cool, so I'm going to open up my messages. >> I've made the Doc public.

Should I text her the link with the note? >> Yeah, here's the customer support issue summary. >> Yep, please do that. >> On it. Sending that now. >> Amazing. So, I just sent a text message with Yeah, this is very cool. Hey, can you actually just add a little extra note there that says like, "Hey, can we get these fixed by the end of the day?" >> Sure, I'll add that. Do you want me to send >> Can we get these fixed by the end of the day?

Send it. Send it. >> Got it. Sending that. I'm sending it out. >> This is Jarvis. You are literally Jarvis. Okay, so now let's test this new Codex voice feature which I call Jarvis in a new environment. So, I'm going to open it up in a new chat and in the browser I have Excalidraw open and I'm going to just going to call this Jarvis features and we can start up Jarvis. Yo, what's good? Are you there? >> Here. What are you working on today? >> Okay, so that was really weird.

So, I if you have a browser open before I open up this voice mode, it'll like automatically close that browser tab which is pretty interesting. Um okay, ignore that. Jarvis, can you please look at the in-app browser right now? >> Sure, taking a look. Checking the current in-app browser state. There he is. The in-app browser is open to your Jarvis features Excalidraw board. >> Okay, hey. Thank you. Thank you. For context, I want you to know that I'm talking about the new GPT voice mode, which is the exact mode that we are using right now.

And so this is a new Codex voice and it's real-time voice. Can you please just give me some three more things that I could add to what it can do? >> On it. I'll check the current capabilities against your board. Here are three you can add. One, it can look at your screen and it can act on what's there, like a doc, sheet, or diagram. Two, it can research something live while you talk and give you the answers there conversationally.

And three, can build and edit real things in the background, like docs, slides, code, emails, or workflows. >> That is actually very interesting. I'm going to start off with doing research. So, you know what we're talking about now, right? We're talking about the new voice feature. I want you to go off, do some research, and I want you to create a new Notion doc. Just put this in your notes in Notion, and use the tab You know the formatting I like with tabs in Notion?

Um and I like hyperlinks. Just do in-depth research on this, please. >> Sure. >> Actually, wait. Wait. I want to I will kind of want to continue talking to you. So, can you actually do this in a different chat? >> Yeah, I can handle that separately and keep this conversation open. >> Okay, so wait. To get this straight, did you just create a new chat so that we can just continue talking, right? >> Yes, I spun that into a separate task so we can keep talking here. >> Wow.

Okay, so it can open up new chats. Yeah, okay. So, we'll wait for that research to come back. Is there any limitations? Like if you look at your environment, I noticed I can't switch the the model that you are. Is there Is there any limitations on what it can't do? >> Checking the current limits now. Yes, this live voice chat is intentionally more constrained than a full task workspace. You can't swap my model from inside this live conversation.

That's set by the task configuration, and I also can't silently take over apps, send messages, spend money, change permissions, or access private data unless you ask. For longer research or building docs or code, I'll often split it into separate tasks. >> really tap into that like you can't send messages unless I ask. What does that mean? >> It means I can draft a text, email, Slack message, or social post for you, but I won't actually send anything unless you explicitly tell me to. >> No matter what.

Okay, so it'll always make sure, no matter what. Like there's no way I can create a skill which bypasses that. >> Checking that. Right. A skill won't override those safeguards. You can set up an automation you explicitly authorize to send under your rules, but a skill can't give itself permission to send things or make consequential changes on its own. >> I see. That makes sense to me. Can you give me a progress report on that other chat session, and if it's done, I want to open up that Notion doc in a new tab here. >> Sure, I'll check the status and open it if it's ready. >> And by open it, I just I just mean I want it in the Codex browser. >> Got it.

I'll open the Notion note here as soon as the task finishes. >> Hey, using the browser use thing, can you add a use case to this board, and can you add spinning up new sessions under use cases under background research? >> Adding that now. I'm adding spinning up new sessions under background research now. >> Hey, can you also say um controlling Excalidraw? >> Yep, adding that as well. The research note is done and verified, and I'm adding controlling Excalidraw now.

Then I'll open the Notion note. >> [snorts] >> Oh my god, it opened it. Look at this. It literally created this research report with the tabs, the exact style that I like. Here's the launch video. It found it on X. I can open it up inside Codex. Play my day list. >> One moment. Anything you'd like me to tweak or add to that note? >> Nah, dude, this is actually insane. You're like my little buddy that can fully control my computer.

Hey, um can you use the build iOS app skill and um I want to create an iOS app. Um can you please start create uh yeah, I want you to create a Swift app that is like an exact replica of the Apple Notes app, and I want you to run it in the in-app simulator in Codex. >> Got it. I'll start by checking what's possible for an iOS simulator run. >> And you know what, while you're at it, can you please spin up two more chats, and um I want one to be a Notion clone, and then the other one to be a Trello clone.

Spin up two more chats and do that for me, please. >> Sure, I'll branch those out, too. I applied configure, and I'm spinning up two separate iOS build tasks, Notion style and Trello style. Both chats are set up with the same simulator target. >> Very cool. Now, let's talk about this. What's your plan for this Apple Notes app? >> Sure, I'll start by mapping the core screens and data model. >> This is pretty cool. While that's working, do you think you can like open up uh the browser and and like search up you know, in a few different tabs like cool note UIs.

I I'm going to get some inspiration and I want you to look these up on Google Images just to give me some inspo while you work. >> Yep, I can pull those up. I'm opening a few visual references in new tabs now. >> Thank you for doing this. How's the notes app coming along? Wait, what did you What was that browser you just had open? Why did you close it? >> I'll check that. The browser was the in-app simulator preview for Memo Canvas.

It was closed by the browser session cleanup, not intentionally. >> All right. Well, please close all the other tabs and then open that up again. Keep it open. >> Got it. I'll do that. Done. All other tabs are closed and the Memo Canvas simulator preview is open. >> Okay, yeah. Can you show me just go ahead and use the app for a bit? Test out the features on this. >> I'll do that now. >> You know what? Um let's continue to make some styling adjustments.

I actually want the default to be light mode. Change it to light mode and um please make the bottom tab like the liquid glass tab at the bottom higher quality, please. >> Got it. Making those updates now. >> Wow, this is just truly insane. Thank you, Kodak. >> I'll pass that along. Dot. Memo Canvas now defaults to light mode and the bottom navigation uses a polished liquid glass capsule with clearer spacing, depth, and a strong compose action.

You're welcome. >> Okay, check this out. OpenAI also announced, right? They also announced that you can use your chat GPT voice in Codex from an iOS app with paired remote access. Android support coming soon. So, if we open up chat GPT, right? We can also connect real-time voice from our phone to this app. So, let me see how this works. I've actually never done this before. This just came out. So, Hey, buddy. Um so, I'm using you for the first time.

I'm using this on remote. And so, apparently you're working on my Mac. Um what uh can you Is this automatically create a new session? What's going on here? >> Let me check how the session starts. You're in a new voice session, but it can still be connected to the same task in your Mac's workspace. I'll only create a separate task if you explicitly ask me >> Okay, so let me create a new task real quick. I want you to create a new task.

And I just want you to just give me a quick summary of the support emails. Just I just want to see if this works. >> On it. I'll take a look. I'm creating a separate task for a concise summary of of the support inbox using the Gmail workflow. The summary task is set up in a separate chat and checking the inbox now. >> What is the chat called? What is the chat called? >> Checking that now. It's called summarize support emails. >> Okay, so I see it.

Nice. Okay, so now I I'm basically just talking to a voice agent that runs, but I can create you can create new chats in the background basically, right? Is that how this works? >> Let me check how chats are created from here. Exactly. You can keep talking to me here, and I can create separate background tasks that show up as their own chats while I stay here and coordinate what comes back. >> Wow. Can you see my So, you can like look at my other tasks?

Like, can you check the status of my memo canvas UI polish app that I was making earlier. >> Okay. Yep, checking that now. That's the memo canvas app task you made earlier, and it's active. The latest light mode build with the liquid glass tab bar is still running in the simulator. >> Okay, thank you. This is actually blowing my mind. Okay, so I'm absolutely blown away. So, here's just a list of what we did. This was basically the first time I tested this.

It can do It can fully control your computer. It can spin up chats in the background. You can use real-time voice on your phone, and that will talk to your computer, and it can spin up chats, and you can fully just control your computer in real-time voice directly from the ChatGPT iOS app. This is pretty insane. So, obviously, I've only spent like 30 minutes to 45 minutes using this. So, I'll probably do a much more in-depth video once I kind of learn the best practices, right?

This was just me testing it. But, I want to learn actually, how do I use this in my business? How do I use this to stay focused and be more effective? And so, I'm just going to test it for the next few days, and then I will make a more complete guide-style video on exactly how I use this in my business, and I'll see you here for the next one.