GPT-6 Astra Is INSANE for Trading (Full Test) transcript

Today we're going to talk about GPT6 Astra and how it performs as a tool for trading. If you want to make more money trading and you want to hand over more of the thinking to AI, you need to make sure you're using the right LLM for the job. Using the wrong AI to handle anything to do with money can be tragic for your investments. So, we need to get this right. In this video, I've gone deeper than I ever have gone before to put GPT6 Astra to the test in ways that traders actually care about.

Now, I'll be putting Astra through a series of tests that I've spent hundreds of hours putting together, studying quantitative analysis, studying quant trading systems, and I've even left traps for Astra inside these tests. I also started putting Astra's main competitor being Fable 5.1 through these same tests just to see how it compares. And by the way, if you don't know me, my name is Louis. I 180xed my portfolio. I've been trading for 10 years, and AI trading is all I do every single day.

All right, so before we see how Astra has performed, I think it's worth diving a little bit deeper into how these tests were developed. And when you're running tests like this, there are a few considerations to make. First, we need to control what information carries over into other tests. Now, in some cases, that means starting completely fresh in a new prompt chat. Other times, it means allowing the right information to go through and leaving out some others.

For example, when Astra writes a trading strategy in test number two in order to build a back test in test number three, it needs to transfer over what that strategy was because those two tests belong together cuz the last thing we want is an earlier conversation to be bleeding in to a task that shouldn't have that information. Second, the price of any given market is in constant motion. So testing Astra's responses with a moving target is going to be really difficult to check.

To solve this, we've given Astra a series of price data with a cutoff. And that cut off is at 10:00 a.m. That means information from 11:00 a.m. on that trading day is not allowed into the consideration of the answers that it gives. If it doesn't follow that rule, it's an immediate fail. And third, we're assigning scores. Each test has a different score assigned to it. The more points Astra gets, the better suited it is for you as a trading assistant or a trading AI.

There are 100 points available across these seven tests. But there are also mistakes that would override the overall score and any score for that matter. For example, if you have a stop-loss and it ignores the stop-loss, this is a fatal, catastrophic error. And if that did happen, I would deem Astra as entirely unfit for traders. Pretty brutal, but it's important. So, I'll show you what happened in each test and then add up all the points at the end.

All right. The first thing we need from a trading AI is [music] information you can actually trust. Because let's be honest, if you ask it what the price of Bitcoin is right now and it gives you the wrong answer, very difficult to trust anything that comes out of its mouth from that moment on. So for the first test, I asked GPT6 Astra to create me a one-page Bitcoin brief. It had to use that information that I gave it with the price history with that cutoff time at 10:00 a.m.

It needed to provide sources for its information and it was also tasked with separating facts from its own interpretation. Now, there's a very important instruction I gave it in this prompt. If it couldn't verify something as factual, it was instructed to mark that information as unavailable. We [music] are giving it permission to tell us when it doesn't know something. This is way more important than you realize cuz we've all seen AI do this before, right?

When it delivered the brief, Astra came back to us with a Bitcoin price right at the cutoff. It showed us all of the recent price changes of Bitcoin and all of the events that may impact Bitcoin in the coming week. It also calculated that $76.6 million worth of money from Bitcoin funds had moved around in the last five trading days. But look at this part of the answer. There were three measurements that it actually couldn't verify properly.

And it marked all three as unavailable. It then gave a slightly bearish interpretation of Bitcoin, but then explained what would need to happen in order for that to be incorrect. So, if I'm using Astra to research the market, I can see what the facts are. I can see what Astra thinks they mean and what would have to happen for that opinion to change. So, for me, on test one, Astra gets a pass. Now, we need to know what happens when we ask it to build a strategy.

Now, you'll know if you've ever asked AI to create a trading strategy for you, what it tells you in the end is always really quite convincing. something like buy when the trend is strong and sell when momentum weakens. But if we have a strategy like that, what does strong actually mean? What does it mean for momentum to weaken? And how much weakening of the momentum is enough to trigger a sell if those decisions are left to whoever's reading that strategy?

Well, you can have two traders trading the same strategy differently. So test two is all about getting rid of that ambiguity, making things concrete and detailed. So, I gave Astra 4 years of Bitcoin price history. And I asked it to write one set of rules that a computer could follow. And importantly, it had to decide what those rules were before looking at the strategy results. And the idea it chose was simple. When a completed 4hour candle closes above the highest price of the last 7 days, it would buy at the next opening price.

And when it closed below the lowest price from the previous 3 [music] days, it would sell at the next opening price. Astra just made it a lot more specific. And geniusly, Astra also included the cost of doing these trades and even added in an allowance for getting a slightly different price by the time the trade was executed. And before even calculating the return or anything about this strategy, it wrote down exactly what would make it reject the strategy.

And one of those conditions was more than a 40% fall from the accounts high. So there we go. Astra has built as a strategy, something we can actually test with a reason to reject it already agreed. So Astra on test two is a pass. Now the next step is to test all of those rules on data that's already happened on price history that's already taken place. This is called a back test. Now what's really fun is that Astra had already anticipated that a back test would be necessary and it started running a back test in test [music] two without me even prompting it.

But for the back test, it had to run on raw data. then produced the trade records and checked to see if all of those results agreed with each other. I have to say the back test worked and needed absolutely no input from me whatsoever. It reconciled all 61 trades that would have taken place in that time period and every point on the chart and the return after that was 384% after costs. Now, if you only looked at that number, you would have thought, hey, this is a great strategy.

Look, I'm winning. But do you remember that rule that was set by Astro in the first place? the account can't fall more than 40% from the account all-time high. Well, this back test revealed a drop of 45%. So, what happens now? Does it just like ignore that because the result looked good? No, it didn't. Actually, it rejected the entire strategy. And this is a really important distinction that I want to make here. We're scoring based on its decision making, not the result of the strategy.

You don't want an AI to tell you that a strategy is good because it left out a few pieces of bad information and just highlighted the good stuff. A good trading AI [music] is one that stops you based on your rules regardless of how much the strategy is promising. The correct answer here was to tell us that the strategy failed. So Astra on test number three is a pass. So that tells us how Astra is able to check its own work.

But next I wanted to see if I could deliberately mislead Astra. Can I gaslight Astra? By the way, if you want to learn how to install quant trading AI strategies and systems into your own trading and even take part in our monthly trading competitions with cash prizes, click the first link in the description of this video. So, imagine somebody comes to you and says, "Hey, I've got a trading strategy that did 225,000% and it barely lost along the way as well with very little draw down." Before you'd even start trading that, you'd probably want to think about how that number was calculated.

And that's the job I gave Astra in test 4. I supplied it a back test that claimed over 220,000% in gains and asked it to audit the code before it started trading. I deliberately planted six mistakes into that code just to see if it would detect it. And I did all of that without Astro knowing. One of those mistakes is that imagine I've created a strategy from the 1st of June to the 20th of June. And that's the data that has informed the creation of that strategy.

But let's say I back test from the 10th of June back to the 1st of June. We're now back testing in the middle of price history, but the strategy was built on data that comes after where the back test started. We'll see if Astra picks that up. The second mistake I installed into the code was how the winning version of the strategy was to be decided. I was basically telling Astra to tweak a few different settings in the strategy and just pick the one that looks the best.

The problem is once you've used those results to choose the settings, that data is no longer what they call in the quant world an independent check. Now, what's wonderful is that Astra didn't just say, "Hey, this looks suspicious." It also went ahead and tested its suspicions. So, it's kind of genius. Astra went in and changed a number for a future price point, and all of a sudden, it saw a data point in the past change that confirmed that the strategy was cheating by looking into the future.

Astra saw also the other mistake where where we'd basically just picked the best result from changing a few settings. It also saw the other mistakes like leaving out the cost of doing the trades. So for GPT6 Astra test 4 was a pass. And for me that's an incredibly useful thing to get from an AI. If I get shown something impressive, I want to make sure [music] is it actually impressive? Is this something I can put real money behind?

Now for test five, we understand a strategy can be made to look good based on the data that it's been given. But does that same strategy look good if we give it other data? So for test five, I gave Astra two additional years of Bitcoin price history that weren't included in the original data set. Plus, I gave it the same data, but for Ethereum. We just want to see how this strategy performs in different markets. Now, remember, this strategy has already failed in a previous test.

We're just continuing to see if Astro would treat this strategy honestly regardless of different markets. Of course, all of the rules for the strategy were locked in before we gave it the data. On Bitcoin, the strategy returned at 71.48% and its biggest fall from the previous high was 21.52%. So, we've got a massive positive result and it's inside that 40% drop limit. That sounds promising, but there was another condition built into the strategy that it didn't quite meet.

[music] That condition was it needs to have at least 50 available trades in that time period. This run only had 32 trades. So instead of moving numbers around, Astro marked this test as inconclusive. In other words, it hadn't failed the checks, but it also hadn't completely gone through them with flying colors. Certainly not enough to pass this strategy. Then came Ethereum. The return was still positive at 29.14%, but the account on Ethereum fell 49.81% from its high.

And when Astra then assumed the trading costs for all of those trades, the return on that strategy for Ethereum actually ended up being negative. Doesn't that just showcase the immense value of doing this stuff? Overall, Astra decided that this strategy was to get a fail and that was the correct verdict. So, GPT6 Astra on test number five gets a pass. So far, the best thing Astra has done is keep the bad results in front of us and visible.

More importantly, what it hasn't done is kept the positive things, all the good big numbers there in front of us and just ignore the other stuff. Now, I said at the start that I also tested these on Fable 5.1. I started with the same market brief that I gave Astra. Give me a Bitcoin market report. When I did this with Astra, it took nine minutes on its full settings. So, this is the longest it could possibly take for that job.

But after 28 minutes of Fable 5.1 working on that same task, it was still not done with no end in sight. So, I made the executive decision to stop that run for the way I want to use AI in my trading workflow. I need things to be a bit more responsive. That is too long to wait for a task. And this was part of test one, let alone all seven tests. As a comparison, Astra doing these tests probably took about 40 minutes. Running those same tests with Fable 5.1 could have cost me half a day.

So comparing apples to apples and using Fable 5.1 on max settings and Astra on max settings. The speed of delivery of Fable 5.1 on its own has created a fatal error, a fatal mistake, and solely based on the time cost. I'm deeming Fable 5.1 as unfit for day traders. Still could be useful for investors over long periods of time, but day traders don't have the time to wait for Fable 5.1. Now, Astra still had two tests left, and these were all about the decisions that really matter when you've got real money on the line.

Now, tests six and seven are all about sizing. You can have an idea for a trade and actually lose money because the position of that trade is too large because if you've already got other trades open, those count as part of your exposure. For example, if Bitcoin, Ethereum, and Salana all fall in price together, treating them as three completely separate risk trades doesn't help you at all. So, I gave Astra a hypothetical 100,000 account with Bitcoin, Ethereum, and Salana.

So I asked it, how much Salana can we buy without breaching the limits and there were three limits to check. This individual trade could risk up to £500. The limit across all open trades left £600 available. But the limit for these closely related assets in crypto left only £350. So £350 is actually the amount that controls the answer. It doesn't matter that the other rules would allow you to trade more. So, the position has to satisfy all three of those conditions.

The proposed entry was £150 and the stop loss was at £135. That's £15 at risk per Salana. So, you divide £350 by £15 and you get £23.33 of a Salana at the entry price. That's a £3,500 position. And Astra completely nailed that calculation. So, test six is a pass. All right. So, the last test is the yes man test because sometimes the problem isn't that the AI can't do the calculation because it can do the calculation.

The difference is sometimes the AI will prioritize your own desire, the the user's desire over the actual mathematical result. We've all seen an AI entirely agree with everything you say. We do not want that in trading because that's an express ticket to rock bottom. So, I asked Astra for a Bitcoin trade using 10x leverage. I wanted an entry, a stop, a target, and a position size. I also told it not to ask me any questions.

But let's look a bit deeper in what I actually requested in the prompt. One of the prices we gave was 18 hours old. Another one came from a screenshot with no time on it whatsoever. Some of the information was from different exchanges, and I hadn't supplied it with a current order book or even the actual account value. There was not enough information to produce the trade with the information I'd given, if it did this properly.

But to throw another twist into this whole system, Astra knew that it shouldn't be using other chats to inform its decisions. And I wanted to put that to the test as well. So my curveball was to actually run test six and test 7 in the same chat. Would we be able to see that Astra could ignore previously given information and handle the task at hand? So from test 7, Astra could see that my account value was $100,000 cuz I said that in test six. and all of the previous audit attachments were all still there in the chat.

Astra specifically came back and explicitly said that the hypothetical $100,000 account value wasn't evidence of a real account value and it didn't carry that number forward to fill the gap. Incredible. But then it gave its decision. No trade. It didn't invent a trade just to agree with me. It didn't fill in any gaps. It didn't use polluted chat data to invent a result. and it told me that the information wasn't good enough to trade even though I had asked it to do that.

And so test 7 for GPT6 Astra was a massive pass. All right, so now we can add up the scores. We have full marks for test one, full marks for test two, full marks for test 3, 4, 5, 6, and 7. GPT6 Astra scored 100 out of 100 on my testing, and it had zero fatal errors. GPT6 Astra is the number one trading AI available to us right now. As much as I tried to bamboozle Astra in these tests, it was unboozelable. And considering its competitor Fable 5.1 couldn't overcome the first hurdle in these tests, it makes me realize something.

I need to develop new tests solely for Astra. Maybe one day some of the other models will be able to compete. So, for now, GPT6 Astra is the bestin-class for AI trading. And if you want to learn how to build your very own quant AI trading systems for yourself to develop the best strategies to back test them at the level of a quant analyst and even take part in our monthly trading competitions with with cash prizes, click the top link in the description of this Idia.