I Built a Gold RL Trading Bot With 60% OOS Returns [Free Code] transcript

This is a reinforcement trading bot that put itself to trade gold on 20 years of data and achieved 60% returns on out of sample or data it had never seen before. That's either a real edge or one lucky run. So, this whole project was me trying to tell the difference and I'll show you exactly how. Quick version, if you're new, this is a reinforcement learning agent. I don't give it any rules. I let it trade gold on years of historical data and it will learn from experience the same way you'd learn a game by just playing it.

In a previous video, part one, we laid the foundation, we made a lot of shortcuts. This time I'm pushing harder, training on gold, then testing only on data that was never seen or touched during the training. So, we also call it out of sample data or testing data. Then I stress test the uh result until I'm convinced it isn't just a fluke because the real goal here is trying to turn this into something we could actually use for trading.

If you are interested in the coding part, the whole project is coded in Python and the link is in the description of this video. It's free, so you can download it and rerun every experiment yourself. But before we move on to the coding part and the results, we need to look at three things that decide whether this bot sinks or swims. Get any of them wrong and the results will completely fail. First, we need to check what the bot can even do.

Its choices are deliberately simple for the sake of this video. It can buy, sell, or hold. When it opens a trade, it sets a stop loss based on the ATR at that moment. So, the risk adapts to how volatile gold is right then and it sets a take profit as a multiple of that risk, so a profit to risk ratio. That's it. Those are the only levers it can pull, but that's already a lot of combinations of choices between entering the market or holding, two different directions, uh different sets of stop losses, distances, and take profit distances.

So, everything the agent learns is just reading the market and using those few actions as well as it can to grow the account. You could make this far more complex, more actions, more nuance, but for this video, I'm keeping the decision set small and clean so we can actually see what's working. Second, and this is where the is it luck question really lives, how we split the data. The classic way is one train, validation, and test split.

Simple, it's where I started, but my gold data goes all the way back to 2003 and a model trained mostly on 2003 isn't necessarily learning anything useful about today's market. So, there's a big gap between the a time gap between the training date and the actual using or testing of the model. And remember that every curve I was getting came out positive, that's exactly when you should get suspicious. One good run proves nothing, I need to know if this is a repeatable observation.

So, instead of one split, I used anchored walk forward. Train on a chunk, validate and test on what comes next, then expand the window to include that new data and do it all again, rolling forward through history. If most of those folds come out positive, then that's the fingerprint of a real edge, not just a lucky coincidence in one specific case. But, there's a still a catch. In the later folds, the most recent folds, the model is still dragging all of the ancient 2003 data which might be completely irrelevant to the market it's now being tested on.

The fix is a fixed-size sliding window. Train, validation, and test all move forward together, and the oldest data drops off the back, so the model always learns from a recent, relevant slice. In my opinion, this is the best setup for a trading model like this, and it's what I'm using for this video. Third, and the most important of all, the reward function. This is the heart of reinforcement learning. The agent only learns by chasing a reward.

So, whatever you put in this function is literally what the bot decides to become. Get it wrong and it will happily learn the wrong lesson. Ours is a bit simple for the video, but it's totally enough for the trading. Just three terms for now. Term one answers the question, did the trade make money? Whenever a take profit or stop loss hits, your equity changes. We divide that change by the amount you risked. So, 0.5% of your equity, for example.

So, a take profit that earns exactly the amount you risked scores plus one as a score. A stop loss scores minus one and a bar where nothing closes scores zero. So, if there's a win, you get a reward. If there's a loss, you get a negative reward. If there's nothing, the model gets zero or the agent gets zero. Term two, are you winning right now? So, while a trade is open, this is happening while the trade is opened, so it's on the market, and nothing is resolved yet, so the the trade is not closed, we add a tiny signal proportional to the unrealized profit or loss.

It's uh like a gentle nudge each bar telling the agent whether the open position is moving in its favor. The weight is deliberately tiny. You can see it's 0.01 here, so it can never dominate the real reward, which is the first term, term number one. It just breaks the um dead silence while the trade plays out, and it pushes actually the agent towards opening trades that are already winning before they even close. So, it kind of favors those trades that are highly uh profitable or they have high chances to stay on the profitable size, provided this is even a possibility in real life or real trading.

And the third term tells the agent to stop sitting in trades forever. A tiny constant, it's -0.00002, as you can see. So, this is the term in red in our formula, reward function, taking every single bar the agent holds a position. So, on its own this is a very tiny amount, so it's not going to change much, but if the agent holds uh trades for tens or hundreds of bars, then this is going to add up and it's going to be shown in quietly shown somehow in in the reward.

So, quietly pushes the agent to prefer trades that resolve quickly over positions that are going nowhere and taking a lot of time to close. We don't want to open trades for two or three weeks. We prefer trades that can resolve in few hours. So, this is what we are telling the agent using this term right here. And here's the thing to take away. Change any one of these terms and you change what the model learns and how the results will look like.

So, that's the fun part. If you download the code, start tweaking this function. This is the easiest part. You can start tweaking this function, run the whole pipeline, and watch the behavior shift. Now, just a heads up though, training such models, especially if you use multiple folds, might be time-consuming. The sliding window approach, for example, using multiple folds, in my case, took me around 36 hours of compute time, multi-threaded over four threads.

So, I'm just using an i5 this time. It's an i5 Intel. But, uh I mean, it's totally doable. It's nothing that you need a big server for. So, how do we select our champion model? Because we have multiple folds, we will have multiple models within each folds. Uh we need to avoid overfitting, but also choosing the most adapted model. Inside of each fold, the model is not saved only at the end. As it trains, we save many checkpoints.

Every checkpoint has to pass two small scores, one on the tail end of the training data and the other one on the validation data. These scores are represented by the blue train eval reward curve and the red validation reward curve that we can see here on this on this figure. Again, the out of sample test data we still have as a slice that we kept unseen from the model so that's not what we're talking about here. This is just the training and the validation slices.

And by the way, here we can see how the over-fitting is setting after a few steps of or after 2 million steps of uh training. So at first we have the training reward that is increasing as you can see it kept increasing that's the blue curve while the red curve is the validation and it kept increasing as well. So this is where the model is learning how to trade but then it started decreasing. We have this divergence between the training and the validation and this is happening in this zone around 1 million up to 2 million steps.

So we can safely go up to 2 million steps. The models before this number of time steps is learning but after this step the model is just memorizing the data it's not what we want. So for each test we calculate a score how much cumulative reward the model made minus a penalty if it had a big drawdown we can see the drawdown curves in dashed lines on the training and the validation slices as well. So it's not enough to make money the model is also punished if it makes that money in a dangerous and unstable way.

So to choose the model we're taking into account the reward the validation reward and the drawdowns in both cases. Now here's the important trick we don't keep the checkpoint with the best average score. We keep the checkpoint or the model whose weaker score is the strongest. In other words, we're looking for a model that does good in both training and validation and we only let the models compete between each others on the lowest or the minimum reward in between training and validation.

So using the worst reward between these two categories. Think of it like a student taking two exams. If he gets 95 on one exam, but 20 on the other, we don't say he's excellent. The low score tells us something is wrong. Maybe he got lucky, or maybe he only understood one part. Same thing right here. If a model looks great on validation, but weak on the tail end of training, I don't trust it. And if it looks good on training, but fails the validation, I don't trust it either.

And only after choosing that best checkpoint, we test it one time on the held out test data, data that it has never seen, and that was not used to pick the model. But, we still don't deploy it yet. We still don't have a champion. After all the folds are finished, we apply one final filter, the consistency gate. This gate asks a simple question, did enough folds make money independently? Because one profitable fold could be luck, two profitable folds could still be luck, but if enough separate folds across different market periods are profitable, then the edge starts to look more believable.

If the consistency gate passes, we promote the model from the last fold as our champion. Why the last fold? Because it was trained on the most recent data, and it has this advantage. And if this gate filter fails, then we don't have any model that is selected. We need to go back to features engineering and changing the reward function and the hyper parameters of the learning model. Which leaves the one question we started with, does it really work?

Let's take a look at the results and find out. Now, just for the sake of keeping this video as brief and simple as possible, I'm not going through all the codes or the files or the Python scripts. I'm going to show you this Jupiter notebook file where I test the best model. So, this is the model was already selected using the gate and selection we already mentioned before, but this notebook is for testing on a slice that wasn't seen yet.

So, it was completely locked and it's going to show us the performance of the model that was selected on on this. So, we we do the analysis. We have the train and validation, etc. So, we got our data for the model. And then we're going to load the model and evaluate on all three splits to to see how it trained, how it validated, but also how it tested on unseen data. Now, the final model, I have two models here. As you can see, I'm going to increase the size.

Maybe I can show you better and maybe we can close this one. So, the final model is not the best model. It's the overfitting model and you can see it. We have a plus 86% on the training. Then the validation is dropping to plus 12% and then we have just plus 6.9% on the testing and sometimes you might get something negative here, which is not surprising because this is basically an overfitting. The last checkpoint run through all the time, the millions of time steps.

I think I put 5 million as a maximum and this is not what we want. We want the last learning model before the overfitting starts settling in. So, we have this one that was selected by our criteria. Again, we have plus 87% on the training, the validation as well and the testing actually, we got exactly as the validation. Now, don't be surprised if you get 87% on the training. That's totally normal because the training slice is is wider in time.

So, you have more time. The model has more time as well to do more returns. So, this is not the annual return. We need to do the annual return to be able to compare the percentages together. But we can compare the number of trades and guess that this training phase was most of the data and then a small slice was only for validation and testing. The good thing is that we have on the validation 307 trades leading to 15% in returns.

By the way, I took commissions, I took trading fees, all is included. What we are doing is that we have 1-minute time frame data and then we are resampling into H1 into the 1-hour time frame. So, the trading happens on the 1-hour time frame, but we backtest on the 1-minute time frame in order to resolve those uh uh specific cases where you might hit the take profit and stop loss within the same H1 candles, so you don't know which one was hit first.

Going down to 1 minute uh will be resolving this just for the sake of being precise in the backtesting. So, I also built a very silly model using EMAs for comparison. That's what we call the baseline evaluation on testing the performance summary. So, we can see the baseline, uh the testing, and the validation, and the training uh final model, as well as the uh best model. And since we are including the maximum drawdown as well based on the validation and the testing, we can see that the model made 15% on unseen data and with only minus 6% of maximum drawdown.

That's not even the average drawdown, that's the maximum drawdown. And the um returns per year is 38 or 39% while it's 35% on the validation. I'm going to show you the curves. I think I have some curves here. This is like almost 5 years of training. Then we have the validation, that's the best model again. So, the best model does see the training, does see the validation at some point because we are tuning to get the best validation results, but the test data is kept unseen.

And then we are on the model only once on the test data, and as you can see, it's positive. Now again, if we have a a good model, it should run on most of the folds, and it should score positive on most of the folds as well. Also, in the same Jupiter notebook, we can see the monthly returns. So, I can see from December 2025, Jan 2026, and so on. So, we have some winning months, some losing months. So, overall, it's looking okay.

And you might think that gold was mostly going up, so it's it's a long signal most of the time. So, the model might learn how to long the market most of the times. But, that's not the case actually. As you can see, we have a lot of trades, long long trades, long entry positions, but we also have a lot of shorts. And this is how the exit reason is broken for the long and the short positions. We have stop loss in red, take profits in green, and the flip closes when we have an opposite signal.

The model detects an opposite signal, and you have a manual close as well, probably when the the trade is taking too long. So, it's going to close it manually because this is what we included also in the reward function. We can also check the behavior of the agent. So, how many times did it trade? The number of trades, for example, with the take profit twice the stop loss distance. How many times the stop loss was 1.5 or two times the ATR or one time the ATR and so on.

So, mostly mostly it was this one this part. So, we have 811 trades where the stop loss is equal to 1.5 times the ATR, and the take profit is equal to twice the stop loss distance. So, two to one reward to risk ratio. And we can also check when the entry timing and the exits actually were happening. So, is it a Mondays? Is it on Tuesdays? And so on. And you can have the time of the day as well. So, mostly it was in this area.

We can see we have a high concentration here. It could be probably, for example, an overlap between the London and the New York session. I also was plotting the rolling sharp ratio just to check how things are behaving with the equity, but mostly I also like to plot the chart with the trades. Let me re-plot it here. So, as you can see, we can visualize the chart with the trades, the entry positions. These are the triangles, red triangles are short signals, green triangles are long signals.

And then for every signal, you have those dashed lines where is the red line is the stop loss level. The green line is the take profit level. And then we have those dashed lines showing the entry and the exit. So, this is a trade that was exited here hitting the take profit. If you have a gray point, that's a manual exit, which means that the agent didn't wait to hit the take profit or stop loss. It took too long for the trade to to close, so probably it was closed manually at some point.

So, this is the gray manual exit right here. Yeah, so you can visualize any slice of the time, see how the agent is is trading. I'll not spend a lot of time on on the notebooks. You can check them out on your own. But, if you are looking for the reward function and you are intending to rerun the pipeline, it's in the environment bracket.python lines 297 to 300. And this is our reward function. This is where I would start experimenting and having fun.

Very important thing that I actually should have mentioned at the beginning of the video. These are the features. What is the model taking as input features? Nothing is an absolute value. I'm not giving the open close and high and low prices. This is meaningless for reinforcement learning, for any machine learning actually and neural networks. You need to give something that's relative, so a relative value. So, the ATR is okay to to provided.

The RSI as well because it changes between let's say minus 50 or plus 50 or between 0 and and 100. So, it's not an absolute value. It's a percentage. It's a relative value. Then we have the time, the sign, and the cosine of the time, the exponential moving averages. I'm not adding them as they are. So, of course, we compute them here, but then they are not fed into the model. We check the closing price minus the EMA 20 divided by the ATR, for example.

So, it's not really the ATR on its own. It's not the EMA, absolutely not the EMA on its own. It's a difference between the closing price the EMA divided or normalized by the ATR. Or also candle shapes as well, for example, the range of the candle divided by the ATR. So, the high minus the low divided by the ATR and so on. Most importantly, do not use plain or absolute values, absolute closing or high and low price. This is meaningless.

If I tell you gold is, I don't know, $4,100 now, it doesn't mean anything. But if I tell you the range is, I would say, I don't know, $10 per hour, that would mean something for the volatility, for example. And yeah, thank you for your patience and for staying that long. That would be it for this one. I'm thinking of a part three video where we dive more into the details and maybe try to improve this even further. I think we're getting into somewhere where it might look like it's working.

I hope we're not making any mistake anywhere. I checked for any kind of data leakage or information leakage. I couldn't find any. Everything was corrected. So, if you can help me validating this code, it would be really great. Leave a comment in the comment section anytime you have any idea. So, we can even push this further, in my opinion, using LSTM, maybe the long short-term memory with the reinforcement learning models.

Maybe considering taking a time snapshot of the data instead of a single point indicator like we're doing here in this video. But first, I will be waiting for your reviews and comments. Maybe I can get even better ideas from you that we can include in the video. Until our next one, trade safe and see you next time.