This Is What The Meta Ads Algorithm Is Doing transcript

people think about creative testing wrong at the Baseline and so what I want to talk about right now is how to think instead about what the machine learning for meta ads is actually doing to forecast the performance of your ads and to do that you need to understand the difference between probabilistic forecasting using Basi and logic and a randomized controlled [Music] trial so this is going to sound like I'm getting into the weeds for a minute but let's just start with a sort of scenario okay um the scenario is like this you see a an ad let's say you launch an ad and this has happened to anybody who's ever run cost controls okay again if you're new to my podcast you need to go back and look I have a bunch of episodes about running bid caps and cost caps and tros targets and I've showed it in some opening the books episodes so go check those out there's a bunch of stuff that you can go go look out there uh but uh if you're running manual bids okay and you're running cost controls there's this experience that happens all the time right you see an ad let's say it spends $100 it gets six purchases and you're running and let's $50 average order value you're running a three to one because you've got $300 let's say actually it creates eight purchases okay um and you're like this is awesome this ad is smashing um I'm running a four to one Ros on this new ad but then this thing happens that's really annoying which is that it doesn't keep delivering that ad instead meta will prefer and deliver other ads even if you've set a $50 cost cap okay you're getting a $50 CAC meta will prefer other ads that are running at like a one and a half to one um and you can't figure out why you can't figure out why it's the case that meta is preferring these other ads even though this ad that you're running that you're running a create you know you launch new is is generating this massive return for you and you're like I don't get it there's eight purchases there it's telling you something really significant okay on the other hand there are ads that run so just keep that as scenario a on the other hand and uh hand there are ads that run that are spending up to $500 and they're running at a 75 or $1,000 they're running at a075 and you can't figure out why meta keep spending on those ads when it's below Rass Target or or above your CAC Target okay and this and and there's a third scenario let's let's so let's do there's scenario a scenario B let's do scenario C here okay and there's a this is this is the one that's maybe the most frustrating for people is an ad that spends a few dollars a day right it spends five dollar and that's it total across the first week it spends $10 $20 Etc and that is maybe the most aggravating of all because you went to the pain of developing this creative and you can't figure out why it's not spending and what almost everybody will say in all three of those scenarios what all of us will do is we'll look at and this is definitely my instinct as well uh on my own what we'll do is we'll look at um the purchase Behavior and the click behavior on these ads and we will say I don't understand why meta isn't taking that as a stronger signal isn't purchase Behavior the most important signal of value that you that you can get on an ad okay now um this is where the problem fundamentally lies and and with scenario C where it's not spending at all there's this thing where you go you know if if purchase behavior is the most important thing you go like I don't understand how can meta possibly decide not to spend more on my ad if it's only spent $ five or $10 like it can't possibly have learned anything significant about what's going on with my ad and here's my basic contention about that is going to drive everything I'm gonna say here that meta knows way more way faster based on what appears to you and me to be way less data than uh than seems possible okay in including that scenario C $5 scenario $10 scenario okay if you are thinking of hiring anyone new in your business right now pause for a second and consider doing it in the Philippines with my friends at more Staffing if you're thinking about hiring in the Philippines as like $5 an hour virtual assistance only you are missing out on an incredible opportunity to make your business better with Incredible Talent at a price that is much more affordable than equivalent talent in the US and I'm talking about manager director even executive level talent I have hired seven people in my business at this point with more Staffing I love them they have made my business way better go do it right now more Staffing will connect you to the best talent in the Philippines great English speakers people who they will recruit train onboard coach and they'll even give you a one-year guarantee so if the hire doesn't work out they'll replace them for you at no additional recruiting cost go to Mor staffing.com today what I believe is $5 or $10 tells meta a lot more than we think um okay now let's put aside one possible way of making this case one possible way of making this case is that meta in fact is reading the ad reading the content um you know with machine learning and and AI basically uh uh in advance and is predicting actually in advance whether your ad will work even before it delivers it okay I think it's possible if not actually even likely that meta is doing something like that but I'm just going to leave that aside okay so that I do not need that to make the case that I'm going to make right now uh I think instead it's possible or so yeah okay so I do think it's possible but we'll just leave that aside okay I want instead to think for a second about how we evaluate the performance of ads okay most people let's start with sort of the alternative to my approach to doing this most people approach creative testing campaigns and they have to make a decision about how to know when an ad is a winner or loser so the standard ad setup for for a lot of advertisers is you build creative testing campaigns where you force spend on an auto bid to a bunch of new ads that you make okay and as you make a bunch of new ads you analyze the performance of those ads and then for the ones that are winners you move them into your scale campaign okay that's a separate campaign so you take the ad you kill it in your creative testing campaign you say this ad is performing well in Creative testing I'm going to move it into scale okay and now my scaling campaign is going to make it work okay so so that is like that is like the classic way people run these ads my alternative to that is to eliminate your creative testing campaign instead launch everything with a manual Bid And if meta spends on it it it's a signal of success if meta doesn't spend on it it's a signal of failure okay that's it that's all you do if that is uh uh so okay so one of my critiques of creative testing campaigns is this how do you decide when an ad is a winner and ought to be moved to the scale campaign okay how do you do that and uh my experience is that most people have act actually like no real rigorous methodology for doing this at all okay so they they they sort of go like oh had a few purchases uh you know CPC is good hook rate's good whatever something like that normally it's about purchases conversion rate and they say so that means it's a winner and So within a few purchases you go and scale it and put it in your winning campaign now there are manifold problems with this way of doing things like first of all if you're running scale campaigns as cbos or as asc's where there's already some way and it's a lot of people are for meta to prefer uh to to rank which ads is going to deliver why not just launch it there in the first place if you're going to use so few purchases to determine what a winner or not is okay but secondly it's that so few purchases tell you so little every one of you watching or listening to this at some point has seen an ad like a scenario a situation right where an ad gets a bunch of purchases really fast and you're like boom we solved it this ad's producing a four to one scale the budget it's working and then the moment you scale the budget it stops working right if you're doing it an autobid setting I ran ads like this for so long I would just be so excited about an ad and you know I have all this bias I make the ad I think the headlines awesome I think the visuals are awesome and so it starts to perform well at all you know I launch it and I'm checking it every two hours opening it up like everybody's who's been around media buying for a little while knows that feeling okay and so um and so we have all this bias going into it and you scale it and it doesn't work and it's like wait why didn't it work well that that error is a problem all the time and the reverse error is true too which is what about that ad that is spending but getting no purchases um you know how do you know you should shut it off right because um because if it's the case that sometimes you scale an ad you push spend to an ad that looks like it's going to perform scenario a that I mentioned earlier but it doesn't actually perform as you scale then it's probably also the case that some ads like scenario B that I mentioned earlier that some ads um look like they're not going to perform but actually could perform and so there is this problem methodologically of picking winners and the best version uh that I could imagine somebody saying as an alternative to this as an alternative to creative testing is like okay Andrew that's fine but here's how I make the decision about which out is a winner and which one is a loser I use statistical significance okay and what they mean is they look at two cells in a test okay two different uh an A and A B just like every a test and they wait until they get statistical significance when they have stat Sig and I'm gonna say stat Sig because it's easier to say for the rest of this when they have stat Sig they're going to um use that and pick the winner now first of all I believe almost nobody actually does this uh for one at least very obvious problem which is that it would cost huge huge huge amounts of money in Creative testing to make every ad you run get to stat Sig before you make a decision about it okay that's first secondly the test design is nonsense and this is where we go to the idea of a randomized controll trial right if you go to any AB testing tool online what you are going to want to see is a true holdout test for it to work like a well-designed test right let's say you do a geolift study somewhere okay or let's say you run an AB test on meta ads for ASC versus baau okay something like that if you run a true AB test what meta is going to do is split the audience between people who um between two groups of of uh potential people potential users on the platforms okay it's going to split the audience and make sure that um cell a ads let's say let's say you're doing ASC versus Buu your ASC campaign only serves to group a okay and your Buu campaign only serves to Group B and they don't see the other one and that way you get a true hold out of each audience or a geolift study would do the same thing right you take uh Goa and go and you make sure that you don't run the ads to both and you you know whatever you see what I'm saying you split the audiences nobody is doing that with their creative testing campaigns okay but in a randomized controlled trial that's what you do you take two audiences you treat them differently and uniquely and then you analyze the data for St for statistical significance n [Music]