How The Meta Ads Algorithm Really Works transcript

one of the reasons Brands lose so much money on Creative testing campaigns and if you've listened to me before you know that I believe creative testing campaigns are like maybe the biggest cost center in all of e-commerce I think that on Facebook ads running separate creative testing campaigns is a gigantic waste of money that is sucking dollars out from your eitaa that is creating all kinds of silly problems in your advertising account it's making it harder for you to scale your winners harder for you to suppress your losers like I just think there's cost all over it and it's one of the most absurd things that we do in all of e-commerce I think creative testing campaigns are a huge problem in terms of the value they suck out of e-commerce businesses you might have heard me say that before I got a couple episodes on Creative testing if you want to get more sort of familiarized with how I think about this and what I do instead the links for those are in the show notes by the way you can go check out uh both of my my full length episodes on Creative testing campaigns and why I think they're so problematic but the reason that a lot of people do them is because they don't understand something really fundamental and important about how the meta ads machine learning that drives the placement of your ads actually works okay and that really fundamental and important thing is that people think about creative testing wrong um at the Baseline and so what I want to talk about right now is how to think instead about what the machine learning for meta ads is actually doing to forecast the performance of your ads and to do that you need to understand the difference between probabilistic forecasting using Basi and logic and uh and a randomized controlled trial and so this is going to sound like I'm getting into the weeds for a minute but let's just start with a sort of scenario okay the scenario is like this you see a an ad let's say you launch an ad and this has happened to anybody who's ever run cost controls okay again if you're new to my podcast you need to go back and look I have a bunch of episodes about running bid caps and cost caps and TS targets and I've showed it in some opening the books episodes so go check those out there's a bunch of stuff that you can go go look at there uh but uh if you're running bids okay and you're running cost controls there's this experience that happens all the time right you see an ad let's say it spends $100 it gets six purchases and you're running and let's it's a $50 average order value you're running a three to one because you've got $300 let's say actually it creates eight purchases okay um and you're like this is awesome this ad is smashing um I'm running a four to one Ros on this new ad but then this thing happens that's really annoying which is that it doesn't keep delivering that ad instead meta will prefer and deliver other ads even if you've set a $50 cost cap okay you're getting a $50 CAC meta will prefer other ads that are running at like a one and a half to one um and you can't figure out why you can't figure out why it's the case that meta is preferring these other ads even though this ad that you're running that you're running a create you know you launch new is is generating this massive return for you and you're like I don't get it there's eight purchases there it's telling you something really significant okay on the other hand there are ads that run so just keep that as scenario a on the other hand and uh hand there are ads that run that are spending up to $500 and they're running at a 75 or $1,000 they're running at a075 and you can't figure out why meta keep spending on those ads when it's below roas Target or or above your CAC Target okay this and and there's a third scenario let's let's so let's do there's scenario a scenario B let's do scenario C here okay and there's a this is this is the one that's maybe the most frustrating for people is an ad that spends a few dollars a day right spends five and that's it total across the first week it spends $10 $20 Etc and that is maybe the most aggravating of all because you went to the pain of developing this creative and you can't figure out why it's not spending and what almost everybody will say in all three of those scenarios what all of us will do is we'll look at and this is definitely my instinct as well uh on my own what we'll do is we'll look at um the purchase Behavior and the click behavior on these ads and we will say I don't understand why meta isn't taking that as a stronger signal is it purchase Behavior the most important signal of value that you that you can get on an ad okay now um this is where the problem fundamentally lies and and with scenario C where it's not spending at all there's this thing where you go you know if if purchase behavior is the most important thing you go like I don't understand how can meta possibly decide not to spend more on my ad if it's only spent $5 or $10 like it can't possibly have learned anything significant about what's going on with my ad and here's my basic contention about that is going to drive everything I'm going to say here that meta knows way more way faster based on what appears to you and me to be way less data than uh than seems possible okay in including that scenario C $5 scenario $10 scenario okay what I believe is $5 or $10 tells meta a lot more than we think um okay now let's put aside one possible way of making this case one possible way of making this case is that meta in fact is reading the ad reading the content um you know with machine learning and and AI basically uh uh in advance and is predicting actually in advance whether your ad will work even before it delivers it okay I think it's possible if not actually even likely that meta is doing something like that but I'm just going to leave that aside okay so that I do not need that to make the case that I'm going to make right now uh I think instead it's possible or so yeah okay so I do think it's possible but we'll just leave that aside okay I want instead to think for a second about how we evaluate the performance of ads okay most people let's start with sort of the alternative to my approach to doing this most people approach creative testing campaigns and they have to make a decision about how to know when an ad is a winner or loser so the standard ad setup for for a lot of advertisers is you build creative testing campaigns where you force spend on an auto bid to a bunch of new ads that you make make okay and as you make a bunch of new ads you analyze the performance of those ads and then for the ones that are winners you move them into your scale campaign okay that's a separate campaign so you take the ad you kill it in your creative testing campaign you say this ad is performing well in Creative testing I'm G to move it into scale okay and now my scaling campaign is going to make it work okay so that is like that is like the classic way people run these ads my alternative to that is to eliminate your creative testing campaign instead launch everything with a manual Bid And if meta spends on it it it's a signal of success if meta doesn't spend on it it's a signal of failure okay that's it that's all you do if that is uh uh so okay so one of my critiques of creative testing campaigns is this how do you decide when an ad is a winner and ought to be moved to the scale campaign okay how do you do that and uh my experience is that most people have acts actually like no real rigorous methodology for doing this at all okay so they they they sort of go like oh had a few purchases uh you know CPC is good hook rate's good whatever something like that normally it's about purchases conversion rate and they say so that means it's a winner and So within a few purchases you go and scale it and put it in your winning campaign now there are manifold problems with this way of doing things like first of all if you're running scale campaigns as cbos or as asc's where there's already some way and it's a lot of people are for meta to prefer uh to to rank which ads it's going to deliver why not just launch it there in the first first place if you're going to use so few purchases to determine what a winner or not is okay but secondly it's that so few purchases tell you so little every one of you watching or listening to this at some point has seen an ad like a scenario a situation right where an ad gets a bunch of purchases really fast and you're like boom we solved it this ad's producing a four to one scale the budget it's working and then the moment you scale the budget it stops working right if you're doing it an auto bid setting I ran ads like this for so long I would just be so excited about an ad and you know I have all this bias I make the ad I think the head is awesome I think the visuals are awesome and so it starts to perform well at all you know I launch it and I'm checking it every two hours opening it up like everybody's who's been around media buying for a little while knows that feeling okay and so um and so we have all this bias going into it and you scale it and it doesn't work and it's like wait why didn't it work well that that error is a problem all the time and the reverse error is true too which is what about that ad that is spending but getting no purchases um you know how do you know you should shut it off right because um because if it's the case that sometimes you scale an ad you push spend to an ad that looks like it's going to perform scenario a that I mentioned earlier but it doesn't actually perform as the scale then it's probably also the case that some ads like scenario B that I mentioned earlier that some ads um look like they're not going to perform but actually could perform and so there is this problem methodologically of picking winners and the best version uh that I could imagine somebody saying as an alternative to this as an alternative to creative testing is like okay Andrew that's fine but here's how I make the decision about which out as a a winner and which one is a loser I use statistical significance okay and what they mean is they look at two cells in a test okay two different uh an A and A B just like every a test and they wait until they get statistical significance when they have stat Sig and I'm GNA say stat Sig because it's easier to say for the rest of this when they have stat Sig they're going to um use that and pick the winner now first of all I believe almost nobody actually does this uh for one at least very obvious problem which is that it would cost huge huge huge amounts of money in Creative testing to make every ad you run get to stat Sig before you make a decision about it okay that's first secondly the test design is nonsense and this is where we go to the idea of a randomized controlled trial right if you go to any AB testing tool online what you're are going to want to see is a true holdout test for it to work like a well-designed test right let's say you do a geolift study somewhere okay or let's say you run an AB test on meta ads for ASC versus baau okay something like that if you run a true AB test what meta is going to do is split the audience between people who um between two groups of of uh potential people potential users on the platforms okay it's going to split the audience and make sure that um cell a ads let's say let's say you're doing ASC versus Buu your ASC campaign only serves to group a okay and your Buu campaign only serves to Group B and they don't see the other one and that way you get a true hold out of each audience or a geolift study would do the same thing right you take uh Goa and go and you make sure that you don't run the ads to both and you you know whatever you see what I'm saying you split the audiences nobody is doing that with their creative testing campaigns okay but in a randomized controlled trial that's what you do you take two audiences you treat them differently and uniquely and then you analyze the data for for statistical significance if you're still tracking purchases on your Shopify store with the native Shopify Facebook ads integration for your pixel setup you need to move on it is time to get better Data Tracking with Billy go to b. b. to check it out Billy is what I use for every store that I work with it is uh third party pixel tracking with 100% server side tracking insanely fast loading website because you're not using uh in in browser tracking to do it and you're going to get way better event match quality by using Billy and and I've tested this a lot of different ways with my clients it's really simple um I've run side-by-side tests and seen Billy consistently outperform other third party tracking software as well as the native Shopify integration uh uh with the amount of data that it gets uh with purchase tracking on your store and so what I mean by that is like if you run the same ads side by side just set up two separate campaigns right campaign a and campaign B you just put a little bit of money towards each one each day and you run one of them with the same ads with Billy uh as your pixel setup and the other one as any other pixel setup you want to use what I am almost certain you're going to find is that Billy outperforms the other one and what I mean by outperform perform is that both your ads will perform better because uh meta has more and better information with which it can optimize as well as it will record and track the information that it is actually getting more accurately so it will record a larger volume of the purchases on your website for you I have tested this multiple times I've also seen it show up in the objective data of events manager where the event match quality score goes up using Billy versus other software it is Affordable it is a great piece of software it is incredibly easy to install and they're also adding all kinds of other really great stuff to it you can get started for $1 for the first month it's $1 for the first month by going to b. if you want to track your ads more accurately track your purchases more accurately get started with Billy today that's the way that you should set up a randomized control trial but there's a problem which is that that is not actually how people are building these tests in meta ads and on top of the fact that it would be cost prohibitive nobody's actually setting up a randomized controlled trial holding out you know putting up two ads putting one in gbet one in B Etc and they shouldn't I think it's great that they're not doing that because it would be an extremely expensive way and to test ads that would mean you'd have to take every ad that you produce okay and run them through that trial to get a really meaningful test design for this to work and so instead what people do is they say they're looking for stat Sig but they actually have an extremely absurdly ridiculously shoddy methodology for doing that and they're instead using their gut a lot they see a few purchases they scale it they see no purchases they don't scale it and they move on and that is a really big problem now the reason people do that is because they don't believe that meta can learn fast enough about how their ads are going to perform and so um they say okay well wait a minute uh meta can't actually learn enough about my ads uh and uh on on these low spends the scenario C looms large in their mind and they're frustrated because all these ads they make don't spend and look it is frustrating I totally get that it's really really frustrating when you when you put time and effort into create and it just doesn't work okay but let me make the case that it actually works better than you think to do this okay um and and that is by first introducing something and and you may know this concept right and and I'll just be honest with you like I'm not actually super deep stats I'm going to probably get some details slightly wrong here okay my statistical knowledge can be a little sophomoric at times so somebody should correct me if I get some of the explanation of this wrong all right um but there's a different approach here than using a randomized controlled trial because a randomized controlled trial by definition assumes that you know nothing about the potential outcome um that you are trying to drive right it basically says we're going to take these two groups assume we know nothing about them make them as similar to each other as possible and then see the results change one thing right you change one detail in this case it would be the ad you change they each get served ads but they change one detail you know the maybe the most like sort of tightest version of this and I'm certainly not saying you should do this would be they get the same ad but um group a gets headline a and group b gets headline B this is like the kind of ab test you run on your website all the time a lot you know for a lot of Brands test these individual little variants you change that one little detail and then you see how they perform okay that is one way of setting up a test but there is another way of doing this which is saying instead of doing that we are not going to assume we know nothing about either one of these groups and about either one how they're going to perform instead we are going to assume that some things are true in advance in both groups okay and because some things are true in advance we can apply that knowledge to a forecast of what we think will happen with both so we're not going to wait for a holdout test we're instead going to use the knowledge we have to forecast what's going to happen next this is probabilistic forecasting and it relates back the idea I mentioned the idea that it's basian earlier because it relates back to the idea of Bas theorem there's this idea of priors which is this idea that um some some uh in in your test and in your forecast there there are these prior these prior beliefs these prior ideas that probably hold true in the future as well and those priors get adjusted over time but that's not the case and so let me give you one such example right you might assume okay coming in to a uh to a a forecast that you are going to convert let's just say 4% of your clicks okay um and that might be a prior esentially essentially an expected conversion rate okay off of the clicks that you drive that is going to go in so as meta starts to scale your ads it is assuming and I've mentioned this concept before it is it starts with a prior that you're going to have a 4% conversion rate on Those ads now it's going to adjust that prior in real time as it learns okay but that that concept is that it's going to probabilistically forecast the performance of your ads based on uh based on its expected conversion rate which is the prior coming in in advance now why does it have that expected conversion rate okay because it has a whole bunch of clicks before and a whole bunch of data before and it knows that down probably to the level of the customer and to certain psychographic and demographic details and all these things and placement details you know how how much does a does a uh that 4% conversion rate probably here's what I'm saying that is probably actually made up of a lot of small conversion rates that roll up into it right so a Facebook desktop click meta would expect to perform really differently it would have a prior it would perform really uh and let's call it from a 35y old rich woman okay uh that is going to perform really differently than a let's say a Facebook Stories click so that's mobile now and let's call it from a 25 let's call it a 22-year-old male College student who doesn't have a job okay those two clicks meta is going to expect to perform really differently and it should it has a lot of information to assume that it's going to do that and it's meta delivers your ads it is bringing that information into the delivery of the ads scaling and and uh amplifying and suppressing based on the likely outcomes down to all of these different levels now meta doesn't tell us that exactly but this is really intuitive and really clear um from the data that happens this is why meta this is why it basically never works for you to in Advance go exclude placements and see if you can get better performance it's sometimes frustrating you'll see you know like oh my gosh are we're getting all this delivery on Instagram stories and it's not converting at all what if I just eliminate that right every time I've ever done that I haven't done that for a long time but a few years ago when I would have done that sort of thing right it actually makes the performance worse not better and it's because um meta actually has more information about the likely performance next on your ads and that idea the likely performance next is the notion of probabilistic forecasting now forecasting the future predicting the future is by definition impossible you will never get it exactly right but it but doing it probabilistically is what actually makes sense if you want to get your head around this idea um go back and read Nate Silver's book The Signal and the noise and it will help you think through the way that good forecasting uh works in the world and about how we've tried to apply that across a lot of different domains it's a really good book really helpful really engaging um you'll like it if you want to get your head more around what I'm talking about uh go go read that book okay so there is this notion of probabilistic forecasting and um and basically what is happening there is that meta is taking a bunch of information it has in advance and using that to forecast the future so that you um so that it can deliver your ads with the most likely outcomes relative to what you have told it so if you're running manual bids it is going to deliver as much spend as it can let's say you said a $100 cost cap right or $100 bid cap let's call it a cost cap a $100 cost cap so it's going to try and get you an average of $100 okay um uh and if in that case it's going to use a bunch of signals to say here's what is most likely to happen and meta tells you we're not guaranteeing you that you're going to get a $100 um CAC here we're going to instead just say um most likely on average we're GNA aim for this CAC okay uh and and so it's in and meta is telling you there this is probabilistic um that means it is a is a possible outcome uh but a probable outcome based on the information that it has available to it but of course again the future is unpredictable and sometimes the future is not like the past in a way that makes us difficult and and and and and Etc predicting the future is really hard right um and so um and so you build models and meta's buil has built a model clearly in their machine learning that does this for you okay now all of that said here's the thing that you probably need to understand more deeply as a media buyer if you want to think through this a little bit more and it's this I just said that the expected conversion rate on a 100 clicks let's say would be four purchases for a 4% conversion rate like that's the example I just gave okay but clicks are not the only information that meta has if meta had to get 100 clicks or 200 clicks or whatever to make a probabilistic forecast of the performance of an ad it would require spending a bunch of money on that ad let's say you get a $2 cost per click which wouldn't be crazy in today's world okay if that is the case then to get a hundred clicks right meta has to spend $200 on the ad and that would get it 100 clicks and let's say your actual conversion rate is 4% on that it's going to take a long time for meta to get enough clicks to validate that that conversion rate is quote unquote real there's actually a lot a lot of statistical noise in 100 clicks what happens if your true conversion rate is actually uh 5% but in those first 100 clicks you happen to only get four purchases not for any particular reason but just because that's just how it goes there's like random distribution of outcomes and smaller samples and that's how it goes okay so you get four purchases instead of five if you had held and you'd kept spending you'd actually get a 5% conversion rate at scale well for meta to validate that that conversion rate is at all real especially across all these different types of people all these placements Etc it would have to spend huge tremendous amounts of money to then build a for for your for your ad okay but what if it doesn't need um 100 clicks to make that decision because it has some other information about uh about how people are engaging with your ads and that information predicts the clicks which predicts the purchases okay so for example if 100 clicks could reasonably predict a 4% conversion rate what if um what if let's see what would get what would make that uh uh I think uh I'm trying to think through some some math in my head right now okay go like this if if 100 clicks reasonably present uh predicts a 4% conversion rate all right then what would happen if you had a 1% click-through rate um on your ads that would mean that for every 100 Impressions that you get okay that would get you 100 clicks and uh and so now meta could maybe GA gather its cck your click-through rate based off of let's say uh to to get to 100 um uh clicks okay then then it could actually do that on uh 10,000 Impressions but what if it doesn't need 10,000 Impressions to get an expected click-through rate what if actually on a thousand Impressions you got 10 clicks and that was a relatively strong signal and what if actually there was something before clicks that signaled something more significant and so now I want to talk about the idea of micro engagements if I lost you for a second there hang with me okay every marketer understands the idea of a micro conversion all right which is which is like the little step somebody takes before they make a purchase right the sort of obvious one here be like an add to cart we would consider an add to cart a micro conversion this is a a term I've heard thrown around for a long time micro conversion is some step that uh that a user takes some action they take that is on the way to the full conversion you care about okay so add to cart another classic one is like an email capture an email capture is a conversion a user took a real action at that point they actually gave you some information about themselves and so sometimes marketers have thought in terms of how do we generate micro conversions on the way to conversions as a way of sort of guiding somebody through the purchase process and um getting them to take one little step after another um this is like sort of the classic classic uh reason why sometimes a two-step popup Works where you where you um tell customer you you know instead of just uh putting up your popup on your website and it says you know submit your email address to get 10% off or whatever you say you start with the popup saying would you like 10% off and it's yes or no okay and then getting somebody to click yes is a micro conversion it's a step towards that and it helps them to take the next conversion to next step after that where they actually submit the email address go try that sometimes you've never done that it'll probably make your email capture rate go up just asking for that yes first right this is another thing that's classic sales tactic right get people saying yes like make get them to make small decisions for that okay so that micr conversion ideas is intuitive to a lot of us what if there's another um similar thing going on though at the level of engagement right if a click is an engagement with your ad are there micro engagements with your ad okay what happens before a click well I'll tell you what happens before a click somebody has to stop on your ad in the feed okay somebody has to pause your ad from automatically scrolling past in Instagram stories um maybe they watch your video for 3 seconds or 15 seconds and in fact you can go look right at we know for for sure that meta is recording all of these um micro engagements I'll call them and and part of the way we know that is because you can pull them up in your dashboard you can go look and see um 3se second video views on every ad and in fact we even call that many people call that hook rate the idea is that tells you something about the user's experience with your ad okay so a 3second video view very often that happens on 40% or you know something like that give or take 10% of the ads that you serve uh the video ads that you serve and so if a cut if 40% of the ad Impressions that you serve uh end up in that end up with a 3se second video view end up with a a micro engagement like that if 40% of them do that and you serve a thousand Impressions okay and those th000 Impressions come at a $10 CPM now you've paid $10 to get 400 3 second video views and that's actually starting to create a larger sample size okay now you you know even if those 10,000 impressions are only likely to result in 10 clicks and those 10 clicks are likely to result in no purchases 400 3 second video views matters and if we know for a fact that meta is recording uh 3sec video views what is to make you think they are not recording like stop rate on the ad right or like pause rate like who you know the user didn't scroll past it right those kinds of things like this is not uh this is not tinfoil hat kind of stuff we know that meta is recording all kinds of interactions that customers are having all kinds of engagements that customers are having with their ads and we probably and we can reasonably assume that they're using that data to power their um to power their uh algorithm and machine learning okay so I saw somebody joke about this the other day somebody said like um I stopped for half a second on uh you know on a photo that has a picture of somebody's butt on it or something like that and suddenly I just get Butts in my feed for forever right somebody tweeted something like that the other day like that's right right like everybody's experienced this that you do something like this and uh suddenly you get tons and tons of content like this meta clearly is seeing all these things and it's algorithmically um uh serving more of it along the way if that's the case then your ad maybe actually only needs $5 to get hundreds of 3second video views plus whatever other micro engagements are there which meta can then ladder based on your account's history based on all of the data that it has before that using probabilistic basian forecasting okay to what is the likely conversion rate on that ad and what is the likely aov on that ad and in fact it doesn't even need aov if you're running bid Caps or cost caps because aov is not a factor okay uh in fact it it's just it's just conversion rate and clicks it's because it's a CAC based um uh forecast okay so uh so it has all of this information and that I think is something people don't think through when they think about this that there's actually a lot more data accre to the level of every single ad in your ad account and this is why over and over and over and over again I am amazed when I force spend to ads like a client will be frustrated that ads are not new ads are not spending client will be frustrated about that and so I will force uh spend you know set up a creative testing campaign just sort of this is a way to just sort of uh reinvestigate my assumptions going in all right um and so we'll we'll Force spend we'll say like maybe we should do creative testing campaigns lots of smart people do them let's try it they seem to be having success uh and so we'll Force spend and over and over and over again The Meta will um suppress the same losers and amplify the same winners that it did when I launched them in manual bids I have yet to get to a point I'm sure there are false negatives out there okay uh future forecasting is probabilistic I'm yet to get to a point where I have seen a situation where forcing spend to an ad and I mean this I have no examples in my own ad accounts where forcing spend to an ad uh generated value in that ad that the manual bids did not pick up on their own I haven't seen it and I think it's because meta can learn a lot more a lot faster using things like micro engagements and lading that micro engagement performance to the to the broader chain of events that a customer is likely to take uh Than People let on and so what is the point of all this the point is if you think through the incredible data set that meta has this like think about all of the people constantly saying things like oh Facebook is definitely listening to everything we're saying and it's serving you know everybody's heard that right it's it's it's listening to your microphone and it's serving you content based on that right like the idea here is that like there's so much more information in these platforms than we let on meta can use that information to predict a lot more a lot faster than we think and therefore it is good news for you because now you do not need to waste all of this money on Creative testing campaigns you don't need to spend it um burning through money on ads that are going to underperform your targets and instead you can launch all of your manual all of your ads from the start with manual bids take spend as the ultimate indicator of performance and trust that at scale in the vast vast majority of cases if the ad is spending it is working if the ad is not spending it is not going to work and you save a lot of money along the way thanks so much for watching or listening to this episode if it was helpful to you you should subscribe I do uh Deep dive meta ads uh media buying cont content fairly often and I have a whole backlog of it you can go check that out as well there's a lot of content like that and if you want to reach out to me and you have questions more about something things I'm talking about please do that podcast a jf.com or reach out to me on Twitter at andrewj Ferris don't forget to visit uh my sponsor that I mentioned on this show that's Billy b i l y Billy and go to billy. a to start working with Billy today um it's the best conversion tracking software uh serers side tracking software that I am aware of uh for your pixel for your meta ads so go check that out to today thanks so much for watching listening I'll see you next time [Music]