What's The Best Way To Test New Creative On Meta? transcript

it's really really frustrating when you when you put time and effort into creative and it just doesn't work okay but let me make the case that it actually works better than you think to do this and that is by first introducing something there is a different approach here than using a randomized controlled [Music] trial because a randomized controled trial by definition assumes that you know nothing about the potential outcome um that you are trying to drive right it basically says we're going to take these two groups assume we know nothing about them make them as similar to each other as possible possible and then see the results change one thing right you change one detail in this case it would be the ad you change they each get served ads but they change one detail you know the maybe the most like sort of tightest version of this and I'm certainly not saying you should do this would be they get the same ad but um group a gets headline a and group b gets headline B this is like the kind of ab test you run on your website all the time a lot you know for a lot of Brands test these individual little variants and you change that one little detail and then you see how they perform okay that is one way of setting up a test but there is another way of doing this which just saying instead of doing that we are not going to assume we know nothing about either one of these groups and about either one how they're going to perform instead we are going to assume that some things are true in advance in both groups okay and because some things are true in advance we can apply that knowledge to a forecast of what we think will happen with both so we're not going to wait for a holdout test we're instead going to use the knowledge we have to forecast what's going to happen next this is probabilistic forecasting and it relates back the idea I mentioned the idea that it's basian earlier because it relates back to the idea of Base theorem there's this idea of Prior which is this idea that um some some uh in in your test and in your forecast there there are these priors these prior beliefs these prior ideas that probably hold true in the future as well and those priors get adjusted over time but that's not the case and so let me give you one such example right you might assume okay coming in to a uh to a a forecast that you are going to convert let's just say 4% of your clicks okay um and that might be a prior esentially essentially an expected conversion rate okay off of the clicks that you drive that is going to go in so as meta starts to scale your ads it is assuming and I've mentioned this concept before it is H it starts with a prior that you're going to have a 4% conversion rate on those ads now it's going to adjust that prior in real time as it learns okay but that that concept is that it's going to probabilistically forecast the performance of your ads based on uh based on its expected conversion rate which is the prior coming in in advance now why does it have that expected conversion rate I recently had a month where I made 3500 ads in one month for five clients and when I say I made them what I really mean is that my team made them and that would have been impossible without my friends at Behind the-scenes studio Behind the-scenes studio is a meta ads focused design and edit agency based in the Philippines so it's incredibly affordable people with deep talent and in fact in fact I've been trained in the way of thinking about meta ads that I talk about on this podcast all the time which is building lots of volume lots of variance for every ad and so they're well set up to execute a great meta ads design and edit strategy giving you beautiful ads that you or your clients will love at serus scale go to BTS studio.co.uk certain psychographic and demographic details and all these things and placement details you know how how much does a does a uh that 4% conversion rate probably here's what I'm saying that is probably actually made up of a lot of small conversion rates that roll up into it right so a Facebook desktop click meta would expect to perform really differently it would have a prior it would really uh and let's call it from a 35-year old rich woman okay uh that is going to perform really differently than a let's say a Facebook Stories click so that's mobile now and let's call it from a 25 let's call it a 22-year-old male college student who doesn't have a job okay those two clicks meta is going to expect to perform really differently and it should it has a lot of information to assume that it's going to do that as meta delivers your ads that is bring that information into the delivery of the ads scaling and and uh amplifying and suppressing based on the likely outcomes down to all of these different levels now meta doesn't tell us that exactly but this is really intuitive and really clear from the data that happens this is why meta this is why it basically never works for you to in advance go exclude placements and see if you can get better performance it's sometimes frustrating you'll see you know like oh my gosh our our we're getting all this delivery on Instagram stories and it's not converting at all what if I just eliminate that right every time I've ever done that I haven't done that for a long time but a few years ago when I would have done that sort of thing right it actually makes the performance worse not better and it's because um meta actually has more information about the likely performance next on your ads and that idea the likely performance next is the notion of probabilistic forecasting now forecasting the future predicting the future is by definition impossible you will never get it exactly right but it but doing it probabilistically is what actually makes sense if you want to get your head around this idea um go back and read Nate Silver's book the signal in the noise and it will help you think through the way that good forecasting uh works in the world and about how we've tried to apply that across a lot of different domains it's a really good book really helpful really engaging um you'll like it if you want to get your head more around what I'm talking about uh go go read that book okay so there is this notion of probabilistic forecasting and um and basically what is happening there is that meta is taking a bunch of information it has in advance and using that to forecast the future so that you um so that it can deliver your ads with the most likely outcomes relative to what you have told it so if you're running manual bids it is going to deliver as much spend as it can let's say you said a $100 cost cap right or $100 bid cap let's call it a cost cap a $100 cost cap so it's going to try and get you an average of $100 okay um uh and if in that case it's going to use a bunch of signals to say here's what is most likely to happen and meta tells you we're not guaranteeing you that you're G to get a $100 um CAC here we're going to instead just say um most likely on average we're gonna aim for this CAC okay uh and and so it's in and meta is telling you there this is probabilistic um that means it is a is a possible outcome uh but a probable outcome based on the information that it has available to it but of course again the future is unpredictable and sometimes the future is not like the past in a way that makes us difficult and and and and and Etc predicting the future is really hard right um and so and so you build models and meta has buil has built a model clearly in their machine learning that does this for you okay now all of that said here's the thing that you probably need to understand more deeply as a media buyer if you want to think through this a little bit more and it's this I just said that the expected conversion rate on 100 clicks let's say would be four purchases for a 4% conversion rate like that's the example I just gave okay but clicks are not the only only information that meta has if meta had to get 100 clicks or 200 clicks or whatever to make a probabilistic forecast with the performance of an ad it would require spending a bunch of money on that ad let's say you get a $2 cost per click which wouldn't be crazy in today's world okay if that is the case then to get a 100 clicks right meta has to spend $200 on the ad and that would get it 100 clicks and let's say your actual conversion rate is 4% on that it's going to take a long time for meta to get enough clicks to validate that that conversion rate is quote unquote real there's actually a lot a lot of statistical noise in 100 clicks what happens if your true conversion rate is actually 5% but in those first 100 clicks you happen to only get four purchases not for any particular reason but just because that's just how it goes there's like random distribution of outcomes in smaller samples and that's how it goes okay so you get four purchases instead of five if you had held and you'd kept spending you'd actually get a 5% conversion rate at scale well for meta to validate that that conversion rate is at all real especially across all these different types of people all these placements Etc it would have to spend huge tremendous amounts of money to then build a forecast for your for your ad okay but what if it doesn't need um 100 clicks to make that decision because it has some other information about uh about how people are engaging with your ads and that information predicts the clicks which predicts the purchases okay so for example if 100 clicks could reasonably predict a 4% conversion rate what if um what if let's see what would get what would make that uh I think uh I'm trying to think through some some math in my head right now okay go like this if if 100 clicks reasonably present uh predicts a 4% conversion rate all right then what would happen if you had a 1% click-through rate um on your ads that would mean that for every 100 Impressions that you get okay that would get you 100 clicks and uh and so now meta could maybe GA gather its cck your click-through rate based off of let's say uh to to get to a 100 um uh clicks okay then it could actually do that on uh 10,000 Impressions but what if it doesn't need 10,000 Impressions to get an expected click-through rate what if actually on a thousand Impressions you got 10 clicks and that was a relatively strong signal and what if actually there was something before clicks that signaled something more significant and so now I want to talk about the idea of micro engagements if I lost you for a second there hang with me okay every marketer understands the idea of a micro conversion all right which is which is like the little step somebody takes before they make a purchase right the sort of obvious one here be like an add to cart we would consider an add to cart a micro conversion this is a a term I've heard thrown around for a long time micro conversion is some step that uh that a user takes some action they take that is on the way to the full conversion you care about okay so add to cart another classic one is like an email capture an email capture is a conversion a user took a real action at that point they actually gave you some information about themselves and so some times marketers have thought in terms of how do we generate micro conversions on the way to conversions as a way of sort of guiding somebody through the purchase process and um getting them to take one little step after another [Music]