The Reason Cost Caps Don't Produce Consistent Performance transcript

all right I've got another video where I am going to diagnose for you a a set of broken cost caps in an ad account I've done a few of these if you you follow my content you know that I uh run almost all my ads on bidc caps and Target Rass and and some cost caps as well but that I'm really strongly Pro cost controls and so it comes up all the time that somebody says I want to run bid caps I want to run cost caps but it's just not working I'm just not getting the outcome that I want and specifically this chart shows you something Beyond sort of the cost caps not working and so this is the context that I had on this video and I want to show you um and on this ad account and so I want to show you what context I came in with okay so what you see here is uh day by day the blue line is CAC Blended across the entire site day by day the red line is a 7-Day Blended CAC so it kind of gives you the trends of everything and the pink lines are overlaid you can see things are going better and worse and and all those things over time so um so what you see is back in here where I'm looking right now this this peak um this was uh a really specific thing that's sort of irrelevant to this conversation some Advantage Plus campaigns with changes and how audiences were defined and ultimately the the key net effect here the only thing that's really going to matter here is that after this when the CAC went back down one of the things that happened over that time was that uh what all existing customers were taken out of Advantage Plus uh audiences so the is a 0% existing customer cap on all the Advantage Plus campaigns because that's the kind of the key thing that's the first point the second point is you can see the trend lines here the pink in particular where the CAC is just steadily going up from what is this is June basically right in here where my mouse is right now um uh so some somewhere along those lines between three and four there and then uh it goes up through number five into past July and you can see CAC is sort of going up up up just uh hitting over $200 after being quite a bit lower than that in May and June and so what is going on for this ad account now this ad account runs at a pretty high cap it's high aov the $500 range so why would this be the case the other thing The Advertiser pointed out to me that is really significant here is the frustration he was feeling about how low and high the differences are here put another way these cacs get so high and in his view those are throwing off the performance going on so these high CAC days are ruining him and so they're hurting the low they if he could just eliminate the high CAC days and have lowack be better that's particularly frustrating since aren't cost controls supposed to deliver for you consistent CAC so why do the cacs range so much why is this why is this at account like that now the first thing I will point out here is that um on the left side of the graph this was not all running on cost controls as best as I understand it and there is still huge amounts of variation in that time from dayto day so you see the CAC ranges day-to- day at every point in this graph there is no point at which it does not do that whether there's cost controls in the ad account or not the CAC is simply inconsistent on a day-to-day basis and that is uh a really important detail to think about when you think about what cost controls do because I think this makes a lot of people crazy what they what they what gets them so frustrated is the way that the CAC varies when isn't isn't it supposed to be the case that cost controls get me consistent CAC so what is going on so before we deal with the the UPS slope and CAC over this time period um I want to talk a little bit about that day-to-day variation that you see because it is the case this ad account is driven primarily by cost caps along the uh the sort of from about number three through number four and into number five on on this graph okay and the first the the big point that I want to make here is that it is not the case that every purchase is going to be at your CAC it is not the case that that's going to happen first of all at the Baseline level you're actually not optimizing in for most people for a single day CAC you're optimizing for a 7day CAC uh seven day click and if you have one day view in there that's going to do that as well okay so you're actually not optimizing for a single day of performance and so those single days are just part of the deal you're going to get higher days and lower days okay secondly the the fewer number of purchases that you are getting the more variation you're going to see right so if if this Advertiser was spending $100,000 a day I can almost guarantee you it would all look way smoother the CAC would not vary nearly so much but every single day is a very small sample size so this is one of the first things I looked at when I looked over at the ad account okay so we can come in here and I'm going to actually go to a different screen here you go what you'll see is that on a we this is the main cost cap campaign there's a top spending campaign in this ad account okay uh in fact what I what I did first of all by the way if you're wondering kind of how I diagnose this the first thing I did was I looked over this time period we'll just take May um through so I'm going outside of the window a little bit I know uh through August and just looked what's the top spending campaign okay um this is something I always do when I diagnose ad accounts the first thing I'm always looking at is where did the money go because that is going to tell you the vast majority of what happened so in this case $333,000 spent the top campaign is this $135,000 campaign uh and the number two is$ 49,000 now we've covered over half the spend now there's a bunch of other stuff in here that definitely does matter uh probably some time limited stuff this is a a shoes brand and so they have you know probably various SKS coming in and out Etc it's even more Stark if we go into May and June I believe let's see yeah so now you got 193,000 total in spend 120 of that on this Advantage Boomer campaign um and that's also a cost cap campaign that's driving the majority of the spend so I wanted to the first thing I wanted to do if I'm going to understand what's going on in this ad account is go follow the money okay so I go and look at this campaign specifically and then what I did was I I had it pre-loaded here just so you don't have to watch any load Time I grabbed this campaign and in this case I've got January through August okay so basically entirety of you know the first eight months of the year here uh and I've got this one campaign selected and we've got it week by week over that whole time period okay and if you look at this week by week over that time period what you'll see is that in any given week in the conversion window this is uh I I check this right away 7day click or one day view that's the attribution setting okay one thing I'll flag right away I personally never include view attribution in my um attribution settings for a number of reasons including that it really muddies the Waters of what is actually happening in an account okay um but at the spend level that we're getting here um uh I I don't mind running ASC in this case you're going to get all all of your top spending ads in one ad set hopefully uh get more consistency on the on the performance because you're getting all those in one place instead of having a million adsets each getting one or two purchases every couple days um because in this case if you see on this $500 aov at this level of spend what you've got is in a given week let's take uh let's just use the the the um uh attribution setting that is the standard attribution for this campaign which is 70 click one day view so if I just have purchased Rass over that time what you see is um I I have split out here these other attribution settings right 17 and 20 day click but this is the standard um setup in this attribution setting what you'll see um or excuse me in the purchases level same thing this is 7day click and one day view purchases what you'll see is uh 34 62 69 up to 100 at the peak you got 140 purchases but down here you've got 76 78 you know until eventually it gets turned off gets turned back on down here and that means that in any given day right you are reporting on on uh on on a 7-Day click basis you are reporting at a maximum at the very peak of this spend 20 purchases and that might sound like kind of a lot but in this campaign but actually 20 purchases is a pretty small sample size and this is one of the where one of the things that I would say U and and by the way a lot of these are much lower than that right you got 10 purchases per day for a lot of these even less than that okay on average on average and and you just need to understand that that sample sizes being what they are means that at a daily level you are going to have more variation you just are you're just going to have more variation that's because people's behavior is all over the place is actually not totally 100% predictable uh who is going to buy from you as good as met metas machine learning is and so uh there is going to be some daily variation in that spend and in in excuse me in that in those in those purchases uh over that time all right and so actually if we go and we look at this you'll see this really clearly the dayto day here shows this really wide range this is taking us all the way back to to February but it illustrates the point nicely look um 9 3 8 7 7 12 9 5 3 11 9 13 like the difference between 9 and 13 might not seem like that much but that's actually like 45% more purchases in one day right because those samples are so small so that's going to make the CAC vary uh maybe a meaningful amount day over day if you look right 53 246 927 112 64 I mean that's a huge huge amount of variation and that's not because anything is wrong with the bit bid that's the mistake to make here it doesn't mean something is wrong with the bidding what it means is people's behavior is erratic it's all over the place you can't predict it exactly and so what you need to worry about is the Blended average and when we go and look over that same time period okay um and this is the campaign where it seemed that the The Advertiser was saying there was also a lot of struggle where he had eventually turn it off if you actually look over that time period and you look at the the 7-Day click one day view attribution there is still a lot of weekly variation and again would actually say even sample sizes of these sizes are subject to lots of statistical Noise Okay um but uh but now it looks a little more consistent over that time period now this is uh from March as you can see in this window down through through the end of June and again it ranges a lot 125 172 185 250 105 this entire time the cost cap was set to 200 and you can see it doesn't really go beyond that now I'll tell you something really honest here I actually don't know if you set a cost cap to $200 on 7-Day click plus one day view I'm not really sure if that is going to give you a click only cost cap or if it's actually a cost per result um or if it's going to actually include view I have heard somewhere that um that view is actually not in that consideration and therefore um and and and meta own documentation on this is actually extremely confusing I it's not very clear about whether that's the case it actually um and the behavior here makes me think it's not so right because if you this is a $200 cost cap the whole time you can see it is like especially in here as there's um as it ranges is up you can it gets to a point where it really consistently hits right around $200 there's some range there some statistical range but um but it's pretty consistent over this time period it's on a seven on a pure 7-Day click basis it's there the reason I don't actually know the answer to this question is because I never bid with view attribution anyway so I just don't really care so there's that okay um then uh so it's running really consistently then the advertiser gets really frustrated because on $10,000 in spend it goes all the way up to $283 in in CAC on a 7-Day click BAS 223 um here on 70 click 1 day View and he shots it off uh because it's like way way above Target here now I at this point am a little confused now you can see by the way if you go back to you just think back to the um the initial chart I started this call with like when I'm looking at this one of the things I see is yeah definitely some increase in CAC happening over time here and we may be able to see some of this but with a custom conversion rate metric where you've got uh maybe no it's about the same over this time period um that said another thing noticed in this ad account is that um conversion rate really really uh tank tanked over a lot more time so so here's my initial read uh before I get to that my initial read on what's going on over this sample size um when I see this CAC get up to 283 for one week on 37 purchases my initial read is you should not have turned it off um that's just statistical noise it's actually nothing it's it's there's all it is is noise and you don't actually nothing actually went wrong here it's just that you have to get good at the idea that you're playing poker here okay and what I mean by that is when you play poker the way you win is not by hitting every single time you don't you play the right hand in poker over and over and over again you win over the aggregate you don't necessarily win every hand you're going to lose a lot of hands and that's the way and now look you do that enough times you make a ton of money and so that's the way this ends up working out and um and so okay so that's the way I think you need to think about cost caps bid caps trass is you're playing poker play the right hand over and over and over again don't worry about if the the other player gets gets the ace on the river don't worry about it um play the same hand the exact same way next time because they won't get the ace on the river every single time okay now that assumes there's people playing against you that's not really the what that's where the analogy breaks down but I hope you get what I mean by that that's noise that's statistical noise it doesn't mean that somebody should have bet when they when they got that okay Blackjack might be a better example but you know I'm not a blackjack guy so there you go all right so so that's the first thing I would say over here what I would have done at this point is kept the campaign on because I would have said something is going on here okay um now the problem is he turns into back on and it actually looks even worse at first it comes back right to the same spot here um and it looks even worse now one thing I'll flag here is that if you're running 7-Day click and you see your performance over the first two or three days of that process like something is going wrong uh nothing is necessarily going wrong you need to allow the full seven days to happen because you're you can't you can't expect that on one day the CAC that you are getting is the same as the s-day CAC that you bid for that's a really important distinction right if you bid for uh $200 CAC uh and you have a $250 CAC after one day you just got to wait for the next six days to pass and probably people will purchase over that time period who clicked today and that will end up uh lowering your CAC over that time period okay but that's the idea all right so um so assuming there statistical noise here now there's another Factor here that's really important too and I'm I'm GNA um I'm going to reload a different screen here and then show it to you all right I've loaded another screen here which is the entire ad account from January through August and uh and I have a conversion rate metric here okay um and and so what you'll see in these times when the performance looks a lot worse when the CAC looks a lot worse um what you'll see is that the conversion rate goes way up into may maybe a Mother's Day bump or something like that the only thing I think of like if this is mostly women's shoes it goes down here in June this is still pretty high and then hits the tank in July and August and this is something that I think is is especially confusing about this if we just go to that same campaign we were looking at earlier we see uh the exact same thing happening here uh where where let's see if we look uh might might help us a little bit more yes if we look at this one campaign specifically you see the same effect right uh goes way up here and then goes down over this time and so this is another thing that is very strange so I reached out to the advertiser here to ask what was going on um over that time because this is where this gets really really tricky now I want you to think for a second about how cost caps and bid caps work okay the way they work is um is that at the uh uh that they um they have two prediction mechanisms work built into them and that is clickthrough rate and conversion rate okay and so uh in the calculation uh that the machine learning is running and this is actually documented right this is not me reading T leaves um meta has an in foure bids in the auction meta has an expected click-through rate and expected conversion rate well you get a lot of clicks on every ad right you're you're you're constantly um not dealing with small samples when you've got clicks right so if you look here you'll see 18,000 17,000 clicks you know over a month 10,000 clicks a month so the expected click-through rate is not really a big deal for meta to keep updated all the time that's what factors into your cost per click but for CAC to be calculated you don't just need cost per click you need cost per purchase right and that's going to be a matter of convert of how many of those clicks actually convert Now by definition purchases are the are the thing that accumulate the slowest of all conversion metrics right why because it's the last step of the journey okay so it's the very bottom of the funnel right the funnel goes from Big to small and and it's the last thing that happens so um so the expected conversion rate and and there's a couple of other reasons for this um updates relatively slowly and when I see a drop off um conver in conversion rate like this this looks to me like the kind of things that precisely is a scenario that cost caps and bidc caps have a pretty hard time with um where the conversion rate drops off particularly if there's some reason for it that is outside of the ad account that there's no way the ad account can read realize like a classic example of this is if you run a sale and then your sale ends you know let's say you run a fay sale and you get you know a thousand purchases over that time and let's say you put them on one campaign okay uh well meta is going to realize that you're it's you're converting let's say at 4% during your sale or 5% let's call 5% okay this is a Rand I'm just making this all up okay at 5% but the moment your sale ends those same exact clicks on the same exact ads might convert at 2 and a half% but meta doesn't know that meta doesn't know that you ran a sale in quite the same way it has to relearn that as it accumulates more purchases at a lower conversion rate and so the expected conversion rate is going to lag the actual conversion rate this is again in a lot of ways mostly a good thing it means that it doesn't over respond to exactly the kind of stati statistical noise that I referenced earlier on this video it doesn't it doesn't get four purchases in one adset and go woohoo scale it right that's exactly what's great about it that's exactly what I do with an adset when I see four purchases um in this case though it doesn't do that and so that's really good the problem is and if if this conversion rate is happening for some reason and I talked to the advertiser he didn't really know why it was happening was confusing to him as well like let's just say it's straight up macroeconomic Suddenly the same customer converts less for whatever reason okay if that's the reason why this conversion rate is dropping off Med is going to have a hard time reading that and for a little while it's going to overspend uh on on relative to the cost cap so if you have a $200 cost cap eventually it will catch up but for a little while you might get if the conversion rate gets cut in half a $300 um CAC or something like that there is a solution to this problem which is to um proactively adjust the C the cost caps and in general if you it's if it's more important to you to keep efficiency than it is volume you can do this in general when you're running cost caps and bid caps right if you're like if you have some reason this is actually what you should do after a sale for example to use the example I mentioned earlier you should proactively adjust the cost caps down even if you need a different number than it's actually going to do right so let's say you're getting a $100 CA during the sale the sale ends and you let's say you still you need a $120 CAC going forward okay then you should maybe set the ca the cost cap after the sale at 60 or something like that just to restrain uh meta from spending too much when it has not yet updated the expected conversion rate okay so in this case that could be part of what's going on the expected conversion rate is lagging the actual conversion rate and therefore it's delivering it to have a CAC so what I would do managing this account is simply lower the CAC on the I would keep running this campaign for sure and then I would just lower the cost cap proactively and then watch the actual CA that I get relative to the um to the cost per result goal that I'm putting in there the cost cap and and then keep managing it until they line up closer okay once they line up closer then you can just let it rip uh classically okay but until then that's what I would do so I hope that's some help to you as I think through this kind of problem you know there's all kinds of other things that could be happening here this is not not the only campaign in the account but hopefully it's some helped you thinking through how to solve a problem like this