Related Posts
Where do analysts in SF / Bay Area live?
Best recruiters for moving in-house?
How are hours in CMAAS?
Rising Star
Pro
Anybody here tagged in Wipro yahoo project !?
Additional Posts in Media Wizards
What’s the inside scoop on GALE agency NYC?
So tell me, how did you land your dream job?
Media buying burn out. Is strategy any easier?
New to Fishbowl?
Download the Fishbowl app to
unlock all discussions on Fishbowl.
unlock all discussions on Fishbowl.
Was the campaign set up as an experiment or are they providing the analysis separately? What was the variable being tested?
I'm assuming this is a brand lift test.
And I'm assuming you're comparing something like Favorability (or something % based) between an exposed and unexposed group.
Where a lot of people get tripped up is that stat significance is more about the difference between "exposed and unexposed" than sample size.
If unexposed Favorability is 20% and exposed Favorability is 80%, you don't need very much sample (which is directly related to your spend) hit statistical significance. The analogy to think of here is that you don't need to see that many shots to be confident Michael Jordan is a better 3 pt shooter than a random copywriter.
If the difference is 50% versus 55%, you need more sample (and thus more cost) to be 95% confident that the +5 difference is a real and not just a random fluke (which is basically what you're saying when you say something is statistically significant).
If the difference is 51% versus 50% you need a ridiculously, ridiculously large sample to be statistically confident that this 1 pt difference is not a fluke.
A lot of people think significance is something you can solve with more sample when in reality the issue is that in most cases you didn't generate a meaningful amount of lift to be detectable within any size sample.
Therein lies the problem. With most BLS’, I’ve seen lifts of 1-2% and everyone calling it success.
Much worse is when you run an incrementality study, and then you find the cpa of your incremental sales is 20x, and they still call it a success when in reality, you don’t need to be advertising on social at all.
Yes you’re correct, spending more money against the same creatives that isn’t driving lift isn’t going to suddenly generate stat sig results.
Precisely
Your sample size could also be too small to gauge whether results you’re seeing is a trend or an anomaly. The smaller your sample size the more unstable your data can be.
CPMs on Meta are rarely about $20. 200k spend is more than enough impressions.
SS1 correct. It’s just Meta trying to cover their ass and our ppl not getting it 🙄
More money = higher frequency = poof, success
More $ = success for the publisher