Author: David Edwards

Marketing measurement has gotten good at telling us what happened after we spent the money. We know where an ad ran, what it cost, who saw it, what they clicked, whether they bought something. We feed that information back into the plan and keep optimizing.

We are worse at answering a more basic question before the money goes out:

Was the creative worth putting money behind in the first place?

We spend weeks or months developing creative. It gets reviewed, revised, approved, tested, revised again, and handed to media. Then we put real money behind it and finally see how people react. At that point, the test has already gotten expensive.

Gary Vaynerchuk has been making this argument from the creative and media side for years. His position is direct: “I think it is insane to spend [even] $1 of working media on creative that has not been validated on social.” He has also argued that too much marketing measurement is “predicated on maybe or potentially vs. actual,” and that bringing creative and media back together pushes the work closer to real business results.

The measurement implication is what interests me. When creative and media split, creative lost one of the most useful things measurement can give it: a live feedback loop.

Put the test before the spend

Digital media gave us an incredible amount of information. We see delivery and response almost immediately. We run experiments. We change audiences, placements, frequency, spend, and dozens of other variables while a campaign is live. That is progress, but it also created an imbalance. We measure how we distribute an ad in incredible detail while doing much less to validate whether people want the ad in the first place.

You can build a sophisticated media plan around mediocre creative. You can optimize its delivery and build a dashboard explaining exactly what happened. The dashboard doesn’t fix the creative.

Gary’s model for organic social moves the learning earlier. Make a lot of creative, put it into the market, see what gets a reaction, then put working media behind what has already shown it can earn attention. Organic performance isn’t proof that something will drive incremental sales, but it gives you information before you’ve committed the bigger budget. You learn while you can still do something with the information.

That connects directly to Velocity. My argument there was that analytics should shorten the distance between useful information and the next decision. Skipping validation so you can get into paid media faster isn’t velocity. You’ve just moved the learning to a more expensive part of the process.

I’ve watched this happen more than once. Performance starts to slow, the team decides the creative is wearing out, and there isn’t a validated replacement ready. Tired creative stays in market while the next round gets developed and approved, and then we spend money finding out whether that round works too.

The problem isn’t that we won’t know everything ahead of time. We won’t. The problem is that we could have known something sooner.

The test doesn’t have to live on a screen

Gary has also been talking about a barbell forming in marketing, with extreme digital on one side and extreme analog on the other. He has talked about how those sides feed each other too: physical experiences create reactions and content that can travel digitally.

From a measurement perspective, there are two different things happening on the physical side. One of them is a fast feedback loop.

An event, pop-up, sampling activation, run club, collectible, or product drop puts an idea in front of real people. Did they stop? Did they participate? Did they bring someone else over? Did someone pull out a phone and post without being asked? Did creators pick it up? Did something that happened with 300 people become something hundreds of thousands wanted to watch?

Those are reactions to creative, and you can get them quickly. The event isn’t automatically the slow, immeasurable side of marketing. It can be the test. The reaction creates content, that content gets another test through organic distribution, and what keeps working earns more investment.

That is a functioning feedback loop. But it isn’t the only value the experience creates.

Some of the value stays in the room

If the only value of an event is the content it produces, we’ve reduced the analog side of Gary’s barbell to another media input. The event matters because it creates content. The collectible matters because people post it. The community matters because it generates reach.

That can’t be the whole story.

Sometimes the experience matters because of what happened there. Someone showed up and met people. They got something they cared about. They became part of a group. Maybe they came back. Maybe they kept what they bought for ten years. Maybe the brand became connected to a period of their life.

None of that requires an Instagram post to have value.

There are two pieces here. One is catalytic. The experience creates a reaction, that reaction creates content, and the content gives us another chance to test the idea. We can get feedback on that fast.

The other piece is harder. Sometimes being there mattered. Belonging mattered. Owning the thing mattered. That value is intrinsic to the experience, and it doesn’t necessarily show itself on the same timeline.

Treating those as the same measurement problem is where we get into trouble.

Some things don’t resolve into a clean number

We can look for signals. Do people come back? Does retention change? Do brand metrics move? Does sales behavior change? For a collectible, maybe secondary-market activity tells us something. For a community, maybe participation over time does.

All of that is useful, but it doesn’t necessarily prove the full value. Analytics has a tendency to keep looking until we find a metric and then feel like we’ve solved the problem. Sometimes we have. Sometimes we’ve just found the thing that was easiest to count.

Someone feeling like they belong to something is real. Turning that feeling into an incremental ROI number is a different problem.

There is some irony in this because even the parts of advertising we treat as highly measurable aren’t always as measurable as they look. Randall Lewis and Justin Rao studied 25 large advertising field experiments involving millions of customers. Even at that scale, the median confidence interval around advertising ROI ran more than 100 percentage points wide. Advertising effects are hard to estimate precisely, even with randomized experiments and enormous samples.

A dashboard can show a number to two decimal places. That doesn’t mean we know the true effect to two decimal places. We already accept a lot of uncertainty in marketing. We just feel better about it when the metric updates every morning.

Then we get to community, experience, brand meaning, or identity and suddenly want a perfect causal chain before we’re willing to defend the investment. That’s where measurement starts creating its own bias. The things that are easier to count become easier to fund, even when ease of measurement and value aren’t the same thing.

Not everything runs on the same clock

This isn’t an excuse for lazy measurement. If we can design a better experiment, we should. If we can tighten the feedback loop, we should. “We can’t measure it” can’t become a shield for work that simply hasn’t been measured well enough.

But the opposite is a problem too.

If you’re building a community, you don’t get to decide after two events whether it worked. If you’re creating a collectible people care about, a seven-day conversion window won’t give you the answer. If people start folding your brand into their identity, you’re dealing with something that can take years to build.

We should still look for evidence. Brand tracking can tell us something. Retention can tell us something. Long-term sales behavior can tell us something. A causal experiment can give us stronger evidence about a specific effect. Those are all useful inputs into the decision.

They still may not capture the whole thing.

That’s the part I don’t want analytics to paper over. If the argument is that some value develops differently and on a different clock, the answer can’t always be to invent another KPI and pretend we’ve made the uncertainty go away.

There may not be a KPI for belonging. Part of running a business is making decisions under uncertainty.

Measurement should help us make the decision

That’s the part I would add to Velocity now.

The goal is still to get useful information into the decision faster. If something can give us a useful answer tomorrow, waiting six months is bad analytics. If something takes years to build, demanding a seven-day answer is bad analytics too.

Those aren’t the same problem, and they shouldn’t be treated like they are.

I want creative tested before we pour working media behind it. I want measurement involved early enough to change the decision, not just document what happened afterward. Wherever we can learn faster, we should.

But I also want an organization capable of saying, “We don’t know exactly what this is worth yet, but we believe it’s worth building.”

That’s harder to put into a budget deck. It’s also sometimes the more honest answer.

Measurement should tell us what we know, what we don’t know, and where we can learn faster. It should make the next decision better. What it shouldn’t do is turn uncertainty into a reason not to act.

Because once the only things we’re willing to build are the things we can prove quickly, measurement isn’t informing the strategy anymore.

It’s writing it.

An analytics philosophy

I think about analytics as sitting in the middle of the feedback loop.

You make a plan, put it in market, see what happened, figure out what it means, decide what to do next, and then do it again.

We sit right in the middle of that. So yes, getting the answer right matters. But if it takes three months to get there and there’s nothing left to change, I’m not sure how useful that answer actually was.

I’ve spent a lot of years watching good analysis land too late to change anything. At some point that stopped feeling like bad luck and started feeling like a design problem.

That’s what I mean by velocity: how quickly a team can learn something, do something with it, and start the next loop.

Why speed matters

The obvious answer is that faster decisions are useful. I think the bigger thing is that the learning stacks.

Say one team gets through a meaningful learning cycle every six weeks and another one does it twice a year. The first team hasn’t just learned more things by the end of the year. The questions they’re asking in June are informed by things they learned in March.

The slower team is still waiting on March.

Then the same thing happens again. Every answer changes what becomes worth testing next, so the distance between the two teams keeps getting bigger.

We spend a lot of time arguing about whether one method is better than another in isolation. That matters. But I think we underweight the value of simply getting through more good learning cycles.

A slightly better answer three months from now isn’t necessarily worth more than a good-enough answer while you can still do something with it.

The two levers

Analytics has two broad ways to make the loop move faster: technology and measurement.

Technology is mostly about how long it takes to know what happened.

That’s pipelines, cleaning, tagging, automation, AI. Basically all the stuff between data existing somewhere and somebody being able to actually use it.

Measurement is about how confident we can be in what happened and why.

Stats, models, test design, causal inference. It’s the difference between seeing a number move and having enough evidence to believe something caused it.

Most of what we do comes back to one of those two things: make the information available faster, or make it trustworthy enough to use.

The problem is that sitting in the middle cuts both ways.

Analytics can speed that process up dramatically. We can also be the bottleneck.

We can always ask for another cut, another control, another two weeks of observation. Sometimes that’s absolutely the right call. Sometimes we’re just making the answer more defensible without making the decision any better.

There’s a point where more rigor changes what you should do, and there’s a point where it mostly changes how comfortable the analyst feels presenting the result.

We have to get better at knowing which one we’re doing.

Measurement can’t be the last slide

The failure mode I run into all the time is somebody asking near the end of a planning conversation, “Okay, so how are we measuring this?”

By then the budget’s allocated. Audiences are locked. Creative is built.

And most of the important measurement decisions have quietly already been made for you.

Maybe the markets weren’t set up in a way that lets you test anything. Maybe there’s no holdout. Maybe the creative wasn’t tagged properly. Maybe the campaign structure makes it impossible to separate the thing somebody suddenly wants an answer about.

Whatever you design at that point can probably grade the plan.

It’s much harder for it to inform the plan.

Take a pretty basic example.

Two teams run essentially the same campaign.

The first team builds beautiful reporting. Three weeks after the flight ends, the readout is clear: short-form creator content beat the polished product demos.

That’s genuinely useful to know.

The budget is also entirely spent.

The second team started with the question. They knew they wanted to understand whether creator content performed differently, so they built that into the campaign. Creative was tagged correctly. Spend was distributed across the hypotheses on purpose instead of by accident. The results were readable while the campaign was still running.

Halfway through, they had enough signal to move more of the remaining budget toward creator content.

The analysis doesn’t even have to be better.

One team still had money left to move.

That’s the difference I care about.

What are we actually going to do with this?

This is probably the question I come back to most when we’re designing measurement.

What are we actually going to do with the answer?

Not “what KPI are we measuring?” Not “what study can we run?”

What decision changes?

Say we run a study and the result is positive. Great. What happens next?

Do we move budget? Change the creative? Change the audience? Stop doing something? Scale something?

What happens if it’s negative?

What happens if it’s inconclusive?

If none of those answers would actually change the plan, I start questioning what we’re buying with the measurement in the first place.

That doesn’t mean the work is useless. Sometimes you really are trying to establish a baseline or understand brand health. That’s fine.

But that’s different from pretending the study is there to drive an optimization decision.

The same problem happens when the method doesn’t match the question.

If the actual business question is incremental sales and the only thing you’re running is a brand lift study, you can execute that study perfectly and still not answer what the team actually needed to know.

That’s why I want the decision first and the method second.

If there is no plausible result that changes what you do, you’re not measuring for a decision.

You’re documenting.

Bias toward action, not certainty

The purpose of measurement isn’t to eliminate uncertainty.

It’s to get uncertainty low enough to act.

That distinction matters because I think analytics teams naturally gravitate toward “do we know the true answer?”

Of course we want to know the answer.

But in practice I care more about: do we know enough to make a better decision than we would’ve made without this?

You can usually answer that question much earlier.

It also helps to remember you’re not getting one shot at this. We can take a read now and move, then keep learning. Maybe a platform read points one direction, an incrementality test gives us a better answer a month later, and MMM eventually gives us another view at the mix level. Sometimes they agree. Sometimes they don’t. That’s useful too.

Tests can help calibrate the models over time. If a geo test says one thing and the model says another, I want to understand why. Maybe the test changes what we believe. Maybe it exposes something the model is missing. The point is that each read gives the next one more information to work with.

So acting on an early read isn’t the same as betting everything on it. It’s the first pass. Later work can reinforce it or tell us we were wrong. Either way, we’ve learned something sooner than the team still waiting for the perfect answer.

That’s why I like setting decision rules up front.

If we see X, what are we going to do?

If we see Y?

What if the answer is basically noise?

Have that conversation while you’re designing the test, not after you’re staring at a number you have feelings about.

It forces everybody to admit what would actually change their mind before they know which answer they’re going to get.

And I’m not arguing that every decision should ride on a directional read after three days.

Moving $50K between two creative approaches and setting next year’s $50M channel mix are not the same decision.

The cost of being wrong is different, so the amount of certainty you should demand should be different too.

More rigor is valuable when more rigor can change the decision.

Past that point, sometimes we’re just slowing ourselves down.

This also isn’t an argument for running more tests. Testing everything fragments your budget and your signal, and then nothing reads cleanly. It’s part of why I want a learning agenda in the first place — so we’re covering the questions that actually matter instead of answering whatever came up last week.

The tell

I’ve started noticing this in planning meetings.

You can usually tell how serious a team is about learning by when measurement comes up.

If the buy is basically finished before somebody asks how we’re going to measure it, most of the damage is already done.

If the measurement questions are being asked while the plan is still being shaped, now you have a chance to build something that actually teaches you something — while there’s still time to use the answer.

That’s the philosophy.

The rest of this site is mostly about how to build it.