Ahmed Ben El Cadi is a marketing analytics manager at GetYourGuide. He discusses how marketing analytics has evolved over the past decade and dives into the key ways companies can measure the impact of their marketing efforts.
Over the last ten years, the field of analytics has seen a huge transformation, especially for tech B2C businesses such as GetYourGuide. The ability to track and process gigantic amounts of data using distributed data storage and processing opened up new roles such as Data Analyst and Data scientist.
*I’m not going to enter the debate on what the difference between these two roles is, but simply let’s consider analytics as to the part of the business that helps make the right business making decisions in a lean and agile fashion.*
As a result, analytics is now a well-defined function and can’t be overlooked anymore by businesses wanting to grow fast. I remember when one of my previous managers used to say our department (analytics) was a luxury. That was 12 years ago. Since then, not only the function has become a key driver of growth but now we also expect everyone in the company to be data-driven and understand analytics concepts. As a result, we see this field to be quite structured and standardized.
When I look at the many job descriptions in Berlin tech B2C companies, I can see that they tend to separate analysts into three separate teams:
Indeed, I’ve spent a lot of time working on Marketing analytics (both on the provider and client sides), and am currently leading the team at GetYourGuide. So I want to help define it better and explain how it also offers standard best practices.
To start, marketing analytics is about:
1. Guiding our marketing investments at macro and micro levels, and
2. Optimizing their execution
The right focus on marketing is essential to business growth. As startups have limited brand awareness to begin with, they need to find ways to acquire customers while still being able to generate a positive return on investments. A significant budget amount is put into marketing investments, whether classic Search engine marketing, affiliate, or online/offline brand marketing.
The key to finding a sustainable return on investment comes from the ability to assess how much return one euro of a given marketing investment generates. There are classic methodologies that can be used to calculate this. You may have heard of attribution, incrementality, or media mix modeling (MMM) to name a few.
These methodologies have their advantages and disadvantages but in the end they all aim at giving a measure of what that return may be. Let’s take a look at them:
This is the most straightforward approach to understanding the value of marketing investment. The idea is to measure what would have happened if that investment didn’t exist in the first place. Indeed incrementality or lift studies try to test the absence of a marketing action on a given audience and measure the difference in revenues between the people exposed and not exposed.
This is usually easy for digital marketing providers as they control when to display an ad (hence they can easily decide not to show a certain creative half the time). From this, you can clearly measure the total revenue from the population exposed to the marketing actions (ie. certain ads, or emails) in comparison to the one that wasn’t, using classic hypothesis testing.
And as long as the two populations don’t interfere (which could happen in adtech as sometimes you can’t totally isolate two environments) you could get the exact incremental value your action generated, and understand its true ROI. (This can sometimes deliver strong surprises, like affiliates not generating any additional revenue).
Incrementality studies are powerful, but the main caveat here is that this whole calculation isn’t easily available to businesses as they need to get resources from their marketing provider (Google, and Facebook provide their own lift studies as a service, for example). In other words, one can’t run incrementality tests by themselves.
This is an issue because most tech companies are working on increasing their experimentation velocity and this dependency becomes an obstacle. Also, there’s no access to the underlying data, so one wouldn’t be able to check for factors that influence the results or understand them, which is sometimes as important as the result itself.
Fortunately, there are still ways to come up with an agile testing process. Although not as precise as a traditional A/B test, you could apply a marketing action in an area and find a “similar” area to control against (the area is a country or a vertical, you can learn more about this approach in this great post from our Senior Analyst Valentin Mucke)
Summary: Generally, incremental studies are the most accurate but are hard to run regularly.
Attribution is still a methodology to capture incrementality, even though it shouldn’t be used to quantify it. The idea here is to allocate a conversion to all (usually in a specific time window) the known marketing interactions that happened before the conversion.
A good attribution model is a key to guiding operations and making ROI measurement available to marketers for all their investments. It’s lean, as any marketing manager can look at how much revenue has been attributed to their campaigns, on a day-by-day basis and at low levels of granularity.
This however comes at the cost of accuracy.
Attribution comes with many limitations. The first limitation is the “selection bias”. Most data-driven attribution models (let’s leave aside last click or U-shape which are heuristics, and thus are by definition not accurate) are trained on a company’s own dataset, so they will be biased towards the most common marketing interactions leaving all-new interaction paths less covered and potentially less accurate.
The second limitation comes from “averaging”. To make data-driven attribution work you’ll have to average different marketing actions together, otherwise, you’ll have too many types of touchpoints to model, and fewer data to predict a customer’s probability to convert. This means that you will have to treat CRM or paid search as one bucket, while not all CRM or paid search ads are the same, and thus don’t generate the same incremental revenue.
The last and biggest limitation is that you can also only use attribution for the data you track which is mainly direct clicks on ads (and possibly email opens), however not all marketing assets need a click to influence the prospect. Display, and more generally Brand ads don’t expect you to click on them. For those, being exposed and increasing brand awareness is already a success.
And what about out-of-home (OOH) campaigns that you can’t track at all by definition? Missing those touchpoints is an issue as you can’t attribute revenue to these assets although they probably trigger future consideration for the product.
Summary: Attribution is the best way to report generated value on a regular basis, but you can’t take the values as fully incremental (even though it’s the objective).
Remember how I said that attribution will, by design, discriminate against ads and marketing that don’t generate trackable clicks? Well, media mix modeling (MMM) is the solution to this.
MMM’s goal is to find the relationship between high-level marketing investments by channel (media) and revenue. It does this by recommending a closer to optimal media mix that would lead to optimal revenues. (See Robyn the solution proposed by Meta)
You can do it through a regression model, where you take the different media investments over time (and other explaining/intermediary variables) as inputs, and the topline revenue generated as outputs. You need a lot of data to make this regression accurate, but you also need to work around the strong collinearity in your investments.
Investments that are collinear are investments that are always run at the same time. For example, if you spend on TV every time you are buying space on out-of-home billboards, then how could we know which media is generating revenues?
All this makes MMM something you can only run every one or two years depending on your marketing investments.
Summary: MMM is key to understanding the impact of all marketing investments, including the ones that aren’t trackable or are offline. However, it says very little about the total incrementality of the marketing spend.
All the models above help understand how effective different marketing actions are, yet they’re all evaluated on the same output metric: topline revenue. In general, for B2C products, you rarely acquire customers just for one conversion. You believe that once acquired, some customers would want to reuse the product, perhaps activated by another marketing campaign. As a business, you’d want to focus on these customers first.
The way we generally consider the extra value of customers is by calculating the customer lifetime value. In fact, this is the signal that we’d ideally like to map against our marketing investments. Yet to be able to do this, we would need to wait for many months to measure it (which would just be a proxy of the “lifetime value”). This is a problem, as you’d remove the ability for your marketing managers to adjust quickly.
The solution to this would be to predict the customer lifetime value, which is a tremendous challenge. First of all, there is less literature on this problem. Second, it can’t be generalized, as customer cycles vary a lot between businesses. To succeed, you have to fit a model that’s very specific to the company's needs. For example, GYG’s industry (Travel tours and activities) has structurally lower repeat rates than retail, which makes less training data available.
Summary: To understand marketing efficiency, you also need to account for the future value of your acquired customers. But this part is the one that’s less structured in the industry.
Measuring marketing impact is becoming increasingly structured, which is great for the analytics field and the ad tech industry. This provides more opportunities to learn from the community but also inspires new business services that replicate or automate the most accurate models for the benefit of their clients.
This is very interesting for the industry, as it will eventually leave more room for people to focus on strategic and operational insights that are what make the difference. Stay tuned as I’ll focus on this category in part II of this article.
If you’re interested in joining our Engineering team, check out the open roles.
Make Memories: Behind-the-Scenes of Creating a Brand Campaign
CRM Checklist: Do These 6 Steps Before Preparing Your Strategy