Whitepaper: The Evolution Of Measurement

Embracing Change

We often get questions about how we built our unified measurement platform, Polaris. We thought that it would be helpful to share our thinking in this blog post.

The mobile advertising industry has always been a dynamic one that has attracted brilliant minds to embrace its constant change and bring to life businesses that wrestled with the messiness of advertising data to deliver increasing value as they evolved.  Now is no different.  So we think it is important that as we approach the privacy apocalypse and its measurement challenges, that we commit once again to center ourselves, continue to embrace change and innovate.

Let’s begin by taking stock of the positives.  Firstly, this major industry shift offered increased user privacy.  This is in alignment with our core values as end users ourselves and as a mobile ad tech company.  Secondly, it allows the industry to ditch the issues with last click attribution and forge a new chapter of measurement innovation.

Thirdly, our unified measurement platform will ensure that you have granular visibility in order to continue to successfully optimize and execute. You will see more detail about this lower down in this post.

Our Quest For The Future Of Measurement

The IDFA restrictions, that rolled out in 2021, introduced a huge but welcome change to the mobile advertising ecosystem. After an Apple user upgraded to iOS 14, any iOS 14-compatible app that wants to access their device’s persistent identifier (IDFA), most often for the purposes of retargeting, user profile linking, or measurement, needs to ask for their permission (as shown in the example below).

IDFA User Consent Messages

This presented an interesting challenge for the industry since so many players in the ecosystem rely on the IDFA to varying degrees. At MetricWorks, we’ve focused on how measurement data will flow in the post-IDFA world. Back in 2020, based on the spirit of Apple’s new terms, we decided to evaluate this challenge with the assumption that MMPs will no longer be able to send events with attributed channel, campaign, country, and publisher app information. In its current state, SKAdNetwork, Apple’s on-device measurement offering, does not seem to provide enough granularity, nor does it handle post-install events well, making retention and LTV prediction impossible.

Early in our search for a suitable measurement solution, we identified incrementality as an important piece of the puzzle.  For those not as familiar with the concept – incrementality refers to the incremental value directly caused by each advertising touchpoint in the user journey. This is impossible to measure in the last touch attribution model (Fig.1 below), which credits the final touchpoint with 100% of the value for the user without consideration to the possibility that the user could have been acquired with fewer, more valuable touchpoints or even zero advertising as an organic install.

Last Touch Attribution Model. MetricWorks

A methodology known as incrementality testing is an ideal solution to prove the causal relationship between advertising touchpoints (ad buys) and uplift.  It prescribes a rigorous scientific process similar to randomized clinical trials used by pharmaceutical companies where a population of users is randomly split into a test group that is delivered an ad and a control group that receives a “placebo” (often an unrelated public service announcement or PSA). However, it can be costly since, if you’re displaying PSAs, you still have to pay for those impressions.  There is an even bigger problem with the advent of iOS 14. Most forms of incrementality testing require a large list of device IDs so that the audience can be split. iOS 14 will make this difficult to accomplish.

Another powerful tool in the measurement toolkit that also considers incrementality is media mix modeling (MMM). This technique uses regression models to find correlations between ad spend and business value. As a “top-down” technique (Fig. 2), versus a “bottom-up”(Fig. 3) approach like last touch attribution that works at the device level, it eschews device IDs in favor of aggregated data, and is therefore naturally aligned with user privacy. As you can see in Fig. 2 and Fig. 3 below, both top-down and bottom-up measurement techniques attempt to allocate the same users and app activity data (installs, opens, revenue) to the same four campaign/publisher app combinations in the proper proportions, but come up with different answers.

Top-Down Measurement Process By MetricWorks
Bottom-Up Measurement Process By MetricWorks

Key Idea: Based on the measurement output of the two different methodologies, we can see that they somewhat agree on the value of ironSource campaign B, publisher C (red). However, ironSource campaign A, publisher A (purple) didn’t get credited much in the bottom-up last touch methodology (just 1 install with little revenue) while top-down shows that, even though it might not be getting the last touch, it is providing significant incremental value. On the flip side, bottom-up attribution gives Vungle campaign A, publisher A (orange) a solid amount of credit for last touches, but statistically, it is providing almost no incremental value. We would have acquired those users either through other campaigns or organically anyway.

Coming back to the top-down media mix modeling (MMM) technique (Fig.2), let’s look at its advantages:

  1. Requires few inputs – mostly just a time series of spend and target outcome measurements.
  2. Robust to incongruities among ad channels, both online and offline, in terms of functionality and data availability.
  3. Can be used to predict the change in outcomes as a function of different spend inputs which is quite handy for planning purposes including what-if analysis.
  4. Can also be used in conjunction with other algorithms to optimize towards a given goal, which can be used for budget allocation optimization.

MMM has its own problems though, which also eliminated it from contention in our quest for the future of measurement. When we attempted to apply it to mobile app advertising, some major issues became apparent including:

  1. Requires several years of data at a minimum due to aggregation at the week or month granularity.
  2. Not used for quick decision making since it takes weeks or months to update a model with new data.
  3. Usually custom built for an individual advertiser by very expensive specialist consultants.
  4. Can only prove correlation, not causation (eating seafood may be highly correlated with personal wealth, but that doesn’t mean eating a lot of seafood will make someone more wealthy).

While MMM wasn’t a perfect fit, we recognized early on that the MMM concepts held a lot of potential. The key is that we found that most of the downsides could be mitigated through a combination of creative feature engineering and automated model validation through constant backtesting and live experimentation.

Keep in mind that the entire measurement process including modeling and validation must be completely automated in order to be scalable. No need to worry though because our existing automation technology in UACC is being enhanced to make this a powerful reality for you.

Our Solution For Mobile Measurement

Thank you for bearing with us and making it this far.  Let us now look at the high level overview of our solution:

  1. We believe that measurement of campaigns at the country and publisher app level (when applicable) remains critical for UA decision-making.
  2. Compatibility with the current iOS14 technical specifications and alignment with the spirit of the new rules are equally crucial for any future measurement solution  including ours.
  3. Regression models similar to those called for by media mix modeling can be augmented with techniques that address problems unique to mobile app advertising in order to finally deliver a measurement solution that considers incrementality.
  4. Using daily data helps solve the data volume issue inherent to MMM and allows the model to be updated more quickly so it can inform the types of fast decision-making required by modern UA teams.
  5. Backtesting allows us to automatically evaluate a wide array of approaches and select only the most accurate models based on their ability to predict known historical outcomes.
  6. Automated controlled experimentation enables live testing of incremental value predictions in the real world so that models can be validated and improved by rejecting poor models and feeding results back into the next model.
  7. Our unified measurement platform, Polaris, (based on our analysis outlined above) ties aggregated cohort activity with the channel, campaign, country, and publisher app dimensions, instead of the MMP.
  8. Our measurement module will be provided as an option at the app level so that advertisers can rely solely on MMP measurement for their Android apps, should they wish.

We hope that the above was helpful and invite you to connect with us about any questions so that you have a chance to shape our solution.

Best,

The MetricWorks Team

 

 

 

 

 

MMP 2.0 Launch At MAU

Following our VentureBeat launch by Dean Takahashi of MMP 2.0 (Polaris), we got to launch Polaris at one of the best mobile industry shows, MAU Vegas on May 23rd!  The reception was fantastic as mobile marketers are desperate for a measurement solution that closes the three value gaps (poor accuracy, lack of privacy-safety, and limited measurement scope) in last touch MMPs (MMP 1.0) to deliver superior performance.

In summary, we had a fantastic MAU! Thank you to the MAU Vegas team, our partners, clients, attendees and everyone who took the time to learn more about MetricWorks‘ MMP 2.0. If you missed our MAU presentation by our CEO, Brian Krebs, then do sign up for our FREE TIER to start enjoying the benefits of MMP 2.0. We’ve had a ton of signups. Make sure you don’t miss out on the solution that will help you demand more from your data.

If you would like to learn more, submit a demo request or visit our Polaris product page.

How MMP 2.0 Drives The Mobile Business

 

Reliable measurement is at the heart of decision making for the mobile business. As you can see from the diagram above, MMP 2.0 leverages multiple measurement methods (including geo lift testing and MMM), blending them to produce an output (single source of truth) that drives the business processes and delivers superior performance.  Our team has worked tirelessly to continue innovating so that we can deliver a solution that:

1. Looks like a MMP so that it is easy to use

2. Fits into your existing UA process to avoid any disruptions or productivity problems

3. Merges the strengths of multiple measurement methods to deliver far superior accuracy than individual methods and MMP 1.0

4. Offers sophisticated algorithms that removes the guesswork and signal conflict by outputting a single source of truth the plugs into your existing business processes

We hope you like Polaris and find it valuable in your quest to achieve higher performance while preserving your budget. 

Best,

The MetricWorks Team 

MMP 2.0 Is Here And You Can Start For Free

Thank you VentureBeat and Dean Takahashi for covering our launch of MMP 2.0 (Polaris) and how it closes the three value gaps (poor accuracy, lack of privacy-safety, and limited measurement scope) in last touch MMPs (MMP 1.0) to deliver superior performance.

We are excited to launch this latest innovation in measurement which our CEO, Brian Krebs describes as “fulfilling our mission to help marketers demand more from their data”.

Additionally, in order to make Polaris more accessible to marketers, we are now offering the Polaris Free Tier so that you can start using MMP 2.0 for free! Sign up today. To learn more, submit a demo request or visit our Polaris product page.

How MMP 2.0 Drives The Mobile Business

 

Reliable measurement is at the heart of decision making for the mobile business. As you can see from the diagram above, MMP 2.0 leverages multiple measurement methods (including geo lift testing and MMM), blending them to produce an output (single source of truth) that drives the business processes and delivers superior performance.  Our team has worked tirelessly to continue innovating so that we can deliver a solution that:

1. Looks like a MMP so that it is easy to use

2. Fits into your existing UA process to avoid any disruptions or productivity problems

3. Merges the strengths of multiple measurement methods to deliver far superior accuracy than individual methods and MMP 1.0

4. Offers sophisticated algorithms that removes the guesswork and signal conflict by outputting a single source of truth the plugs into your existing business processes

We hope you like Polaris and find it valuable in your quest to achieve higher performance while preserving your budget. 

Best,

The MetricWorks Team 

Europe’s Impending Crackdown On Measurement

Since our Jan 24th blog post, we continue to receive a number of questions about privacy violations in Europe and the impact on measurement. So here’s a short video by our CEO, Brian Krebs, that clearly describes Europe’s impending crackdown on measurement and what you can do. In particular, shifting to privacy preserving measurement is going to be critical. MetricWorks Polaris was designed to preserve privacy and deliver superior marketing measurement. Book a meeting to find out why.

In summary, for most app companies, the only real options to avoid massive fines similar to those we have seen so far are:

  1. Block access to European users completely (avoid jurisdiction of European regulators).
  2. Remove MMP SDKs from all apps and completely cease measurement activities.
  3. Continue using MMP SDKs, but ensure no device data is collected unless consent is granted (e.g., disable fingerprinting), meaning only deterministic last touch would be available and only for the few users that the MMP has double opt-in for (this may not even be possible for many MMPs at the moment and you’d still need a custom consent dialog for Android since there’s no ATT equivalent).
  4. Migrate completely to measurement methods that don’t require the collection of device data such as SKAN (iOS only), MMM, and geo lift testing (avoid collecting device data for the purpose of measurement altogether).


    If you’d like to discuss this topic further, feel free to book a time or contact us.

Complying With ATT Is Not Enough

 

All apps are in violation of the GDPR/French DPA if they meet the following criteria:
a. Have European users
b. Use an MMP SDK
c. Either:
    i. Are fingerprinting on iOS for users who do not consent to ATT (collecting data after an opt-out)
    ii. Have any Android users (there is no ATT-equivalent on Android, so no consent framework exists at all)

Continue reading for the full explanation of why the above is true. Many of you may have already seen Eric Seufert’s latest and greatest Mobile Dev Memo post about Voodoo being fined by the French privacy watchdog for using the IDFV for advertising purposes without user consent.  Up till now, most of the mobile industry has focused on complying with ATT. As made clear by France’s privacy regulator, CNIL, this is NOT enough, especially because complying with ATT does NOT equate to complying with any privacy law.  

 

But, why is that?

Well, ATT says that you can access and do what you wish with the IDFV (assuming it’s not breaking some other Apple policy) even if the user opts out of tracking. The ATT opt out only protects the user’s IDFA. GDPR and the French DPA, on the other hand, make it clear that you cannot do anything with that IDFV without opt-in unless it is:
1. clearly contractual (e.g., the user has already contractually agreed to be tracked) or
2. it’s in the legitimate interest of the advertiser (e.g., the advertiser must use your IDFV for tracking in order to provide the basic functionality that the user expects from the product).
Refer to Eric’s post for more details that support the above assertions.

    Now, in terms of measurement, what does this mean for the mobile industry?

    1. Most apps use an MMP SDK for measurement.
    2. MMP SDKs must collect device data in order to measure (whether that data is IP address, etc. for fingerprinting, which is against Apple’s policies, but has remained generally unpoliced on both iOS and Android, or a cross-publisher device ID like IDFA on iOS or GAID on Android).
    3. GDPR and the French DPA state that device data can only be collected after clear user consent unless the company meets one of five other legal bases, the most common of which are:
          a. the company has a contractual obligation to collect that particular data (contractual basis) or
          b. it must collect that particular data in order to provide the expected functionality of the product or service (legitimate interest basis)
    4. Only on iOS are MMP SDKs requesting user consent before collecting the cross-publisher device ID (IDFA), whereas on Android, the cross-publisher device ID (GAID) is collected unless the user specifically opted out, which is an option buried in the settings; on both platforms, most advertisers have the MMP SDK configured to collect other data like IP address for fingerprinting if the device ID can’t be accessed anyway.
    5. Recent GDPR rulings suggest that a contractual basis isn’t applicable even when only using first-party data like the IDFV (which based on ATT, doesn’t require user consent on iOS) to target ads (the European Data Protection Board ruled that the contractual basis wasn’t applicable in Meta’s case) since users were essentially forced to agree to the contract terms in order to use the product, which is expressly disallowed.
    6. Recent regulator advice suggests that a legitimate interest basis isn’t applicable even when only using first-party data like the IDFV to target ads (the Irish DPC advised TikTok to abandon their plans to use the legitimate interest basis for targeting ads with first-party data) since targeting ads ostensibly doesn’t constitute a legitimate business interest.
    7. Measurement is unlikely to be interpreted differently from ads targeting in any significant way in terms of the applicability of the contractual or legitimate interest bases (e.g., measuring the performance of marketing is not necessary to fulfill contract obligations to users nor is it a part of the expected functionality of the product).

      What can we conclude from this?

      1. All apps are in violation of the GDPR/French DPA if they:
          a. Have European users
          b. Use an MMP SDK
          c. Either:
              i. Are fingerprinting on iOS for users who do not consent to ATT (collecting data after an opt-out)
              ii. Have any Android users (there is no ATT-equivalent on Android, so no consent framework exists at all)

      2. Even if companies only collect data from users who have consented (which would require them to create consent dialogs on Android since the platform doesn’t have a built-in framework like ATT on iOS):
          a. Fingerprinting would be rendered unnecessary since the company would already be able to collect the cross-publisher device ID
              (much more accurate than fingerprinting) with the consent (currently, it’s used as a nefarious backup if the user denies consent)
          b. SKAN, which has tons of visibility issues, would be the only viable way to measure last touch on iOS
          c. Even worse, last touch measurement on Android would be almost impossible since the MMP would need consent from each user in the publisher app
              (the app that displays the ad) and the advertiser app (the app that buys the ad) to attribute any user (often called the double opt-in problem), which as
              we’ve seen after ATT was released on iOS, is incredibly rare.

      What can you do?

      The mobile industry is approaching another watershed moment. Do you have the right measurement to succeed? Fortunately, Polaris by MetricWorks is a turnkey, privacy-centric incrementality MMP.  Polaris does not need device IDs, painful migrations, heavy lifting, SDKs, or additional skills.  Most importantly, Polaris will help you avoid any issues with privacy regulators because it respects users’ privacy.

      In summary, for most app companies, the only real options to avoid similar massive fines are:

      1. Block access to European users completely (avoid jurisdiction of European regulators).
      2. Remove MMP SDKs from all apps and completely cease measurement activities.
      3. Continue using MMP SDKs, but ensure no device data is collected unless consent is granted (e.g., disable fingerprinting), meaning only deterministic last touch would be available and only for the few users that the MMP has double opt-in for (this may not even be possible for many MMPs at the moment and you’d still need a custom consent dialog for Android since there’s no ATT equivalent).
      4. Migrate completely to privacy preserving measurement methods that don’t require the collection of device data such as SKAN (iOS only), MMM, and geo lift testing (avoid collecting device data for the purpose of measurement altogether).

         

        If you’d like to discuss this topic further, feel free to book a time or contact us.

       


      1. Photo by Marija Zaric on Unsplash 

      Vibe.co offers radical transparency in CTV reporting thanks to new partnership with MetricWorks

      Vibe.co and MetricWorks are teaming up to make it easy for marketers to launch CTV campaigns with accurate incrementality measurement.  Recent years have seen a dramatic increase in global viewers moving from linear TV to CTV streaming services.  This trend has created a fantastic opportunity for marketers to leverage new  programmatic platforms like CTV DSP Vibe.co. 

      Launched just over a year ago by adtech industry veterans, Vibe.co aims to radically democratize access to television advertising by developing the first self-serve platform in the industry; becoming, in essence, the “Google Ads of CTV.”  Businesses big and small can now launch hyper targeted CTV campaigns in mere minutes with CPMs at or below other social media or Google prices, and 3rd party, real-time reporting. 

      Fig. 1: Vibe + MetricWorks allows advertiser to make big moves in CTV

      Vibe.co is singularly focused on developing the most powerful, simple, and transparent CTV tool in the industry. Nothing more, nothing less. 

      “We obviously believe in the power of CTV, and so do our investors who just funded a $6.35M seed round a mere three months after we released our self-serve feature, but advertisers are still wary of this new technology, and I don’t blame them,” says Vibe.co co-founder Arthur Querou. “For years, ad tech solutions were plagued with bloated account management teams, unnecessary features, and confusing reporting. But here’s what I want them to know: the CTV of 2022 is incomparable to its early days. Measurement tools and standards, especially, have developed in ways we couldn’t have dreamed of, even just a year ago. Vibe.co has kept its promise to our clients: launching the simplest, most transparent, most powerful platform. And now, thanks to MetricWorks, we finally have the opportunity to prove the true power and scalability of this exciting new channel,” he concludes. 

      MetricWorks is the leading provider of MMM-based incrementality measurement which closes the gaps in last touch measurement. MetricWorks’ incrementality MMP, Polaris, in particular, is a low effort, easy, drop-in addition that doesn’t require any extra SDKs, additional skills, device IDs or heavy migration lift.  Polaris offers an easy, turnkey “MMP experience” so that you can get started in 24 hours. It also integrates directly with your last touch MMP to provide a single repository for side-by-side comparisons of your last touch and incrementality data. 

      “We are excited to partner with Vibe.co to deliver a powerful solution that will make it easy for marketers to try out CTV with the assurance of comprehensive and granular incremental measurement that is far superior to last touch measurement,” said Brian Krebs, CEO and co-founder at MetricWorks. “Our partnership will enable marketers to leverage media mix modeling (MMM) to ensure that they achieve performance marketing success on CTV which has been an undervalued medium.” 

      If you are looking for an edge over your competitors that will elevate your marketing effectiveness, please contact either Dmitri or Chris (info below) to see how easy it can be:

      Dmitri Souffan
      Head of Mobile at  Vibe.co
      dimitri.souffan@vibe.co

      Chris Hoyt
      Chief Growth Officer at MetricWorks
      demo@metric.works