Predictive Modeling: Basic Concepts and Measurement Set-Up
Why Build Models?
There are numerous benefits to predictive modelling in mobile marketing, but we narrowed it down to two key marketing activities:
- User Acquisition: Knowing your typical user behavior and the early milestones that separate users with high potential and users with low potential can be useful on both the acquisition and re-engagement fronts. For example, if you know that a user needs to generate X by day 3 to make a profit after day 180, and that number comes in under your benchmark, you know that you will need to adjust bids, creatives, targeting, or other things to improve the cost or quality of users acquired, or else improve your monetization trends. If it is over your benchmark, then you can feel confident in raising budgets and bids to get even more from where they came from.
- Re-engagement: If users with some potential are nonetheless under-performing early on, you can focus your re-engagement efforts (whether through paid or owned channels) on those users to help them catch up with your curve. If you identify at-risk customers, you can try to re-engage with them long before you have to win them back. If your data shows there is a very low probability of long-term success, you can exclude them from your paid campaigns altogether to prevent further investment in the wrong users.
What Should I Measure?
To understand what you need to measure to be best positioned to get your predictions right, let’s explore which data points are useful for which means, and which are not.
Just like the square and rectangle relationship, while all metrics are data points, not all metrics are key performance indicators (KPIs). But just because a metric isn’t a KPI, doesn’t mean that the metric is not useful. Metrics are easier to calculate and mature much sooner than KPIs, which tend to take longer or involve complex formulas.
1) Legacy metrics (low confidence in predicting profit, fastest availability):
- Click to install
Required data points: impressions, clicks, attributed installs.
2) Early indicator metrics (medium confidence in predicting profit, fast availability): In the age of down-funnel focus, an install is no longer a sufficient KPI. That said, the following metrics, while not useful for use in predicting profit, are still useful as early indicators informing marketers whether their campaigns stand a chance of turning a profit.
Examples of early indicator metrics:
- Cost per install
- Retention rate
Required data points: cost, attributed installs, app opens (retention report)
With the exception of retention rate, metrics tend to be tied to a marketing model rather than your business model, and as such, are not useful for determining whether the users you acquired will turn a profit for your business.
If you pay $100 per click or per install, chances are that you’re not going to turn a profit. If your CTR is .05%, chances are that the auction mechanics will force you to pay a high rate per install, again leaving you with less margin to turn a profit.
Where metrics fail to support predictions is when you try to tighten your confidence range to a finer accuracy, such as where the line of profitability is within a $2 to $6 CPI range.
It is important to sub-divide the common KPIs into two types:
1) Tier 2 KPI confident predictors (medium-high confidence in predicting profit, slow availability): Useful to serve as early benchmarks of profit, offering more confidence than leading indicators (metrics), but taking more time to mature than leading indicators and possessing less confidence than tier 1 KPIs.
- Customer acquisition cost (paying user)
- Cost or conversion of key actions (e.g. ratio of games in first day played, or ratio of content views during first session)
- Time-based cost or conversion of key actions (e.g. cost per number of games played in first day or cost per content view during first session)
- Cost per day X for retained user: Total spend per day times the number of retained users on that day.
- Vertical specific in-app events: e.g. tutorial completion, completing level 5 on day 1 (gaming), number of product pages viewed in 1st session, number of sessions in 24 hours (shopping), etc.
Required data points: cost, attributed installs, app opens (retention report), in-app events configured and measured, session data (time stamps, features used etc.)
For most business models, these KPIs cannot serve as confident predictors because, while they do account for cost and events commonly correlated with profit, they miss the full monetization side of the profit equation, given that app opens don’t always equal in-app spend, and paying users may buy more than once.
2) Tier 1 KPI confident predictors: early revenue and consequent ROAS as indication of long term success (high confidence in predicting profit, slowest availability): These KPIs either take a longer time to fully mature or else involve complex processes to determine. However, they tie directly into your business model, and as such, are suitable for predicting your marketing campaigns’ profitability.
- Return on Ad Spend (ROAS)
- Lifetime Value (LTV) of a user
Required data points: cost, attributed installs, app opens, in-depth revenue measurement (IAP, IAA, subscription, etc.)
While ROAS is easier to calculate, it requires weeks or months of time for users to continue generating revenue curves. LTV, while immediately providing an estimate of profit alongside CPI or CAC, is a complex model that can easily lose accuracy depending on applications and assumptions.
To wrap up, here’s where each approach is situated in the following chart: