Best Practices for Building Mobile Marketing Prediction Models
Building predictive models is a complex process with numerous factors affecting its robustness. Clearly, there is a lot of experimentation that’s required to find the right model. The following are some important tips to remember:
1) Feed the Beast
When building data models or systems that are used to guide significant decisions, it’s not only important to build the best system possible, but also to perform ongoing testing to ensure its effectiveness. For both purposes, make sure that you continuously feed your profit prediction model to keep it trained on the most relevant data.
In addition, always check whether your model’s predictions come to fruition based on new observations, or at least close to it. Not following these steps could mean that a model with an initial useful prediction power could go off the rails depending on seasonality, macro auction dynamics, your app’s monetization trends, or many other reasons.
By observing your leading indicators or early benchmarks and looking for significant changes in data points, you can gauge when your own predictions are likely to break down, too. For example, if your model was trained on data where the average day 1 retention rate ranged from 40%-50%, but for the stretch of a week, the day 1 retention rate dropped to 30%-40%, this could indicate a need to re-train your model. That might be especially true given that quality signals from the users you most recently acquired have shifted, likely leading to changes in monetization and profit, all else equal.
2) Choose the Right KPI for Predicting Profitability
There are several options to choose from, each with a set of trade-offs in viability, accuracy, and speed to produce recommendations. Go ahead and test different KPIs (e.g. more or fewer days of ROAS or LTV) and use one or all of the following to compare the profit prediction power of several KPIs:
- A ratio of success-to-failure at satisfactory predicting
- Mean Absolute Percentage Error
You may be surprised at how poorly correlated the standard measures prove to be.
3) Segment Your Data
Segmenting users into more homogenous groups is not only a great way to improve conversion rate, but also a proven method to reduce noise and improve the predictive power of your model. For example, applying the same model to both interest-based campaigns and value-based lookalike campaigns could lead to less effective results. The reason for this is that monetization and length of lifetime trends of users from each unique audience target are likely to be significantly different.
4) Remember to Factor Time
Most marketers are aware of the influences of seasonality on breaking down predictions, but the lifecycle of your app/campaign/audience/creative can also influence the ability of your model to make accurate predictions.
The acquisition cost trends in the first week of a new app launch will be much different from those in the fifth month, the second year, and so on, just as the first thousand dollars in spend in a previously untapped lookalike will be different than the ten thousandth and fifty thousandth dollar in spend invested into the same lookalike (especially without changing the creative used).