Everything you need to know about marketing testing

Welcome to the eighth edition of MAMA Boards, an AppsFlyer video project featuring leading mobile marketing experts on camera.

For today’s mini whiteboard master class, we have Janos PereiCMO at TreasureHunt, an award-winning, Berlin-based mobile games studio.

Even though marketing testing is a well-known, integral part of user acquisition and marketing efforts, many marketers still miss at least some part of this process. Janos takes a deep dive into the complete process of mobile marketing testing, from hypothesis to detail testing, and later offers solutions to the top 4 major challenges typically faced. 

Real experts, real growth. That’s our motto.



Hello, everyone. Welcome to another edition of MAMA Boards by AppsFlyer. My name is Janos. I am the CMO of TreasureHunt, an award-winning, Berlin-based mobile studio which has delivered a number of casual mass market games to the gaming app market.

Today, we are going to be talking about everything that you need to know about marketing testing.

Marketing testing will be very important for you because that’s the one activity that will help you to create sustainable business results through testing and iterating and making your products better.

Let’s start with the objectives.


The beginner’s guide to incrementality for remarketing



So whenever completing marketing testing, we will need to be clarifying our research objectives. Some questions to consider: What do we want to achieve with these activities? What is the reason we are testing at all? Our objectives could be broken down into two main branches.

The first main branch would always be exploration. In the case of exploration, we will be asking open-ended research questions and trying to figure out qualitative questions about our target audience.

To give you an example of that: whenever you try to identify, essentially, the personal profile of a user base. Let’s say that, when you get some acquisition campaigns, you do the route targeting and, after that, you use some sort of tool to read out the audience profile – for instance, Facebook Audience Insights.

As the second branch, we will be talking more about optimization. Optimization is a more hypothesis-based, iterative optimization loop, in which we would be looking more for qualitative results. A typical example of this could be, for instance, creative optimization for different marketing creative, let it be for user acquisition, or more for branding and visibility.

So, once we have identified our research objectives, we will need to go to our robust process of testing. The reason why we need to have such a process in place is to make sure that we will be using the same methodology, which is how we will be getting comparable results over time and making sure that our testing is as accurate as possible.

marketing testing

Testing process

Whenever talking about research process, we will, of course, first start with the goal setting. Just like discussed previously in the research section and objectives, we will either need to formulate our hypothesis or ask some research question – in both cases, what we are looking to answer.

The second step would always be the actual high-level concept testing. High-level concept testing essentially refers to the idea of testing high-level, rough ideas at a very crude level to make sure that we identify strategic directions.

To give you a concrete example, let’s talk about an app icon testing. This would be the stage at which you would use different type of high-level conceptual ideas. For instance, in gaming, you could use a character, a close-up of a character, an abstract symbol, a weapon, a key element, or item inside the game, etc. Once we are testing each and every one of these against the others, we would be able to see the strategic direction. Which direction should this research and optimization be going?

At that point, we move to detail testing. Detail testing is when we will come up with a final, fine grain product for the test results. We would essentially use the high-level strategic directions previously identified and push our goal to a final polish.

Continuing with the app icon example, let’s say that we have selected an abstract symbol and the hero character from our concept testing results. Now, we’re going to push and polish both of these assets to the final stage, the final readiness. This is where we will decide about the final coloring, layout, and execution, whether they will be 3D, 2D, or flat designs. Then, we take these final assets and put them into a very detailed, native context – for instance, a fake app store.

Once we are able to complete this testing phase, we should be getting a final result that’s ready for publishing and production. This is the moment that we get into the live testing.

Live testing simply refers to the idea that, whenever we test something, we always need a limited sample size. We cannot test with our whole audience. However, at the final stage, we will still need to be publishing all of these assets and ideas live.

For example, this would be the moment when you publish your tested app icon into the app store to your general population and audience. Based on the learnings, how the actual general population responds to your tested icons, you will then be able to further reiterate and make sure that that icon will be the best it can be. This is the moment when the loop closes.

Once we are getting live actual data from the performance, we will be able to look ahead to the next (and first) stage and formulate the next set of research question for further optimization.

marketing testing processes

Test set-up

Setting up testing essentially relies on five key pillars. Now that we have identified how the process works, the first and most important pillar will be setting up the test correctly.

The first step will be the design itself. We need to come up with a structure, an idea, what we would like to test. This is the moment we will build a test plan – in an Excel sheet, a bullet point list, a Power Point deck – where we will be asking the research questions that we discussed previously, as well as what we are aiming to achieve.

In the second step, we will be setting up the test environment. It is very important that we always test an individual element as much as possible, instead of multiple. If we have too many variables changing and we don’t control for the test environment, our results will be inconclusive. We will not be able to identify and measure what is actually causing the positive, or for that matter, the negative change in performance.

The third step will, of course, be the data collection. We need to make sure that we have the correct tools, the correct measurement systems in place that are able to collect the data in a correct and valid fashion. It’s something that will also be comparable and used for analytics at a later stage.

And that ties into the final two stages.

The fourth step is that we do need to analyze the data. We need to run it through the different analytics processes. Let it be something as simple as an Excel table to very sophisticated like Python-scripted analytics and high level analysis.

Last but not least, please don’t forget that we will need to be working on interpreting the results and understanding the findings. This step will concretely help you to formulate the goals we need to achieve and the actions for implementing it into practice. Because, of course, having abstract results is really good, but at the end of the day, we’re doing this to drive business results.

One thing that is very important at this stage – please don’t forget to always remember the limitations of your research. It is always very tricky to have over-generalized outcomes and make very wide assumptions about how certain things might be behaving in a context you haven’t tested.

So once we have all this test setup correctly, that’s when we are ready to go ahead. Now, at this stage, I would like to talk a little bit about the most well known and common caveats.

marketing testing - setup

What are the potential pain points and challenges when working with marketing testing?

I collected a number of key challenges that I have seen in different projects that have come up as issue.

The first one is unrealistic test planning.

Again, setting up for testing is very pivotal. If you do not have the right setup, you will not be asking the right questions and therefore you will not be arriving at the right conclusions. Because of this, make sure to always have a very robust process with your peers, with your colleagues.

Have your test plan challenged and verified by other people, not just on your own. That will be helpful for both you and the company in cultivating a very robust testing culture and pushing your goals forward.

The second challenge is usually with having too many variables that we change in a testing. To give you an example, we would not just change, let’s say, a high level color, but also different characters, different items.

For that test, we would introduce too many variables to the creative process. In turn, that makes it very difficult to point out testing insights. “Okay, did the positive change come from this or that element?” “Did we change the character because of the color coding? Or is it because of the layout and the design?” So the simpler you can design your test, the better it is.

In economics, there is a basic principle called Ceteris Paribus, which refers to leaving everything else unchanged. Therefore, if possible, try to have one variable, or maybe a very limited set of variables, that you change and whose impact you can understand and control.

The third challenge would be analysis paralysis. Having been there and done that, unfortunately, sometimes, we all get into a situation where we have an ample of data set. We just try to dig deeper and deeper going down the rabbit hole and we get into a situation where we don’t see the forest because of the tree that is standing in front of us. Again, relying on data is pivotal.

Data will be the mechanism that helps you drive your business forward. Having said that, please make sure that you never lose sight of the original objectives. It is always very important to remember what you were actually trying to achieve and what are your research questions, your hypothesis.

These objectives will guide you through the testing process. And again, we need to highlight the importance of clearly formulating a research question and hypothesis, since they will help you to stay on track.

The fourth and final challenge is over-generalization. As we discussed when we were talking about the correct testing setup, we always need to be aware of the limitations of our research. What are the individual building blocks that we cannot push further?

For instance, you run a test, and you run it to a limited audience of a couple of thousand users. It would be incorrect to apply your test results to millions of other people and assume that they are behaving in the exact same fashion as your test group.

Therefore, whatever actions we take based on our test plan, do understand the limitations. Of course, we can always be hopeful and push for broader boundaries, but we still need to have both feet on the ground, so to say.

In that sense, again, well-designed testing boundaries and a well-designed test plan will ultimately guide you in understanding and acting on your results.

marketing testing - key challenges

Three key ideas for successful marketing testing

To summarize what we have been talking about, in terms of the testing process, we can identify three key ideas that will help us with running tests successfully.

The first one is planning. Planning is of pivotal importance. Good planning will save you a lot of time later on if you take the necessary time and effort to plan things out correctly. You will actually be able to get through a faster execution cycle with higher precision.

The second key idea is leveraging your data analytics. Please make sure that you have data analytics, with your environment control and all the research methodological pieces well in order. You do need to produce results that will be comparable and could be used over a longer period. Because of that, you will need to make sure you have robust methodological underpinnings.

Finally, please don’t forget that all these findings and all these results are only valuable if put into action. You will need to consistently iterate on the entire process, and making sure that you are pushing the boundaries, both in terms of keep testing continuous, as well as ensuring that your testing process is more robust and generally improved.

These three pillars combined together in synergy will underlie the foundations of solid testing, and will help you drive sustainable business results in the future.

So this concludes our session on marketing testing. Thank you all for watching. Please make sure to leave your comments in the section below. Make sure to also check out some of the other very interesting MAMA Board videos through this URL.

Wishing you the best of success for your testing. Bye!

Ready to start making good choices?