“Mobile apps in retail: Effect of push notification frequency on app user behavior”

Push notifications are a core functionality of mobile apps and allow app publishers to interact with existing app users and send promotional content. Since every push notification can also interrupt or annoy app users, the frequency of push notifications is a critical success factor. This study investigates how different frequencies of push notifications affect the behavior of app users of mobile apps in retail. In an experiment with 17,500 app users, five different frequencies are tested over seven weeks, and the effects on real observed app user behavior are analyzed. The results show that as the frequency of the non-personalized push notifications increases, uninstalls increase, and the direct open rate of push notifications decreases. A significant influence on indirect opens cannot be proven. The results provide practitioners with important insights into the potential harm that a too high frequency of push notifications can cause. Furthermore, the results support the importance of relevant content tailored to the respective user.


INTRODUCTION
With the spread of the smartphone, the mobile share of Internet traffic has increased strongly in recent years. Around 90 percent of Europeans are connected to the Internet (Eurostat, 2018), with mobile devices accounting for more than half of global Internet traffic (StatCounter, 2019). At the same time, digitization is becoming increasingly relevant in many areas (Deckert, 2019;Deckert & Wohllebe, 2021;Diez, 2020). In this context, the importance of mobile apps has also increased massively. The small applications from different categories like communication, organization, games, education, or retail are among the most relevant functionalities of smartphones (Ross, 2020;VuMA, 2017;Wohllebe et al., 2020).
From a company's point of view, push notifications are the central function of smartphone apps: The small messages can be sent via installed apps and appear on the lock screen or in the notification bar of a smartphone user. The user does not need to open the respective app to see the notification. Typically, companies or app publishers inform their existing app users about new content in the app in order to encourage them to open the app and -e.g., in retail or e-commercemake a purchase.
Earlier research suggests that notifications of software applications in a broader sense can also be perceived as interrupting and therefore annoying (Fischer et al., 2010;Iqbal & Horvitz, 2007;McFarlane, 2002).

LITERATURE REVIEW AND HYPOTHESES
Push notifications are a key feature of mobile apps. However, literature also emphasizes that notifications from software in general, but also from smartphone apps in particular, can be perceived as annoying. App users do appreciate a certain amount of entertainment value (Jacob & Gupta, 2017) and react very quickly to received notifications (Alsayed et al., 2019). However, the busier they are at the moment of receiving a notification, the more annoying they find these (Mehrotra et al., 2016). Push notifications are perceived as both, informative and annoying at the same time (Sahami Shirazi et al., 2014). Therefore, the literature suggests that any notification received by a user should be seen as an interruption and therefore as a form of cost to that user (Fischer et al., 2010). That is why, besides interactions with push notifications, app uninstalls have also to be taken into account to determine the success or failure of a push notification (Westermann et al., 2015).  (Vagrani et al., 2017). In fact, the frequency of push notifications tolerated by users seems to increase with the frequency of app usage, as a cluster analysis across multiple smartphone apps in the mobile health segment shows (Chen, 2017 The frequency of advertising messages in general and of push notifications in particular has often been the subject of scientific work already. However, there is a lack of quantifying the effect of the frequency of push notifications on app uninstalls and app opens. Both metrics can provide concrete information about the business consequences of a potentially too high frequency. In particular, experimental papers investigating real observed user behavior are missing (Wohllebe, 2020). Existing work either explores effects other than uninstall (Freyne et al., 2017;Pham et al., 2016) or is based on survey data instead of observed user behavior (Vagrani et al., 2017).
The aim of this study is to find out what influence different frequencies of generic, non-personalized push notifications of a mobile app in retail have on app user behavior. In line with the reviewed literature and the identified research gaps, the focus will be on app uninstalls and app opens.
Accordingly, the following four hypotheses are to be investigated: H4: The negative effect of frequency on direct opens is stronger than the effect on indirect opens.

METHOD
To test the hypotheses stated, an experiment is conducted with the mobile app of a German retailing company. In total, 17,500 app users are randomly divided into five groups of 3,500 users each. To exclude other factors than frequency, all groups are treated the same during the experiment. The groups are furthermore excluded from any other messaging activities. The experiment is conducted over a period of seven weeks in June -July 2020. In total, 16 generic non-personalized push notifications are sent, each drawing attention to products, special discounts or current promotions. The notifications are always sent at the same day of a week (Saturday) and at the same time of the day (5:30 pm). For the group receiving two notifications per week, the second day to send the message is also always a fixed one (Wednesday) at the same time. For technical reasons, notifications can only be sent to users that have not opted out from receiving notifications.
As the notifications do not contain personalized content, all look the same for all users receiving them. As the retailer's app is available only in Germany and the app is in German, all push notifications are in German as well. In the following, a couple of notifications are translated and shown.
• "Trend: Timeless products in black & white" • "Make your rooms cozier" • "20% discount on the most expensive product of your next order" • "Just today and tomorrow: Many products with free shipping!" • "Don't forget: Discover our most current offers now!" The frequency is experimented with • two messages per week, • one message per week, • one message every two weeks, • one message per month, • no message during the experiment period.
To determine the effect of the frequency over the test period, all groups receive a message at the beginning and end of the experiment period. This first and last message are then compared in terms of uninstalls (during the experiment period) and direct as well as indirect app opens (at the beginning and end of the period). Although interesting to examine as well, the data set provided by the company does not contain data about how time or money spent per frequency group changes over time. To better compare direct and indirect opens, Table  2   To test the four previously stated hypotheses, three regression analyses are calculated, whereby the frequency is interpreted as the number of messages per week and used as an independent variable. The dependent variable is chosen accordingly for each regression.
Investigating frequency and uninstalls, the uninstall rate is calculated as the quotient of the difference between the receivers at the beginning and end of the experiment and the number of receivers at the beginning of the experiment. For example, for two notifications per week an uninstall rate of 1 -(3274/3500) *100% = 6.457% is calculated. Due to technical restrictions, the exact time of the uninstall event is unknown. It is only known that an uninstall has happened between two push notifications. However, the experimental setup looks at isolated groups completely treated the same during the experiment. It is therefore assumed that the differences in uninstalls must be due to the differences in frequency.
In the case of the hypotheses for the direct and indirect opening rate, the open rate of the group "None" at the end of the experiment serves as the starting point. The difference between the open rate of "None" and the respective test group at this time is calculated as a percentage value. For example, for the group "Two per week" it is calculated that the direct open rate at the end of the experiment is 1 -(10.93%/15.35%)*100% = 28.79% lower than in the control group. Since the groups were randomly divided, it can be assumed that the minimal differences in open rates at the starting point across the different groups were caused randomly. This is verified by comparing the groups with the highest and the lowest number of opens. A chi square test does not indicate significant differences (X1) ², N = 7,000) = .4342, p = .5099).

RESULTS
First, the influence of push notification frequency on app uninstalls is investigated.
H1: With increasing frequency of push notifications, the probability of an app uninstalls increases. Table 3 shows the results of the regression analysis with the uninstall rate depending on the number of messages per week. Although the number of cases is quite small (n = 5) due to the consideration per frequency group, the overall model is significant (F = 50.17, p = .0058). It explains a large part of the variance of the dependent variable (R² = .9436).
According to the regression analysis results, one additional message per week increases the uninstall rate by 2.50 percentage points over the experi-ment period (β = .0250, t = 7.08, p = .006). Without a single message, the uninstall rate during the experiment period is 1.85 percent (β = .0185, t = 5.09, p = .015) according to the model. The actually observed value was 1.37 percent during the experiment. H1 сan therefore be confirmed: With increasing frequency of push notifications, the probability that a user will uninstall the corresponding app increases.
With regard to app opens, firstly direct app opens by tapping the notification are investigated. After that, indirect app opens by opening the app within a period of twelve hours after receiving a push notification (without directly tapping it) are examined.
H2: With increasing frequency of push notifications, the probability of a direct open decreases.  Assuming a significance level of α = 0.05, the corresponding overall model must be rejected or at least interpreted with great caution (cf. Table 5, F = 8.65, p = .0605). Nevertheless, the explained variance of the dependent variable by the model is still to be considered relatively high (R² = .7425).
When interpreting the influence of frequency, it is negative (β = -.0818), but overall it is not significant (t = -2.94, p = .060) and within a confidence interval that cannot be interpreted clearly (-.1704 < β < .0067). In this respect, H3 stating a negative effect of frequency on indirect app open rate is rejected. As a higher frequency on the one hand leads to a lower direct open rate (cf. results for H2), the results for H3 may show that users still remain interested in the content at a higher notification frequency.
Based on the results of the regressions (cf. Table 4 and Table 5), the hypothesis is tested that a higher frequency has a stronger negative effect on direct open rate than on indirect open rate.
H4: The negative effect of frequency on direct opens is stronger than the effect on indirect opens.
Even when assuming significance of the regression model and the coefficients for H3, the effect of the frequency on direct open rate is stronger than on indirect open rate (cf. Table 6). In this respect, H4 is confirmed regardless of whether the coefficient for the indirect open rate was significant.

DISCUSSION
Contributions to scientific theory and practical implications are summarized below. Subsequently, the limitations of this paper are pointed out and suggestions for further research are made.
The existing literature has already examined user behavior and acceptance of smartphone apps, and the influence of push notifications, in particular, frequently and in many different facets. In particular, the positive effect of push notifications on the activation of app users is emphasized again and again (Bidargaddi et  Lastly, a limitation of the research is the dataset. It was provided by a retailing company and does not contain data regarding time or money spent in app per frequency group. Different frequencies may influence these metrics as well. Further research should also take these metrics into account to gain even better understanding.
The limitations give rise to two areas, in particular for further research. On the one hand, the topic is still largely unexplored for other sectors like social media, gaming, or messaging apps. On the other hand, it should be explored to what extent employing user attributes and user behavior for sending notifications changes the acceptance of higher frequencies.
The data set used here covers a period of about two months. It could therefore also be interesting to take an even longer-term view.

CONCLUSION
This paper investigates the impact of push notification frequency in the context of mobile apps in retail on app user behavior. The focus is on the question of the impact on uninstalls and open rates. Based on the existing literature, four hypotheses are derived, three of which can be confirmed. To the best of the authors' knowledge, this is the first time that the effects of push notification frequency on app user behavior have been studied in an experiment with real app users.
Especially for practitioners who use push notifications as a marketing tool, three important implications arise.
First, the probability of an uninstall increases with the frequency of notifications sent. In this respect, every message sent, especially standardized, non-personalized, should be checked to see if the content is actually relevant enough and if it adds value for the app users.
Second, this work provides a concrete indication of the "costs" of a push notification, in particular in the form of uninstalls. For marketers, the increasing uninstall rate depending on the frequency can be an important basis to calculate the costs of a push notification. In practice, knowledge of the acquisition costs or the costs for an app install is necessary to do so.
Third, the H3 results indicate no negative effect of higher frequency on indirect app opens. Consequently, app publishing companies should also look at indirect app opens when evaluating the effects of push notifications. As a higher frequency does not lower indirect app opens significantly, an app publisher can reach out to their app users more frequently without any negative implications.