Finding a Depression App: A Review and Content Analysis of the Depression App Marketplace

Overview

We used a systematic review and content analysis approach based on a study by Bender et al [30] to guide the collection and characterization of available depression apps. The review was carried out on the five major app stores: Apple (iTunes), Android (Google Play), BlackBerry (AppWorld), Nokia/Symbian (Ovi), and Windows Mobile (Marketplace). On March 5, 2013, we entered the keyword “depression” into the search field on each of the four marketplace websites. The Apple apps were accessed through the iTunes interface using the same search term. The search term was applied across all store categories in the five instances. The two reviewers (MJL and NS) recorded the links and the titles of apps found in the search yield. Based on their availability, one reviewer (NS) compiled apps found in iTunes and the other (MJL) focused on the remaining app stores. For the eligibility assessment of the apps, the entire inventory was split into two equal samples for independent review.

Selection Criteria

Apps were organized as either “potentially relevant” or “not relevant” based on the app title, store description, and available screenshots. Apps were categorized as “potentially relevant” and included in the final analysis if they met three criteria: (1) the term “depression” was in the title or store description, (2) the app targeted health consumers (ie, those who self-identify as needing support for depression, including family or caregivers), rather than health care professionals, and (3) the app had an English-language interface or English translation (if in another language).

Apps were excluded from the study if they did not provide sufficient information, did not have a clear focus on depression, used the term depression in an unrelated context (eg, the Great Depression), used the term depression as a keyword in a list of unrelated items or as background information, and were duplicates appearing in multiple markets or for other devices (ie, optimized for tablets). The duplicate that provided the most information for data extraction was retained based on the following hierarchy (most to least information): Google Play, iTunes, AppWorld, Ovi, and Marketplace.

After independent screening for relevance, the 2 reviewers exchanged a random selection of 5% (104 apps) of their search yields to verify eligibility. Interrater reliability (IRR) of the random samples, as determined by Cohen’s kappa (kappa=.77, P<.001), was statistically significant. According to Landis and Koch’s guidelines [31], the score indicated that there was a “substantial agreement” between the 2 reviewers. Because the IRR exceeded the pre-determined minimum kappa threshold of .7, independent reviews of the whole sample were not required. Disagreements found in the exchanged sample were resolved by consensus.

Data Extraction and Coding

Information was extracted from the store descriptions of the apps for the following variables: commercial information (ie, year of release/update, cost, developer name, audience, downloads), organizational affiliation, content source, main purpose, user interface, media type, and popularity (ie, rating, number of raters, number of comments). The 2 reviewers (MJL and NS) collectively and iteratively developed a preliminary coding scheme by analyzing the content of 20.5% (108/528) of randomly selected “potentially relevant” apps. The coding for the main purpose variable used the Luxton et al [17] classification of mental health app (ie, self-assessment, symptom monitoring, psychoeducation, psychological therapy, psychotherapeutic skills training) as the foundation for development. An IRR test of 20.4% (22/108) of this pilot sample was conducted to evaluate understanding and application of the codes. The results were all significant (P<.001), yielding “almost perfect” agreement for exclusion (kappa=1.00), affiliation (kappa=.91), content source (kappa=1.00), and user interface (kappa=.91). There was “substantial agreement” for main purpose (kappa=.77) and “moderate agreement” for media type (kappa=.49) [31]. The discrepancies in coding for the multimedia variable were discussed, and problem areas were identified and resolved. The final coding scheme is outlined in Table 1.

The remaining sample was divided for data extraction based on odd and even numbering to ensure that the reviewers had equal proportions of apps from each marketplace. After independent review, 20% (combined 41 apps) of each reviewer’s sample was randomly selected, exchanged, and coded to assess IRR. The results were all significant (P<.001) with “almost perfect agreement” for affiliation (kappa=.89) and main purpose (kappa=.83), and “substantial agreement” for user interface (kappa=.74). There was also “substantial agreement” for media type (kappa=.68); however, the low kappa (kappa<.70) required the reviewers to examine and understand the discrepancies in coding and correct the coding within each of their respective samples. This process was also applied to content source (kappa=.53). Flagged apps were collectively reviewed for inclusion and then coded. Because the exclusion criteria became more nuanced during this process, apps that were labeled not relevant were also collectively reviewed and coded if they were considered relevant.

Data Analysis

Cohen’s kappa and descriptive statistics were computed using SPSS version 20. Chi-square tests of independence examined the relationship between the variables data source, user interface and multimedia, and the main purpose of the app. Statistical significance was set at P<.05. The option to collapse the values within a variable to fulfill the expected cell frequency assumptions of chi-square tests was explored if the research team viewed it as a logical transformation.

General Characteristics

The initial search yielded 1054 apps, of which 53 were excluded as duplicates (31 were available in two stores, eight in three stores, two in four stores, and one in all stores). Of the remaining apps, 243 met the inclusion criteria. Figure 1 shows the exclusion of apps at the various stages of the study. See Multimedia Appendix 1 for a list of the included apps.

Windows (4.5%, 11/243), Nokia (2.5%, 6/243), and BlackBerry (2.5%, 6/243) accounted for less than 10% of the included sample, as the majority of apps were from the Google (53.5%, 130/243) and Apple (37.0%, 90/243) marketplaces. The apps spanned 32 different store categories, with 79.9% (194/243) of the apps found under four categories: health and fitness (41.2%, 100/243), medical (17.3%, 42/243), lifestyle (14.4%, 35/243), and books (7.0%, 17/243). Six (2.5%, 6/243) apps had no categorization. The average price for paid apps (152/243; 62.6%) was CAN $3.15 and ranged from $0.99 to $15.99. The majority of paid apps (73.7%, 112/152) were sold for less than $4.99, with the mode price of $0.99 (18.9%, 46/243).

Only the release date was provided by the iTunes store, whereas Google Play, BlackBerry, and Windows provided dates of the last app update. Nokia did not provide this information. The earliest date reported by the app stores was 2009 (3.7%, 9/243). Two-thirds (66.0%, 156/237) of the apps were released or updated in 2012 (36.2%, 88/243) and the first quarter of 2013 (28.0%; 68/243). Google Play was the only market that reported the number of installs (ie, downloaded and installed on an Android mobile device) and was reported in ranges; 40 apps (30.8%, 40/130) were installed less than 50 times. The most frequent ranges of installation were 100-500 and 1000-5000, each registering 16.9% (22/130) of the sample. One app (0.4%, 1/243) was installed in the 1 million to 5 million range, and four apps fell into the 100,000 to 500,000 range.

Developers and Affiliations

There were 190 developers in the sample, with 35 accounting for multiple apps. Of this group, 27 developers created two apps, three developed three apps, and four developed four apps. The top developer, MOZ, created nine apps. Only 5.3% (10/190) of the developers were either medical centers (1.0%, 2/190), universities (1.0%, 2/190), and institutions (3.2%, 6/190). A total of 56 developers indicated that they were a commercial developer (eg, LLC, LLP, Inc.), while 124 developers did not provide sufficient information about their affiliation.

Depression Apps Ratings

Of the 113 rated apps (46.5%, 113/243), there was an average of 37.2 raters (95% CI 21.6-52.81) per app. One app had 583 raters. The average rating (out of five stars) was 3.5 stars (95% CI 3.3-3.7). There was an average of 5.9 comments per rated app (95% CI 4.2-7.7), with a range from zero to 56 comments.

Overall Picture of Depression Apps

Over 80% of the apps had the main purpose of providing therapeutic treatment (33.7%, 82/243), psychoeducation (32.1%, 78/243), or medical assessment (16.9%, 21/243). Apps with multiple purposes accounted for 7.4% (18/243) of the sample. Only 38.3% (93/243) of the apps reported the content source in sufficient detail and mainly cited an external (17.7%, 42/243) or expert (14.0%, 30/243) source. The majority (72.4%, 176/243) featured a dynamic user interface. Over half of the apps were text-only (51.9%, 126/243), while 14.4% (35/243) used multiple forms of media. Table 2 summarizes the distribution of apps across the different variables.

The chi-square tests of independence yielded significant results (P<.001); however, the expected cell count assumption was violated in all cases. Two variables, affiliation and content source, were collapsed into binary variables. The chi-square analysis for affiliation (ie, sufficiently or insufficiently reported) and main purpose showed that there was no relationship between the two variables (χ² ₅=8.8, P=.12). The content source variable (ie, sufficiently or insufficiently reported) showed a significant (χ² ₅=50.5, P<.01) association between the main purpose of the app and the reporting of the source. An ad hoc analysis was conducted between media type and user interface, which yielded a significant relationship between the two variables (χ² ₄=46.3, P<.01).

Characterization of Apps by Main Purpose

Principal Findings

This review found that depression apps provided support on five different dimensions: therapeutic treatment, psychoeducation, medical assessment, and supportive resources. Through the iterative development of this typology and understanding of the available commercial information, the results provided some insights into the user experience of those seeking depression support through apps. Similar to a recent study by Martinez-Perez et al [24], this study found that depression app seekers need to filter through 400+ apps in either the Google Play or iTunes marketplace. In context of the one million app milestone announcements by both Google and Apple in 2013 [42,43], this number may suggest that the app marketplace has entered a phase of “overload” or “diseconomies of scale”, where the large quantity of apps available makes it difficult for users to find the right one [44,45]. The apps excluded from this study indicate that metadata may play a role in this phenomenon. Vendors may leverage the use of metadata or the keyword “depression” to increase exposure of their non-depression apps in the depression app search results. For example, one-fifth of the search yield made no mention of depression anywhere in the app title or store description. One-quarter of the search yield was excluded because the word depression was mentioned only in a “laundry list” of keywords in the app’s description, not in the title. Many of these apps were white-labeled (ie, essentially identical but marketed for different purposes or under different developer names) and were evident by the identical store descriptions (see Figure 3). White labeling was primarily observed for e-book and audio therapy apps. Last, although some apps made reference to depression, their main purpose was to address a different condition (eg, weight loss or acne apps may describe how being obese or having acne may lead to depression).

Of the apps included in the study, there were three times more text-only apps than any other media category; furthermore, almost all the text-only apps with static interfaces were found in the psychoeducation app category. The reviewers found that these apps, based on screenshots and descriptions, were rudimentary in function and minimal in design. The proliferation of these apps may be a result of the low barrier to entry into the marketplace in the form of prerequisite resources and skills, thereby allowing those with minimal programming skills and resources to develop and publish their own apps [46]. This finding could explain why only one-third of the 190 unique developers adequately described or indicated their affiliation and the proportionately low number of apps from formal institutions. Furthermore, only a third of the app store descriptions reported content sources. Many other app reviews [18-23,30,47-51] have also found that the app development process often failed to involve health care professionals or academics and to include content aligned with clinical guidelines or behavior change theories or techniques. The majority of these apps were categorized under the main purposes of psychoeducation and therapeutic treatment.

The lack of apps that incorporate authoritative sources remains problematic. It has been estimated that one in five of paid apps claim to treat or cure medical ailments [28]. Similar to the potential shortcomings of information found on the Internet, the information or therapies provided by apps may be incomplete or based on insufficient scientific evidence. This presents a potential health hazard for consumers who interpret this information incorrectly or try inappropriate treatments [52]. For example, reading about a disease may increase health anxiety, reinforce hypochondriasis, cause unnecessary concerns, or lead people to purchase harmful drugs or engage in risky health behaviors [53]. These harms, however, are often a cautionary claim, as most research on the utility of online health information has focused on the quality of information rather than its effects [54,55]. Only a few studies actually reported instances of harm [56]. This gap between evidence-based recommendations and app functionality continues to be a common theme across different health conditions [20,21,47,51,57-59]. Public attention has turned to these “snake oil” apps, prompted by a US Federal Trade Commission settlement involving two app developers who falsely cited a study from the British Medical Journal of Dermatology in their claims that the colored display screens featured in their apps could cure acne [60]. The proceedings were founded on the premise of false advertising rather than public safety [61]. This case has led to a call for the US Food and Drug Administration (FDA) to regulate mobile medical apps; however, there is debate about the appropriateness of this measure [62]. In September 2013, the FDA issued guidance for developers of apps that perform as medical devices, defined as apps that diagnose or treat disease whereby malfunctions can carry significant risks of harm [63].

Based on the app store categories used in this study, 42 apps were defined as medical; however, this category included apps that are considered innocuous, such as those that help patients organize their health information or look up information about treatments [64]. Perhaps these apps would be better suited for other categories, such as health and fitness, lifestyle, and books, where more than half of the included apps were found. Apps found in these non-medical categories are considered low risk as long as they do not provide specific treatments or treatment suggestions. They may provide benefits to the patient, such as those associated with using a mood tracker to maintain a symptom diary [65]. To help users navigate the app marketplace, Happtique (a subsidiary company of the Greater New York Hospital Association) developed standards for an app certification program in early 2013. Unfortunately, these efforts were brought to a halt when an audit found that 2 of the 19 Happtique-certified apps had privacy issues [66]. There are other initiatives to help curate apps, such as the iMedicalApps website; however, it is a tremendous task to benchmark. Policing the quality of apps is a near-impossible endeavor that is reminiscent of the early days of appraising online health information [67]. Deshpande and Jadad have found that past initiatives to assess the quality of online health information or tools had limited success and recommend that efforts be hedged towards an open, distributed, and collaborative approach similar to Wikipedia [68].

Evaluation

The most common function of depression apps provides users with information about depression through an e-book modality. Despite the potential to translate books or bibliotherapeutic guides, only 13 of the 50 e-books cited a content source. The majority of these books were self-help guides, often with titles that claimed they would help users overcome depression. Examples include “Beat Depression”, “Defeat Depression”, and “Stomping Out Depression”. While these non-sourced books do pose the potential to distribute erroneous or biased information to people seeking help, the Google dataset shows that two-thirds of these apps are installed less than 100 times and indicates that users do exercise some discretion before purchasing or installing apps. Nettleton et al [69] suggested that users are able to make reasonable assessments of health information in the context of other health information seeking practices to complement their formal care. This behavior extends to mobile phone apps: one qualitative study found that the reputation and legitimacy of sources factor into the use of an app [70]. For example, an e-book app that cited the US National Institutes of Health was downloaded within the 10,000 installs range. While promising, this finding could be confounded by the application’s free status. The “Anxiety and Depression” and “Audio Book Anxiety and Depression” e-book apps, which were in the install ranges of 10,000 and 100,000, were also free. One study suggested that consumers exercise more caution when having to purchase apps than when downloading them for free due to the burden of price [71]. The same study also showed that ranking, customer ratings, and content size affect downloading when the app is free. Consumers depend more on their own information and experiences rather than on rankings or ratings when the app requires payment. They closely consider low ratings, including complaints, not mean score when they have to pay [71]. The relationship between price, affiliation, source, downloads, and satisfaction via ratings and comments could be a potential area to explore in future studies.

Medical assessment was the only app category with a high rate of reporting content source. All of these apps were screening tools that allowed users to self-diagnose for depression. There is an absence of published data investigating the impact of patient self-diagnosis using apps or the Internet; however, some studies have identified false positive assessments as a potential source of harm [53,72-74]. Despite this shortcoming, medical assessment apps could help to address some systemic barriers to diagnosing depression in primary care [75]. Depression is often under-detected in the health care system, and the practice of routine screening is a contentious and unresolved issue [76]. Medical assessment apps may help to bridge this gap by assisting individuals in identifying mental health issues, thereby providing the impetus to approach and engage their health care providers. Clarke and Yarborough described this effect as a lowering of threshold of entry-level mental health services so that it extends the reach of care to people who do not seek traditional treatment for depression [5].

Audio therapy apps may have a similar potential to that of medical assessment apps [77,78]. This study found that half of therapeutic treatment used audio therapy and is consistent with a recent report that found that 43% of therapeutic apps used audio for treatment [28]. The effectiveness of audio therapy, regardless of mode of delivery, is not fully understood and is often under scrutiny [47,79-81]. There are many gaps in knowledge regarding the psychological effects of brainwave entrainment and hypnosis on depression [82,83]. Systematic reviews [81] and meta-analysis [84] of existing research have found mixed results on the effectiveness of these types of interventions. A similar review of a hypnosis app found on iTunes reported that none of the 407 identified apps were tested for efficacy or were based on evidence [47]; however, the study did not discuss potential harms associated with using non-evidence-based, non-evaluated apps. The authors do caution against “self-described professional titles”, as certification could easily be purchased online. They also warn that certification does not mean that the individual was adequately trained.

The fourth most prevalent function of depression apps was offering behavior training or therapy, with most apps focusing on CBT. Internet-based CBT (ICBT) has shown to be an effective treatment for depression [85], with the magnitude of effects depending on level of support and content of the intervention [86]. ICBT is considered to be well suited for delivery through an app because it would offer users the convenience of recording and tracking their moods and context in real time, as well as accessing psychoeducational materials [87]. Two-thirds of the CBT apps identified in this study had multiple purposes, which often included tracking, screening, and providing psychoeducation. In practice, one study demonstrated the feasibility of app-based CBT in treating depression, with clinical improvement in the patients [26]. This app was captured in the sample and provided a very brief description mentioning the CBT program and its affiliation with a hospital; however, the raters felt it did not provide sufficient information about the intervention source. This shortcoming underscores the importance for app developers to follow a standardized reporting system to advertise the credibility of apps and to prevent empirically tested apps from going unnoticed. Similarly, it might be necessary to develop a framework that could protect both app developers and users from harm, particularly from liability associated with cases of preventable suicide.

Limitations

While the development of regulations and certification standards for assessing the quality of apps is underway, this study used the information available in the app store description (ie, developer affiliation and content source) to understand how depression apps are advertised to health consumers seeking depression apps. The information provided about affiliation and content source was accepted prima facie based on the developed inclusion criteria. The high percentage of insufficient reporting of affiliation may be an overestimation, since the developer websites were not examined to corroborate their status. Similarly, the reported content sources were not further examined. It is acknowledged that the apps themselves may contain more information and that not downloading and testing the apps is a limitation of this study. The lack of physical testing mirrors the actual user experience when making the decision to download apps [48], where the information provided in the description may serve as an initial proxy measure for quality before downloading and trialing an app. It also underscores the need for a standardized app store description reporting system for vendors to refer or adhere to. With over 190 unique developers identified in our eligible sample and many more in the initial sample, consumers may not have the time to view all the developer websites to verify their affiliations. Requiring vendors to outline their affiliations, evidence base, or content source could provide potential users with enough contexts to assess the credibility of the app.

A second limitation lies in the possibility that many of the apps excluded from this study because they were not depression specific could potentially be useful for people with depression. ICBT apps are prime examples of potentially useful non-depression-specific apps. ICBT is regarded as a well-established treatment for depression, panic disorder, and social phobia, but it is also an option for 25 other clinical disorders. While ICBT apps could be the prototypical depression app [26,88], non-depression ICBT apps were excluded to maintain consistency in assessing the relevance of other apps that provided an intervention (eg, binaural beats [81], yoga [89], spirituality [90]) where a case could be made for their inclusion. To prevent confirmation biases from entering the sample, it was decided that the app was required to be specific to depression to be eligible.

This study represents a snapshot of depression apps found in Canadian app stores in March of 2013. This may be a limitation in three ways. First, the landscape of the depression market will have changed at the time of submission of this publication. Second, the findings from this study may not be representative of all the depression apps available on the global market because certain apps may be localized or licensed only to specific countries. The study by Martinez-Perez et al in Spain found over 1537 depression apps available on the five major platforms. In comparison, the current review yielded 1001 unique apps, with a large part of the discrepancy attributed to Google Play app count. Moreover, a sample of Android apps may be missing because this study was conducted just prior to the Amazon announcement [91] of expanding access to its Android app store outside of the United States to Canada and 200 other countries. A quick search of the Android app store using the search term “depression” yielded 123 apps. Because development standards vary from different app stores, future content analysis studies should consider including the Amazon marketplace to understand its contributions to the app marketplace. Last, frameworks such the Self-Certification Model for Mobile Medical Apps by Health on the Net Foundation (HON) [92] and App Synopsis [93] became available shortly after the data extraction phase concluded (mid-2013). These models provide some important parameters that were not covered in this study (eg, data requisition and management, advertising policy, justification of claims). However, this study demonstrates that most apps would fare poorly against the aforementioned standards and delineates the need for such reporting approaches to be disseminated to mHealth developers to bring the information presented to health consumers to an acceptable level.

Conclusions

This study found that finding an appropriate depression app may be challenging due to the large quantity available. The search results yielded non–depression-specific apps to depression apps at a ratio of 3:1. Over one-quarter of the apps excluded from the study failed to even mention depression in their description or title and exemplify the role of metadata in populating the search results. The lack of reporting of organizational affiliation and content source brings the credibility into question. Whether the content is evidence-based is a whole other issue. This lack of information was most common among symptom management apps, followed by therapeutic treatment and psychoeducation apps. Only medical assessment apps, many of which were based on well-established depression questionnaires, adequately described their sources. As the app phenomenon and health consumerism continue to grow, the user’s ability to find a reliable and credible app may become increasingly difficult. While efforts are underway to populate the marketplace with certifications and professional vetting, this study delineates the need for standards in reporting and for a framework to enable people with depression or other conditions to use proxy measures to assess the legitimacy of apps.