Not What it Used to Be: Characterizing Content and User-base Changes in Newly Created Online Communities

The adoption of distinct norms has long been a feature of online communities. Burnett and Bonnici [4] surface this in an early qualitative study of Usenet newsgroups. They distinguish between explicit norms (e.g., rules and community FAQs) and implicit norms (uncodified community habits). More recent studies raise the possibility that newcomers may actually shift norms in the communities they join rather than merely adapting to the existing norms [7, 10, 11, 16, 22, 31].Danescu-Niculescu-Mizil et al. [10] provide evidence for this, finding that newcomers within an online community tend to use trending vocabulary. Similarly, Dev et al. [11] find substantive differences between newcomers and long-time users in StackExchange Q&A groups. Despite preliminary evidence that newcomers act differently, evidence that they drive changes in norms is limited. Lin et al. [22] examine a small set of Reddit communities that received a massive influx of newcomers, finding no evidence for changes in language use. On the other hand, Chan et al. [7] find that influxes of newcomers in subreddits lead to more comment removals by moderators, suggesting that newcomers may be more likely to violate explicit community norms.

On Reddit, volunteer moderators are often responsible for setting the explicit norms within their communities [31]. A substantial amount of research has explored the kinds of norms that moderators choose to enforce [8, 13, 30, 32]. Within a platform, it is common to see a wide range of explicit norms across communities [8, 14].Grimmelmann [14] argues that this variation in norms helps to promote diverse content.

Mechanisms for enforcing explicit norms vary. Penalties, such as bans and content removals, are most common [21], though example-setting [31] and mediation [30] are also utilized. The latter two strategies may also help to enforce both implicit and explicit community norms.

Moderators play an important role in managing activity as communities grow. Seering et al. [32] finds that moderators may deal with a greater frequency of norm violations as communities outgrow an initially homogeneous user group. Qualitative work highlights a few attributes of communities that contribute to effective newcomer management [18], highlighting the technological and leadership capabilities of moderator teams, as well as a shared sense of responsibility amongst the community [18]. In an empirical analysis of several subreddits that experienced massive, exogenous growth, Lin et al. [22] find that moderator responsiveness was associated with positive perceptions of content quality amongst community members. However, prior work has shown that moderation can lead to churn in community membership, suggesting that over-moderation may negatively affect growth [15, 33].

While we motivate our work through qualitative accounts of user experiences in online communities, we emphasize that we do not make explicit value judgments about any of our study measures. Rather, our measures were chosen based on qualitative accounts of community evolution. We contribute to existing literature by analyzing the relationship between all four of our measures in the context of a large-scale, generalizable dataset.

Following Mensah et al.’s [26] study of growing subreddits, we chose to collect data from all subreddits created between March 1, 2018 and December 31, 2019 that grew to 10,000 subscribers within two years of creation. This ensures that subreddits in our pool experienced a non-trivial amount of growth and allows us to track subreddits from their inception.

We used a three-step process to identify these subreddits. First, we used the public Pushshift comment archives to identify all subreddits containing at least one comment in 2018 or 2019. Then, we filtered out subreddits created before 2018 using the Reddit API. Finally, we used Pushshift to identify the largest recorded subscriber count for each subreddit every month. We filtered out any subreddits that never reached 10,000 subscribers.

For each subreddit in our sample, we use Pushshift to collect all comments from each subreddit‘s first two years of existence. For each comment, we extracted the source subreddit, author name, body text, creation time, and removal status. Because comments can be removed by moderators or deleted by their authors after being indexed by Pushshift, we use the Reddit API to update the comments’ final statuses. Following prior work [35], we apply a few simple filters to drop suspected bot accounts from our dataset. These filters, and an evaluation of their efficacy, are provided in Section B.

We refine our dataset to ensure that each month contains enough data to support reliable content- and user-level analyses. For each subreddit, we only analyze months where at least 50 distinct users and comments are present. Because we use an auto-regressive model in some of our analysis, we drop subreddits that do not contain at least two consecutive months that meet these criteria. We filtered out "Not safe for work" (e.g. pornography) subreddits and non-English language subreddits. Our approach for identifying non-English subreddits is included in the supplement. As a result, our final dataset consists of 1,620 subreddits, 12 million unique authors, 291 million active comments, 10 million removed comments, and 17 million deleted comments.

For each subreddit i in each month t, we compute four community-level measures across a two-year observation period. To respect users’ privacy, we exclude comments deleted by their authors from all analysis.

While the subscribers and removal rate measures are relatively straightforward to compute, our distinctiveness and diversity measures are more complex, relying on neural embeddings trained to capture similarity between Reddit comments and users, respectively [25, 36, 38].

To compute the distinctiveness score for a subreddit i in month t, we first generate an embedding for each comment posted to i in month t. If a month contains more than 1,000 comments, we randomly sample 1,000 comments to reduce computational cost. We compute a month-specific embedding for the subreddit using the average of these comment embeddings. To provide a baseline for Reddit-wide linguistic behavior in month t, we compute the average embedding of 10,000 comments ² randomly sampled from across all active non-NSFW, English-speaking subreddits in that month (potentially including subreddits in our sample of 1,620). Intuitively, subreddits are considered distinctive if their linguistic behavior differs substantially from this Reddit-wide baseline. Thus, the distinctiveness score of subreddit i in month t is one minus the cosine similarity between the subreddit-specific vector and the Reddit-wide baseline vector.

We use a fine-tuned S-BERT model [28] to compute the comment embeddings. To fine-tune our model, we use the generalized end2end (GE2E) loss function [38]. Intuitively, this loss function takes in an input set of comments, grouped by subreddit, and rewards comments for: (i) being close to their subreddit’s centroid and (ii) being far from other subreddits’ centroids. As such, subreddits whose centroids are closer together are subreddits that are harder to distinguish from one another in terms of linguistic behavior. This makes the loss function a good fit for our purposes, since subreddits with low distinctiveness scores will be harder to distinguish from the baseline Reddit-wide sample. All fine-tuning was conducted on a separate dataset of held-out subreddits (refer to section D).

Formally, GE2E loss takes a batch of N × M comments, where N is the number of subreddits, and M is the number of comments per subreddit. It then computes a centroid \(\vec{v}_{s}\) for each subreddit, as well as a similarity matrix S. This matrix contains the cosine similarities between comment embeddings \(\vec{y}_{ji}\) and the computed centroid. Following the suggestion in [28], we used the modified centroid \(v_{s}^{(-i)} = \frac{1}{M-1} \left(\sum _{m \ne i}^M \vec{y}_{jm}\right)\) when computing the similarity between v_s and y_ji when j = s. We scaled all similarities by learned parameters w and b, resulting in a matrix:

The final loss function is

For qualitative evaluation, we present the most and least distinctive subreddits in December, 2021 in table 1 a and table 1 c. The most distinctive subreddits tended to be goal-oriented. They focus on technical topics like legal advice, technology, or health and wellness advice. In contrast, comments in the least distinctive subreddits tended to be simple responses to humorous content. Notably, these subreddits tended to lack clear community-wide goals around commenting behavior.

We quantify the diversity, v_{i, t}, of subreddit i in month t based on a set of embeddings representing users. To generate user embeddings, we use a modified, context-based word2vec procedure [20] first proposed by Waller and Anderson [36]. Intuitively, this procedure creates embeddings for subreddits so that subreddits (i.e., “contexts") sharing many participants (i.e., “words") will have similar embeddings. We use a skip-gram model and negative sampling to train a set of user embeddings for each month i. The training set consists of all user-subreddit co-occurrences across Reddit in month i, not just those in our sample of 1,620 subreddits. This improves the robustness of the learned subreddit representations.

In each month, we represent a user as a weighted average of the embeddings of all subreddits that appear in their commenting history. These embeddings are weighted based on the number of contributions made by a user to a given subreddit. For each subreddit in our pool, we compute the centroid of embeddings of participating users in a given month. The diversity score of a subreddit is one minus the weighted average of the cosine similarities between each participant and the computed centroid. If there are more than 1,000 active users in a month, we randomly select 1,000 users with which to compute the score. Intuitively, if a subreddit consists of users who mainly participate in the same subreddits, the average cosine similarity between user embeddings and the subreddit centroid will be large, making the diversity score small. To train the embeddings in each month, we use the same hyperparameters as Waller and Anderson [36].

Table 1 b and table 1 d contain the most and least diverse subreddits in our sample in December, 2021. The least diverse subreddits featured insular commenters who rarely commented in other subreddits. In contrast, the most diverse subreddits were associated with more mainstream and general-interest topics. A full list of subreddits sorted by distinctiveness and diversity scores in December 2021 are included in the supplementary materials.

We used a Bayesian auto-regressive linear regression model to analyze the month-to-month changes in distinctiveness of each subreddit in our sample. We model each month-to-month change as a function of the growth in subscribers that a subreddit experienced from month t − 1 to month t. We measure growth by computing the difference in log-scaled subscriber counts between consecutive months (i.e., log (s_{i, t}) − log (s_{i, t − 1})), allowing us to estimate the effect of growth on changes to community distinctiveness, answering RQ1.

To assess the variation in trends across subreddits, we model the growth coefficients (α_i) hierarchically. In other words, each subreddit has its own coefficient governing the association between growth and change in distinctiveness. We model growth coefficients as being drawn from a Normal distribution with mean μ_α and standard deviation σ_α. This allows us to reason about the “typical" association between growth and change in distinctiveness and how this association varies between communities. We also include varying subreddit-specific intercepts β_i to account for the first month of data, where no distinctiveness change can be observed. Month-specific varying intercepts (θ_T) were used to account for real-world events, such as the start of the COVID-19 pandemic, that may have shifted Reddit-wide linguistic behavior. We use T(i, t) to refer to the absolute month corresponding to the t-th month of data for subreddit i.

Because the range of distinctiveness scores is between 0 and 1, we use a Beta likelihood function with logit-link to ensure that the model predictions d_{i, t} also range between 0 and 1. This yields the following final model:

We use a multivariate-normal prior for the subreddit-specific slope and intercept terms following McElreath [24]. Prior distributions for all model parameters are included in Section E.

We first characterize changes in diversity and distinctiveness over the study year range. We compare the distributions of subscriber counts, diversity scores, and distinctiveness scores in each subreddit’s first and last available month of data. On average, the first available month was between months four and five (M = 4.66, SD = 5.57). This is because many subreddits had little-to-no activity in the first few months of data. The last available month was between months 22 and 23 on average (M = 22.1, SD = 4.58).

fig. 2 illustrates the distribution comparisons. 60.5% of subreddits saw a decrease in distinctiveness (M_δ = −.019, SD_δ = .088), while 66.9% of subreddits saw an increase in diversity (M_δ = .019, SD_δ = .061). 97.1% of subreddits saw an increase in subscribers (M = 54, 040, SD = 176, 517), providing an extra level of validation that our inclusion criterion successfully identified growing subreddits.

We now look at the covariation of measures at a fixed point in time. We calculate Pearson’s r between each pairing of community size, distinctiveness, and diversity in the last month of data for each subreddit. We find a slight positive correlation between size and diversity (r = .0764, p < .05, CI\(_{95\%}\) [ − .0285, .121]), a slight negative correlation between size and distinctiveness (r = −.168, p < .05, CI\(_{95\%}\) [ − .212, − .118]), and a slight negative correlation between diversity and distinctiveness (r = −.118, p < .05, CI\(_{95\%}\) [ − .161, − .0747]).

Qualitatively, we find many subreddits that exemplify these observed trends. For instance, high-distinctiveness/low-diversity subreddits tend to be knowledge sharing groups for niche hobbies like r/IndieMusicFeedback and r/Pathfinder2e (a table-top roleplaying game). Meanwhile, low-distinctiveness/high-diversity communities tend to center on sharing humorous content and have few specific rules for commenting behavior. These include subreddits like r/CoupleMemes or r/SailorMood, two meme-sharing communities.

Still, fig. 3 demonstrates that there are many communities that run counter to the overall trend. For example, many subreddits related to physical and mental wellness (e.g., r/waterbros, r/veggieshake, and r/LifeAfterSchool) scored high on both diversity and distinctiveness. Low-diversity/low-distinctiveness communities tended to center on memes and casual discussion for niche user groups, like r/teenagersnew and r/Jesser (a subreddit for fans of a particular YouTuber).

We now present the results from our linear model. This model allows us to assess the association between monthly distinctiveness changes and our other study measures, answering our primary research questions. We emphasize that results from our regression should be interpreted as comparisons rather than causal statements. That is, our regression model allows us to estimate the average difference in monthly distinctiveness change between two subreddits that differ by one of the predictor variables.

Table 2 contains 95% credible intervals for key model parameters. Recall that our model is hierarchical in nature, meaning that each subreddit is given an individualized \(\bar{\alpha }_i\) parameter. Thus, the main effect of growth on distinctiveness is subreddit-specific. μ_α corresponds to the average of these main effects across all subreddits. As expected, on average, growth is significantly associated with a decrease in distinctiveness, answering RQ1. However, the estimate for the standard deviation of these subreddit-specific main effects is large, indicating substantial variation around this mean.

Surprisingly, we do not see significant interactions between removal rate and growth (γ, RQ2) or diversification and growth (η, RQ3), conflicting somewhat with results from prior qualitative work [18, 32]. We provide several possible explanations for this in section 6.Note that although the main effect of removal rate on distinctiveness change is significant, the size of the association is extremely small.

To make the strength of associations in our model more intuitive, we visualize effect sizes in fig. 4. We plot the predicted distinctiveness change associated with different monthly growth factors. We do this for all combinations of two hypothetical comment removal rates (0% and 5%) and three hypothetical changes in diversity score (-0.1, 0, and 0.1). Consider a set of subreddits with a 0% removal rate that experiences no diversification. Our model predicts that, after doubling in size, these communities’ distinctiveness scores would drop by .014 (95% CI [.012, .016]), on average. In communities that increase ten times in size, we would expect an average change in distinctiveness of .051 (95% CI [.044, .057]). To put these numbers into perspective, a change of .014 is roughly equivalent to the difference in distinctiveness between r/pothos (.513) and r/monstera (.494) – two subreddits about houseplants. A change of .051 approximates the difference between r/ratemydessert (.420) and r/ratemyplate (.370), subreddits for rating user-submitted pictures of desserts and more general food, respectively. Figure 4 also demonstrates that the predicted average distinctiveness changes are similar across different removal rate and diversification numbers, even though the main effect of removal rate is marginally significant. Although the predicted changes to distinctiveness are small, they correspond to modeled averages for changes over the span of a single month. Thus, our results are still consistent with larger changes experienced over broader time spans.

Although growth is associated with a decrease in distinctiveness, the mechanism underlying this phenomenon is unclear. One explanation is that newcomers, unfamiliar with community norms, make comments containing linguistic patterns that are different from that of the community. Thus, distinctiveness would decrease during growth periods when there are many newcomers. To test this hypothesis, we compare the distinctiveness of comments created by subreddit newcomers and returning users. For each subreddit-month pair, we define a newcomer to be a user who commented in the subreddit for the first time that month and a returner to be a user who commented in the subreddit that month and also in some previous month. We compute two distinctiveness scores for each subreddit-month pairing: a newcomer distinctiveness score, using only the first comment from each newcomer that month, and a returner distinctiveness score using the first comment from each returner that month. The returner-newcomer gap is the difference between these two scores. We use only the first comment for each user since newcomers may learn community norms during subsequent contributions. We compute these scores in months where there are at least 50 newcomers and at least 50 returning users. We compute these scores using a maximum of 100 newcomers or returners.

We use a hierarchical mixed effects model to analyze the data. We model the newcomer and returner distinctiveness scores as a linear function of the subreddit growth that month, plus a set of subreddit-specific random intercepts (refer to section F). In fig. 5 we plot our model’s posterior predictions for the returner-newcomer gap. We do this for 80 randomly selected subreddit-month pairs, half of which had high overall distinctiveness (above the 90^th percentile) and half of which had low overall distinctiveness (below the 10^th percentile). Although we do find that the returner distinctiveness scores are larger than those of newcomers on average, these differences are extremely small. Further, the returner-newcomer gaps were not significantly larger during periods of growth. Thus, it is unlikely that the returner-newcomer gap entirely explains the changes to distinctiveness observed during growth periods.

Our regression analysis indicates that, on average, subreddits become less distinctive during growth periods. Size and activity level are often used as measures of success in foundational online communities research [2, 5, 9, 17]. However, our findings provide quantitative support to a line of qualitative work that suggests communities may have a more complicated relationship with growth [16, 18, 32]. Hwang and Foote [16], for example, found that users felt smaller communities allowed for topical focus and niche discussion, which our linguistic analysis supports.

Although we find an aggregate negative association between growth and distinctiveness, our hierarchical modeling approach reveals substantial variation around this trend – in some communities, growth was actually associated with an increase in distinctiveness. The presence of this variation is nearly as important as the observed aggregate trend, as it indicates that growing communities are not destined to lose distinctiveness. Inspired by prior work, we explore two potential sources of this variation: degree of moderation [18] and community diversification [32]. Surprisingly, we find neither degree of moderation nor diversification to be significantly associated with responsiveness to growth. Below, we provide several potential explanations for this finding.