Local trend discovery on real-time microblogs with uncertain locations in tight memory environments

Abstract

This paper presents GeoTrend+; a system approach to support scalable local trend discovery on recent microblogs, e.g., tweets, comments, online reviews, and check-ins, that come in real time. GeoTrend+ discovers top-k trending keywords in arbitrary spatial regions from recent microblogs that continuously arrive with high rates and a significant portion has uncertain geolocations. GeoTrend+ distinguishes itself from existing techniques in different aspects: (1) Discovering trends in arbitrary spatial regions, e.g., city blocks. (2) Considering both exact geolocations, e.g., accurate latitude/longitude coordinates, and uncertain geolocations, e.g., district-level or city-level, that represents a significant portion of past years microblogs. (3) Promoting recent microblogs as first-class citizens and optimizes different components to digest a continuous flow of fast data in main-memory while removing old data efficiently. (4) Providing various main-memory optimization techniques that are able to distinguish useful from useless data to effectively utilize tight memory resources while maintaining accurate query results on relatively large amounts of data. (5) Supporting various trending measures that effectively capture trending items under a variety of definitions that suit different applications. GeoTrend+ limits its scope to real-time data that is posted during the last T time units. To support its queries efficiently, GeoTrend+ employs an in-memory spatial index that is able to efficiently digest incoming data and expire data that is beyond the last T time units. The index also materializes top-k keywords in different spatial regions so that incoming queries can be processed with low latency. In peak times, the main-memory optimization techniques are employed to shed less important data to sustain high query accuracy with limited memory resources. Experimental results based on real data and queries show the scalability of GeoTrend+ to support high arrival rates and low query response time, and at least 90+% query accuracy even under limited memory resources.

Appendix: Trend Line Slope

GeoTrend+ uses statistical linear regression slope to measure the trendiness of a certain keyword. The following Lemma derives the equation that determines the trendiness of a keyword:

Lemma 1

Given a keyword consecutive frequencies vector f =[f₀,f₁,...,f_N], thekeyword trend line can be estimated with the following formula:

$$ Trend_{reg} = \frac{{\sum}_{i=1}^{N} [i \times (f_{i}-f_{0})]}{N(N+1)(2N+1)} $$

(3)

Proof

The simple linear regression slope Trend_reg of x and y is given with the following equation:

$$ Trend_{reg} = \frac{Mean(xy)}{Mean(x^{2})} $$

(4)

Where Mean(x) is the average value of the vector and xy is a vector that results from value-wise multiplication of the vectors x and y. In GeoTrend+, the vector x values are always constants while the vector y contains the frequencies of a keyword W. Thus values of vector x are always be [1,2,3,...,N] while values of vector y are [f₁,f₂,f₃,...,f_N]. Thus, Mean(x²) can be simplified as $\frac {(N+1)(2N+1)}{6}$. On the other hand, Mean(xy) can be calculated as $\frac {{\sum }_{i=1}^{N} i \times f_{i}}{N}$. Substitutes both variables to Equation 1:

$$ Trend_{reg} = \frac{\frac{{\sum}_{i=1}^{N} i \times f_{i}}{N}}{\frac{(N+1)(2N+1)}{6}} = \frac{6 {\sum}_{i=1}^{N} i \times f_{i}}{N(N+1)(2N+1)} $$

(5)

The equation above assumes that the measurement is used from the start of the stream and each keyword W starts from frequency 0. However, in GeoTrend+, we need to consider the start position of a keyword W by using the previous frequency, namely f₀. Thus, the equation above can be modified to:

$$ Trend_{reg} = \frac{6 {\sum}_{i=1}^{N} [i \times (f_{i}-f_{0})]}{N(N+1)(2N+1)} $$

(6)

□