The Tip of the Buyer: Extracting Product Tips from Reviews

Our research was conducted over five publicly available product datasets [26] from five e-commerce domains: Baby (baby clothing and supplementary products), Home Improvement (tools for home improvement), Musical Instruments (musical instruments, parts, and related accessories), Sports & Outdoors (equipment for sports and outdoor activities), and Toys (children’s toys and games). The datasets contain, per each product, its metadata (title, image, etc.) and all its associated user reviews. Table 1 depicts the main characteristics of the five datasets, including the number of total products and reviews, length of reviews in sentences, and so forth. The largest dataset is Sports & Outdoors containing 18,357 products with nearly \(300K\) reviews in total, while the smallest is Musical Instruments, with 900 products and a little over \(10K\) reviews in total. The median number of reviews per product ranges from 8 to 11, while the median number of sentences per review is between 4 and 5. The average number of sentences per review is around 6, varying from 5.71 in the Sports & Outdoors domain to 6.89 in the Home Improvement domain. To demonstrate the robustness of our methods and the ability of the proposed models to generalize, we use datasets from additional domains, including Automotive, Cellphones, Electronics, Fashion, and Health. For evaluation, we first consider all products above the 90th percentile according to their number of reviews in each of the five domains. Then, we randomly sample 50 products from each domain and select all review sentences of these products. Then, we apply our method on the sentences and present the top sentences (ordered by classification score) to be evaluated as a tip or not.

We define a tip as a short, concise, practical, and self-contained piece of advice. In general, tips can be useful both before and after the purchase. Before-purchase tips are useful to learn more about the product, and after-purchase tips are helpful when the product is already in hand. Both of these use cases are important for e-commerce platforms: Tips before the purchase help with the purchase decision by providing more information. Useful tips after purchases can increase customer satisfaction and motivate return to the site for additional shopping.

Despite the straightforward definition of a tip, there are some borderline cases that should be discussed. First, many review sentences may look like a tip, while they are very subjective and contain a personal experience (e.g., “In addition, we live in a colder climate and do not heat the house above 62 degrees at night, so combined with a little heater, this sleep sack does the trick.”). Such sentences are not considered as tips. Another type of borderline case not considered as a tip is a descriptive sentence (e.g., “Highly recommend it for outside purposes especially in the cold weather.” and “The cards are high gloss with full color pictures on them.”). Other non-tip sentences are obvious or trivial remarks (e.g., “Just make sure you have a bunch of batteries to get started.”) or those that repeat details already provided as part of the product description, without adding any new information (e.g., “Perfect for tuning electric guitars.” for a BROTOU Guitar Tuner that includes the following sentence in its description: “Fits most electric guitars”).

Labeling for training and evaluation in this work was performed by in-house annotators, after 3 hours of training and qualification tests. The pool included a total of 10 annotators, who were assigned tasks from each of the five domains randomly.² Unless otherwise stated, each evaluation was performed by a single annotator. Labeling was performed using a dedicated tool developed for this task. The tool’s user interface is depicted in Figure 1. As shown in the figure, each review was split into sentences. For each sentence, the annotator was asked to select whether it is a tip (as defined in Section 3.2) or not. For each selected tip, the annotator was asked to select the tip type (out of 10 pre-defined types, presented in the first column of Table 3). Afterward, annotators had to indicate if the sentence is a standalone tip or needs additional context. Moreover, annotators selected if this tip is useful before the purchase, after the purchase, or both. They could also (optionally) choose the previous and/or next sentence as useful information for extending the selected tip, regardless if it was marked as a standalone or not (see “extend tip” checkbox in the second column in the annotation tool screenshot). To measure the agreement between the annotators, we asked four of them to annotate the same 100 review sentences as tips or not. The Fleiss Kappa [20] among them was 0.815, indicating a high agreement level. All data annotated using the tool will be publicly released as an extension to the original public dataset.³

We sampled uniformly at random 14,000 product reviews across the five domains (2,800 per domain) from the original datasets (Table 1). The annotators labeled these 14,000 reviews, which included 85,171 sentences in total. Table 2 depicts the full statistics of the annotated dataset, including the number of labeled sentences and the collected tips across the five domains, along with the most common tip types. Overall, 3,846 sentences were annotated as tips, accounting for only \(4.52\%\) of all labeled sentences. This is a substantially lower percentage than the \(23.3\%\) reported for reviews of tourist attractions. Table 3 depicts the distribution of the 10 different tip types in our labeled tips. The most popular tip types were “Warning” and “Usage,” accounting each for slightly over \(30\%\) of all tips. The third most popular type was “Workaround,” followed by “Complementary product” and “Size.” The least popular tip type was “Other,” while about half of these tips related to differences between the actual product received and the seller-provided information (product image, title, or description; see example in Table 3). As depicted at the bottom of Table 2, some variance can be observed for the distribution of the top tip types across the five domains. “Usage” and “Warning” are at the top of the list in each of the five domains, with “Usage” the most common for Home Improvement and Musical Instruments and “Warning” for Baby, Sports & Outdoors, and Toys.

Another interesting characteristic is the connection to the surrounding context sentences. Most of the tips ( \(81.96\%\) ) were standalone, while only \(18.04\%\) were labeled as non-standalone sentences. As can be seen in Table 2, these results are rather consistent across the five domains. Overall, \(26.91\%\) of the tips could be extended to the adjacent sentence, with about two=thirds of these to the succeeding sentence and a third to the preceding sentence. For example, for the standalone tip sentence: “One note of caution, this is a very heavy router because it is a large plunge router,” the succeeding sentence was marked as an extension: “I mounted it on a Rockler X-Large router plate which is 1/4 inch thick aluminum, but it has a very slight bow in the middle.” For the non-standalone tip sentence: “You need a pipe cleaner to really get it,” the preceding sentence was annotated as an extension: “My one complaint is that there is an area that is hard to clean in the 4 piece nipple apparatus.”. Overall, however, the low portions of non-standalone tips indicate that our choice to focus on single-sentence tips covers the majority of the cases. We leave the expansion to multi-sentence tips for future work.

The number of tips marked as being useful before the purchase was somewhat higher than those marked as useful after the purchase: \(42.25\%\) versus \(27.61\%\) , respectively. The rest, nearly a third ( \(30.14\%\) ), were annotated as useful both before and after the purchase. These portions varied substantially across the different tip types, as depicted in Table 4. While “Population segment,” “Warning,” and “Size” are typically useful before the purchase, “Workaround,” “First-time use,” “Usage,” and “Maintenance” tips are more often useful after the purchase. “Alternative use” tips are prominently useful both before and after the purchase. We conjecture that alternate-use tips both can influence the purchase decision, as they reveal additional functionalities, and are handy when the product is in possession, extending its potential use. The substantial differences across tip types are also reflected in differences across the domains, as can be seen in Table 2. For instance, domains with a higher portion of “Warning” tips exhibit higher portions of tips that are useful before the purchase.

Tip Analysis. As a first step after obtaining the labeled data, we analyzed additional tip features. Table 5 presents the portion of tips according to different characteristics of the sentence in question, the originating review, and its authoring reviewer. It can be seen that longer sentences have higher likelihood of being tips: of the sentences consisting of 30 words or more, \(8.6\%\) were marked as tips, accounting for \(23.1\%\) of the tips in our dataset. At the other extreme, only \(1.9\%\) of the sentences composed of six to nine words were labeled as tips. Inspecting review characteristics, it can be seen that sentences that originate from short (1 to 2 sentences) and especially long (over 15 sentences) reviews have somewhat lower likelihood of being tips. As can also be observed from the table, sentences that originate from reviews with two or more “helpful” votes were more likely to be considered as tips, whereas opening sentences (positioned first) and those originating from reviews with no “helpful” votes had lower likelihood to be tips. Finally, we inspected characteristics of the reviewer who wrote the originating review. Interestingly, reviewers with many reviews on the site (130 and more) tend to include fewer tip sentences in their review. It could be that such “heavy” reviewers focus on other aspects in their reviews, such as personal experiences and opinions. Additionally, reviewers who had especially a lower portion of past reviews with at least one helpful vote produced a lower portion of tips, as can be seen in the last section of Table 5. We note that the results for each of the five domains demonstrated similar trends to those shown in Table 5.

Tip vs. Non-tip Language.. We also set out to examine the most prominent language differences between the two classes of sentences—tips versus non-tips. To this end, we used Kullback-Leibler (KL) divergence, which is a non-symmetric distance measure between two given distributions [4, 7]. Specifically, we calculated the terms that contribute the most to the KL divergence between the language model of the tip sentences versus the language model of the non-tip sentences and vice versa [24].

Table 6 presents the most distinctive unigrams and bigrams. It is noticeable that the most characterizing unigram of tip sentences compared to non-tip sentences is the second-person pronoun you, while the first-person pronoun i tops the non-tip list. This indicates that tip sentences are usually phrased in second-person language rather than first. The unigram my, which also reflects a first-person language, is also high on the non-tip list, as well as the plural we, while your is on the tip list. Other unigrams on the tip list include different prepositions and verbs, such as use and need, as well as the explicit suggest and note. The non-tip list includes was and had, which are often used to describe a past experience; the verbs like and love, which reflect subjective opinions; and adjectives and adverbs that reflect positive impressions, such as great, very, and nice, and the noun quality, which is often associated with the reviewer’s opinion on a certain feature of the product.

Inspecting the bigram lists, the tip list includes several expressions in second-person languages, such as if you, you can, you have, you need, and you use, and expressions typical to advice giving, such as make sure, will need, and be careful. The non-tip bigram list, on the other hand, includes many first-person expressions, such as i have, and i, i am, i got, and i was, in addition to expressions that reflect personal experiences, such as my son or i bought. It also includes the price, referring to a listing-specific characteristic, which may change from one seller to another, and phrases that reflect subjective feelings, such as a great, a very, and i love.

Since the data is very skewed, before applying any advanced classification approaches, we look for methods that would easily filter out non-tip sentences. After performing an analysis of basic features (e.g., sentence length) and words (KL), we could not observe simple rules that could massively be used for filtering, as done for the task of description generation [47]. This may indicate that our extraction task is more complex. Nevertheless, we identify a few rules that could save up to \(39\%\) of the labeling task. The proposed rules decrease the total number of sentences from 85,171 to 51,929 and increase the total percentage of tips from \(4.52\%\) (as reported in Section 3.4) to \(5.89\%\) . To derive many of these rules, we observed the top KL n-grams for \(n{\in }\lbrace 1,2,3\rbrace\) and considered those that hardly appear in tips, i.e., in fewer than five sentences in our training set, but do appear frequently in non-tip sentences. Our rule list includes the following:

(1) Short: Sentences of five words or fewer generally contain little information and rarely reflect any useful piece of advice. For example, “Recommended,” “Very good quality,” “Useful,” or “Will buy again” were among the most common short review sentences in our datasets. Analogously, in the travel domain it has been previously demonstrated that short sentences cannot serve as high-quality tips [23]. Overall, short sentences accounted for \(11.2\%\) of all review sentences across all domains in our dataset.

(2) Enthusiastic: Some reviewers tend to describe the product using strong-sentiment positive adjectives, such as “wonderful,” “adorable,” “amazing,” and “fantastic,” and verbs such as “love” and “like.” Examples include “I love this product and recommend it for everyone” and “Amazing quality, very useful for outdoor activities.” Overall, \(18.6\%\) of the review sentences matched this filtering criterion.

(3) Listing-specific: Review sentences that focus on listing-specific aspects, which may vary across different sellers of the product, such as price, shipping and return policy, warranty, product rating, and similar. Tokens used for filtering included “price,” “money,” “cheap,” “expensive,” “shipping,” “return,” “warranty” and also the dollar sign “$”. Examples of such sentences include “Very useful product for just 20$,” “Not very cheap but worth its money,” and “The shipping was almost as much as the panel itself.” Overall, \(14.7\%\) of the review sentences matched this filtering criterion.

(4) Personal: Sentences with a first-person pronoun, such as “i’m,” “i’ll,” and “i’ve.” As demonstrated in the previous section, such pronouns hardly ever occur on a product tip. Overall, \(8.4\%\) of the review sentences matched this filtering criterion.

We do not automatically filter out sentences from reviews with low ratings and do not run any sentiment analysis to filter out negative sentences, since these may hide useful tips, such as warnings, workarounds, or alternate use. For example, the sentence “Tried using it lighting with a match which worked, but the off switch would not shut the flow of fuel off completely” is a workaround tip for a Mini Jet Pencil Lighter, and “They fog up almost as quickly as my old Speedo goggles (several years old) which is after about one lap of the pool” is a warning tip for a Speedo Baja Swim Goggle. Both tips were extracted from negative (one and two stars) reviews.

Our rules are designed to filter out sentences that are very likely not to contain a tip; hence we prefer to apply only high-precision rules. As already mentioned, our rules filtered out \(39\%\) of the review sentences, leading to a tip portion of \(5.89\%\) in the remaining set.⁴

After applying the initial rule-based filtering, we explore a supervised approach, by training a classifier to predict whether a product review sentence contains a tip. We use the labeled dataset described in Table 2 and experiment with various classifiers:

Naïve Bayes. We examine a common model for text classification: Naïve Bayes [50]. Our features include textual features, specifically the unigrams, bigrams, and trigrams of the review sentence. We also experiment with a variant that includes additional features, based on the characteristics described in Table 5.

LSTM. A recurrent neural network based on a long short-term memory (LSTM) [29] architecture, with Global Vectors for word representation (GloVe) [49] pre-trained on the Wikipedia 2014 and Gigaoword 5 corpora.

LSTM with Attention. The attention mechanism enables the network to focus on relevant parts of the input [67]. The overall architecture of the “attention network” consists of two components: an LSTM-based word sequence encoder and a word-level attention layer. Given a review sentence split by words, we first embed each word using pre-trained GloVe embeddings, as previously described, and then use the LSTM network to produce the hidden states. The attention mechanism is often used to put more focus on certain words in the review sentence. For example, in the sentence “Make sure to switch off the guitar,” the words “make sure” receive higher weight, and in the sentence “Just be careful when opening the hood,” the higher weight is given to “just be careful.” We feed the word annotations through a single-layer perceptron network to receive a latent representation. Then, we calculate the similarity of the latent representation with a word-level context vector, normalized by a softmax function, to produce the word’s importance weight. We then construct the sentence vector as a weighted sum of the word annotations based on each word’s weight.

For both LSTM methods, we performed hyper-parameter tuning, which included the batch size, number of epochs, learning rate, and number of hidden units in the layers.

FastText. A library for learning word embeddings and text classification created by Facebook’s AI Research called FastText [5]. Each word is represented as a bag of character n-grams, and the final word embedding is the sum of character n-grams. This is useful for generalizing words with similar roots that appear in different forms (e.g., build and building). Hyper-parameter tuning was performed on the n-grams length, learning rate, and number of epochs.

BERT. A state-of-the-art technique for NLP pre-training called Bidirectional Encoder Representations from Transformers (BERT) [16]. This is a deep bidirectional, unsupervised language representation, pre-trained using only a plain text corpus. In contrast to context-free models, such as word2vec [44] or GloVe [49], which generate a single word embedding representation for each word in the vocabulary, BERT generates a representation of each word based on the other words in the sentence. BERT is useful for extracting high-quality language features from text data. In addition, it is useful for fine-tuning a model on a specific task, as we did in our experiments. We used the pre-trained “BERT-Base, Uncased”⁵ model to train a binary classifier for our task. Our hyper-parameter tuning included the batch size, number of epochs, and learning rate.

For all deep learning methods, both LSTM methods and BERT, we used Adam [35] optimizer.

We evaluate the algorithms over 6,118 sentences (a balanced set of 3,059 tips and 3,059 random non-tip sentences) and divide the labeled data to an \(80\%\) train and \(20\%\) test. We perform this evaluation 50 times, each with another random set of 3,059 sentences from the 48,870 labeled non-tips, and randomly split the 6,118 sentences to train and test sets. As previously mentioned, our goal is to produce a small number of high-quality tips, especially suitable for presentation on mobile devices, with limited screen space. Specifically, we aim to produce up to five tips per product with high precision, even at the cost of compromising some of the recall. Table 7 depicts the average recall results of the different classifiers across the 50 train-test samples, while we set the desired precision threshold to \(75\%\) , \(80\%\) , \(85\%\) , and \(90\%\) . Note that when presenting five tips, \(80\%\) precision yields an average of one wrong tip per product.

As can be seen, BERT outperforms its competitors at all precision levels and achieves the best results, with \(58.05\%\) recall for \(80\%\) precision. It can also be observed that the addition of the non-textual features described in Table 5 to the Naïve Bayes classifier does not show a substantial performance gain; we therefore did not work with these features in the remainder of our experiments. The best-performing configuration for the BERT classifier was with a batch size of 8, five epochs, and learning rate of 0.0002.

In order to further test the robustness and significance of the results, we preform a fivefold cross-validation tuning and evaluation. Similarly to previous evaluation, we used a balanced set of 3,059 tips and 3,059 random non-tip sentences. In each iteration, three folds were used for training, one fold for hyper-parameter tuning, and one fold for evaluation. Statistically significant differences of precision, recall, and area under the ROC curve (AUC) were determined using the two-tailed approximate randomization test [46] at a 95% confidence level, with a random sample of 10,000 permutations, following the recommendations in [17]. We apply Bonferroni correction for multiple comparisons. Hyper-parameter tuning was performed over the validation set, optimizing for the precision metric. Table 8 depicts the precision, recall, and AUC results over the different classifiers. It can be observed that the BERT classifier outperforms its competitors by a statistically significant margin. Due to its high performance, we focused on BERT in our exploration of multi-task learning approaches, as described in the next section.

In order to simulate the practical use case of extracting the tips from large sets of reviews, we use the following evaluation method. We run the BERT model on previously unseen products from the five domains, rank the tips by the model’s score, and select the top 5 tips. Then, we ask our in-house annotators to manually evaluate the generated tips. We specifically focus on popular products with many reviews, as for these we believe our method can work well even with low recall (as we mentioned before, we do not want to compromise precision). We check the quality for the scenario where we present to the user only a small number of tips: one, three, and five. This reflects the business need of extracting only few, but high-quality, tips, which can fit within a limited user interface space on the product page. This evaluation method also allows us to gain insights about the number of reviews required to produce high-quality tips per each product. We start by considering all products above the 90th percentile according to their number of reviews in each of the five domains (Table 11 presents the statistics of review number per product). Then, we randomly sample 50 products from each domain and apply the BERT classifier on all review sentences of these products. Finally, we present the top sentences (ordered by classification score) and ask the annotators to evaluate if they are tips. Each sentence is reviewed by two annotators and considered as a tip only if both agree on it. The results of the top 1, 3, and 5 sentences are depicted in Table 12. Inspecting precision@1 (i.e., the portion of sentences with highest classification score that are deemed as tips), Home Improvement demonstrates the highest performance with a perfect \(100\%\) , followed by Baby and Sports & Outdoors domains that attain a high \(98\%\) , and Toys with \(94\%\) . The lowest precision@1 is yielded for Musical Instruments, with \(90\%\) . Precision remains high in the Baby, Home Improvement, and Sports & Outdoors domains for the top 3 and top 5. For Toys, it is down to around 90% for the top 3 and top 5. For Musical Instruments, the sharpest drop is recorded, down to around \(84\%\) for the top 3 and \(80\%\) for the top 5. These results show that our method can produce top tips at high precision: the precision@1, precision@3, and precision@5 across the five domains are 96%, 92.47%, and 90.75%, respectively. The Baby, Home Improvement, and Sports & Outdoors domains include relatively higher portion of tips within their reviews (Table 2) and yield the highest tip precision (Table 12), implying they are especially suitable for tip extraction. Table 13 presents examples of extracted tips, including the product’s domain, title, and tip’s type.

To test the generalization of our approach and its applicability to other domains using the same training data, we trained our model on all the tagged data from the five domains mentioned above and applied them on five new (previously unseen) e-commerce domains: Automotive, Cellphones, Electronics, Fashion, and Health, using the same evaluation technique as in Section 5.2. The results of Precision at the top 1, 3, and 5 sentences are depicted in Table 14. Inspecting the top 1 tip, the best-performing domains were Automotive and Health with 94% precision. The other three domains achieved 90% precision. The Electronics domain was stable around 90% also in top 3 and top 5, in contrast to the Cellphones domain, which performed substantially worse (84% in top 3 and 80.4% in top 5). After an error analysis for the latter domain, we found that the tips are often longer and consist of several sentences, which were successfully detected by the model. However, since we evaluated only one sentence per tip, most of the tips were hard to understand without the surrounding context while presented standalone, and hence achieved lower performance.

The biggest drop was in the Fashion domain (82.67% in top 3 and 75.2% in top 5). We conjecture that the relatively low performance in the Fashion domain is due to the fact that the domain contains many tips of type Size (e.g., “Fits true to your normal size, so don’t order up.” for a Maternity Belly Band and “If you wear size large, purchase size XL.” for Women’s Stretch Cotton Yoga Pants), specifically 34.4% of the total number of tips, compared to only 5.49% in the original set of domains (as mentioned in Table 3); hence it is harder for the general purpose model to recognize those specific tips.

The Health domain also exhibited a drop between the top 1 and the top 3 and the top 5. Ultimately, for top 5, the best-performing domain is Electronics, followed by Automotive. Overall, we see that cross-domain learning works even for previously unseen domains. This finding is consistent across five different unseen domains, with precision equal to or greater than \(90\%\) precision for the top 1 tip, and between \(75\%\) and \(88\%\) for the top 5 tips. Note that despite the rather good performance of the model in the Health domain (and existence of quite a big amount of tips), this domain may be somewhat problematic for our application due to its sensitivity. For instance, consider the following tip extracted for melatonin tablets: “Do not use this as a replacement for visiting your doctor, but it is good enough for a general sense of your vitals between visits.” This example includes medical advice extracted from a review that was possibly written by a non-expert and hence should not necessarily be trusted.

Table 15 depicts the examples of the extracted tips (three examples per each domain). Note that even though the model was trained on completely different domains, it was general enough to find good tips in the new unseen domains. Nevertheless, there was a difference in the performance across the domains, since some domains are broader and contain a wide spectrum of tip types, while other domains include only specific tip types, which are less popular in the broader domains (e.g., Size type in Fashion).

To further explore the generality of our approach, we want to assess if our findings can be expanded to other e-commerce platforms. Specifically, after training the tip model on the five domains from a publicly available reviews dataset, we set out to evaluate it over eBay, one of the largest e-commerce marketplaces. As opposed to other leading e-commerce platforms, such as Amazon and Walmart, eBay’s “first-class” entity is a listed item (“listing”) offered for sale by a specific seller, rather than a catalog product. The catalog products are created by matching listings to existing products and creating new products for unmatched listings. The product notion is therefore somewhat weaker, and the results returned by eBay’s main search engine are listings rather than products. Having said all that, product pages, which include customer reviews, still play a key role in the overall user experience on eBay. Our experiments focus on two different domains, which have been especially popular on the platform during 2020 and 2021: Shoes and Watches. These domains are somewhat narrower than the five domains used for training and evaluation of our model, as described in the previous section. These differences allow us to examine the generalizability of our approach even further. We note that the Shoes and Watches domains are different in their properties. The popular products (exact definition is given below) in the Watches domain have twice⁶ the number of reviews compared to the popular products in the Shoes domain. Moreover, the reviews in the Shoes domain are shorter than the reviews in the Watches domain (average of 16.27 vs. 20.23 words per review and median of 9 vs. 11 words per review, respectively).

Similarly to the method described in Section 5.3, we consider the most popular products (products in the 90th percentile according to their number of reviews). Then, we randomly sample 50 products from each domain and apply the BERT classifier on all the review sentences of these products. Finally, we order the sentences by the classification score, present the top 5, and ask the annotators to evaluate if they are tips. As previously mentioned, each sentence is reviewed by two annotators and considered as a tip only if both agree on it. The results of the top 1, 3, and 5 sentences are depicted in Table 16. Generally, the results are comparable to those reported for other domains (Table 14), indicating that our model can generalize to new e-commerce platforms while maintaining good precision (over \(80\%\) for the top 5 tips). Across all the top-k values, but especially for the top 1, the model performed much better on the Watches domain than the Shoes domain. We conjecture that due to the fact that the Watches domain has on average more reviews per product and longer reviews compared to the Shoes domain, the probability to find a tip sentence in their reviews is greater. Interestingly, while performing error analysis for the results, we identified a noticeable difference from the results of the five domains from the public dataset: most of the wrongly identified tips in the eBay reviews data were seller related. As explained above, since the notion of a listing by a specific seller is strong on eBay, the reviews also contain feedback about the seller, such as the shipping time, responsiveness, money-back policy, price, and so forth. We therefore hypothesize that adding to the training process eBay-specific data can further improve the model performance and its overall generalizability. Nonetheless, the high performance results, especially for the Watches domain, indicate that there are common characteristics to the language of e-commerce reviews and tip expression across different e-commerce platforms. Table 17 presents examples from the tips extracted using our model for the Shoes and Watches domains on eBay.

BERT was the best-performing method in all of our evaluations, in particular for the unseen domains (from various e-commerce platforms) and the cross-domain evaluation. We conjecture that BERT outperformed its competitors due to its ability to learn implicit representations that capture a variety of linguistics characteristics. Our analysis indeed shows that there are domain-agnostic characteristics of tip sentences, such as the use of second-person pronoun and special phrases typical of advice. In addition, since BERT can be pre-trained over a large corpus and then fine-tuned using small data for a specific task, it can effectively generalize to new domains.

The main limitation in the described approach is the low recall. In addition, tip sentences are scarce. Particularly, every 22 review sentences contain only one tip sentence, on average. Even after applying the rule-based filtering, a tip appears every 17 review sentences, or every 4 to 5 reviews on average. These two limitations make our approach applicable to products that have accumulated a large number of reviews. Having said that, popular products are broadly exposed, and so their effect on a large number of users can be high and henceforth pave the way for tips receiving more prominence on e-commerce platforms.

Another limitation is the potential repetition of tips extracted for a given product. As the number of desired tips per product grows, diversification should be applied to avoid redundant tips. For example, the following usage tips, “To remove the marker designs from the screen you need to use a damp cloth then allow to completely dry - just a bit of a pain” and “To clean the screen off, you just run a damp paper towel or cloth over the screen,” were generated for the same Widescreen Light Designer. Semantic similarity can be used to cluster similar tips so that the final list is diverse. The clusters’ size can also assist in selecting the top tips to be presented. Such an approach was found productive in previous work on product description generation from reviews [47] and can be similarly applied in our case.

Tips generated from user reviews can be integrated in various use cases within e-commerce platforms, for both buyers and sellers.

Tips’ Applications for Buyers. The tips can be presented for buyers as part of the product page or for certain search results. In both cases the presented tips can support the purchase decision (e.g., Size, Complementary product, Warning tip types) or provide additional useful information to an already purchased product (e.g., First-time use, Maintenance, Workaround tip types). Figure 2 depicts a visionary user interface (UI) of a product page that presents tips. The UI contains the regular product information, such as its title, image, description, and attributes, together with a newly added section that contains a tip automatically extracted from the reviews. The number of tips may change; e.g., only one tip will be presented for users who browse using mobile devices and up to five tips for web browser users. As already mentioned, another potential use case is showing the tips on the search results page [15, 27, 56]. For example, when a user searches for running shoes, we can show the Size tips (e.g., “I suggest to purchase one size larger than usual cause they are small”) or Complementary product tips (e.g., “Buy extra batteries if you want to use them often”). Such tips may help customers make more informed purchase decisions and reduce return rates [52, 59, 70]. Moreover, the tips also can be used for diversified recommendations; e.g., the Alternative use tips can expose buyers to different types of product and avoid “more-of-the-same” recommendations. This may increase the satisfaction and engagement of the buyers and encourage them to return to the platform for their next online shopping. Another significant reason to return to the platform after the purchase may be the “after purchase” tips. In this list, buyers can view their purchased products together with the relevant First-time use, Maintenance, Workaround, and Alternative use tips.

Tips’ applications for sellers. Sellers can also benefit from the extracted tips. First, sellers may consider selling products suggested in the Complementary Product tips, as separate products or by composing a bundle and/or lot [38, 60] (e.g., batteries, chargers, and additional accessories). Other types of tips such as Size, Alternative Use, and Workaround may help sellers better describe their products, provide additional useful information, and gain more trust from the potential buyers. Warning tips may help the sellers to avoid offering dangerous products or suggesting risky products to the wrong population, e.g., a toy with small pieces to a baby. The platform may also suggest sellers to print the First-time use tips on the package as a nice service for buyers who are about to use the product for the first time. Since the tips are generated automatically, the platform should notify the sellers about newly added tips to the product they offer for sale. The seller should be able to see all the tips for each of their products in one place and the suggested action items as described above (e.g., printing instructions for first-time use, state the warnings upfront, or better describe their products in terms of size, maintenance, population segment, and alternative uses).

In summary, automatically extracted tips can be useful for both buyers and sellers in online marketplaces and their real-world uses should be further explored on different e-commerce platforms.

In order to examine the effect of our extracted tips in a real-world scenario, we designed a user study that compares product presentation with and without tips. We examined tips extracted from products in the following five domains: Baby, Home Improvement, Musical Instruments, Sports & Outdoors, and Toys. The goal of the study was to evaluate the usefulness of showing tips on product pages and get a sense of the potential impact of tip presentation on buyers’ experience and purchase decision. The study included 10 participants (6 female and 4 male, with ages ranging between 24 and 41), who were experienced online shoppers; i.e., each of them performed at least five online purchases during the last quarter of 2021. The study compared the effect of the presentation of products with tips (actual tips extracted using our method) to the presentation of products without tips. It included 25 randomly selected products (5 from each domain) that were presented to all of the participants. For each of the products, we extracted the top tip from its corresponding reviews using our previously described method. Each product was presented to half of the participants with a tip and to the other half without a tip. For each participant, half of the products (12 or 13) were presented with a tip and the other half without a tip. The order of the products was randomized per participant.

In the study, the products were presented to the participants using the user interface shown in Figure 2. Specifically, for each product, we presented the product’s title, image, description, and attributes. As explained above, in half of the cases the product also included an automatically extracted tip, which was presented under the “Tip from our customers” section. In order to examine the effect of tip presentation, the participants were asked to answer two questions: “Do you find this product attractive?” and “Would you consider to buy this product?” Participants were asked to rate their answers to both questions on a 5-point Likert scale, with 1 representing “not at all” and 5 representing “very much.” We did not disclose to the participants the topic of the study.

Tables 18 and 19 show the rating results for both questions in our user study. For each question, the average rating and the portion of five ratings out of all ratings are shown across all products presented with a tip versus all products presented without a tip. This comparison is shown for each of the five domains and in total, across all five domains. In general, as could be expected, the ratings for the first question (product attractiveness) are substantially higher than for the second question (consider buying).

Inspecting the average rating results in Tables 18 and 19, a consistent trend can be observed, where products presented with a tip receive higher average ratings than products presented without a tip. For the product attractiveness question, all differences are significant, while for the consider-buying question, they are significant for three of the five domains. For both questions, the largest rating difference in favor of products with tips can be observed for the Home Improvement domain, while the smallest gap can be observed for the Musical Instruments domain. This is consistent with our findings when evaluating the quality of tips, presented in Section 5, which indicated that the top tip is of highest precision for Home Improvement and of lowest precision for Musical Instruments (see particularly Table 12).

Examining the portion of products rated 5 indicates similar trends. For the product attractiveness question, large gaps can be observed across all five domains, as products presented with tips receive much more often the highest attractiveness rating. Noticeably, the smaller gap is attained for the Musical Instruments domain compared to all other domains, consistent with the generally lower quality of tips for this domain. For the consider-buying question, the portions of products rated 5 are generally low and likely affected by participants’ specific interests and needs (e.g., baby products are especially relevant to young parents, sports products are relevant to those practicing particular activities). The total difference across all domains in this case is still considerable between products presented with tips (17.6% rated 5) compared to products presented without tips (10.4%), but the gaps vary substantially across domains.

Overall, the results of our user study suggest that the presentation of a tip as part of the product page has a significant effect on whether potential buyers perceive the product as attractive and would consider buying it. These results are consistent across all inspected e-commerce domains and are more significant for domains for which higher-quality tips could be extracted. Future work should experiment with in vivo presentation of tips on product pages in e-commerce platforms to validate the findings presented in our study.