research-article

An Ensemble-based Approach to Click-Through Rate Prediction for Promoted Listings at Etsy

Authors:

Kamelia Aryafar,

Devin Guillory,

Liangjie HongAuthors Info & Claims

ADKDD'17: Proceedings of the ADKDD'17

Article No.: 10, Pages 1 - 6

https://doi.org/10.1145/3124749.3124758

Published: 14 August 2017 Publication History

Abstract

Etsy1 is a global marketplace where people across the world connect to make, buy and sell unique goods. Sellers at Etsy can promote their product listings via advertising campaigns similar to traditional sponsored search ads. Click-Through Rate (CTR) prediction is an integral part of online search advertising systems where it is utilized as an input to auctions which determine the final ranking of promoted listings to a particular user for each query. In this paper, we provide a holistic view of Etsy's promoted listings' CTR prediction system and propose an ensemble learning approach which is based on historical or behavioral signals for older listings as well as content-based features for new listings. We obtain representations from texts and images by utilizing state-of-the-art deep learning techniques and employ multimodal learning to combine these different signals. We compare the system to non-trivial baselines on a large-scale real world dataset from Etsy, demonstrating the effectiveness of the model and strong correlations between offline experiments and online performance. The paper is also the first technical overview to this kind of product in e-commerce context.

Supplementary Material

MP4 File (guillory_promoted_listing.mp4)

Download
277.82 MB

References

[1]

Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. 2009. Spatio-temporal Models for Estimating Click-through Rate. In Proceedings of the 18th International Conference on World Wide Web. ACM, New York, NY, USA, 21--30.

Digital Library

[2]

Deepak Agarwal, Bo Long, Jonathan Traupman, Doris Xin, and Liang Zhang. 2014. LASER: A Scalable Response Prediction Platform for Online Advertising. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, USA, 173--182.

Digital Library

[3]

Galen Andrew, Raman Arora, Jeff A. Bilmes, and Karen Livescu. 2013. Deep Canonical Correlation Analysis. In Proceedings of the 30th International Conference on Machine Learning. 1247--1255.

Digital Library

[4]

Olivier Chapelle, Eren Manavoglu, and Romer Rosales. 2014. Simple and Scalable Response Prediction for Display Advertising. ACM Transactions on Intelligent Systems and Technology (TIST) 5, 4, Article 61 (Dec. 2014), 34 pages.

[5]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 248--255.

[6]

Thore Graepel, Joaquin Quiñonero Candela, Thomas Borchert, and Ralf Herbrich. 2010. Web-Scale Bayesian Click-Through rate Prediction for Sponsored Search Advertising in Microsoft's Bing Search Engine. In Proceedings of the 27th International Conference on Machine Learning. 13--20.

Digital Library

[7]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778.

[8]

Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, and Joaquin Quiñonero Candela. 2014. Practical Lessons from Predicting Clicks on Ads at Facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising. ACM, New York, NY, USA, Article 5, 9 pages.

Digital Library

[9]

Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. CoRR abs/1411.2539 (2014).

[10]

Cheng Li, Yue Lu, Qiaozhu Mei, Dong Wang, and Sandeep Pandey. 2015. Click-through Prediction for Advertising in Twitter Timeline. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USA, 1959--1968.

Digital Library

[11]

Corey Lynch, Kamelia Aryafar, and Josh Attenberg. 2016. Images Don'T Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USA, 541--548.

Digital Library

[12]

H. Brendan McMahan, Gary Holt, D. Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, Sharat Chikkerur, Dan Liu, Martin Wattenberg, Arnar Mar Hrafnkelsson, Tom Boulos, and Jeremy Kubica. 2013. Ad Click Prediction: A View from the Trenches. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USA, 1222--1230.

Digital Library

[13]

Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. 2014. Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 1717--1724.

Digital Library

[14]

Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. 2009. Feature Hashing for Large Scale Multitask Learning. In Proceedings of the 26th International Conference on Machine Learning. ACM, New York, NY, USA, 1113--1120.

Digital Library

[15]

Jeonghee Yi, Ye Chen, Jie Li, Swaraj Sett, and Tak W. Yan. 2013. Predictive Model Performance: Offline and Online Evaluations. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USA, 1294--1302.

Digital Library

Cited By

Ban GYun HLee BSung DWoo SSerra ESpezzano F(2024)Deep Journey Hierarchical Attention Networks for Conversion Predictions in Digital MarketingProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680066(4358-4365)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3680066
Borgos-Rodriguez KPiper A(2023)Understanding Participation among Disabled Creators in Online MarketplacesProceedings of the ACM on Human-Computer Interaction10.1145/36101057:CSCW2(1-28)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610105
Demirel ÇTokuç ATekin A(2023)Click prediction boosting via Bayesian hyperparameter optimization-based ensemble learning pipelinesIntelligent Systems with Applications10.1016/j.iswa.2023.20018517(200185)Online publication date: Feb-2023
https://doi.org/10.1016/j.iswa.2023.200185
Show More Cited By

Index Terms

An Ensemble-based Approach to Click-Through Rate Prediction for Promoted Listings at Etsy
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
      1. Ensemble methods
    2. Machine learning approaches
      1. Neural networks
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
    2. Retrieval models and ranking
      1. Learning to rank

Recommendations

Click-through rate prediction with the user memory network
DLP-KDD '19: Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data

Click-through rate (CTR) prediction is a critical task in online advertising systems. Models like Deep Neural Networks (DNNs) are simple but stateless. They consider each target ad independently and cannot directly extract useful information contained ...
LOVF: Layered Organic View Fusion for Click-through Rate Prediction in Online Advertising
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Organic recommendation and advertising recommendation usually coexist on e-commerce platforms. In this paper, we study the problem of utilizing data from organic recommendation to reinforce click-through rate prediction in advertising scenarios from a ...
EXTR: Click-Through Rate Prediction with Externalities in E-Commerce Sponsored Search
KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Click-Through Rate (CTR) prediction, estimating the probability of a user clicking on items, plays a key fundamental role in sponsored search. E-commerce platforms display organic search results and advertisements (ads), collectively called items, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ADKDD'17: Proceedings of the ADKDD'17

August 2017

84 pages

ISBN:9781450351942

DOI:10.1145/3124749

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

KDD '17

Sponsor:

KDD '17: The 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 14, 2017

NS, Halifax, Canada

Acceptance Rates

ADKDD'17 Paper Acceptance Rate 12 of 21 submissions, 57%;

Overall Acceptance Rate 12 of 21 submissions, 57%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
417
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)5

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ban GYun HLee BSung DWoo SSerra ESpezzano F(2024)Deep Journey Hierarchical Attention Networks for Conversion Predictions in Digital MarketingProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680066(4358-4365)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3680066
Borgos-Rodriguez KPiper A(2023)Understanding Participation among Disabled Creators in Online MarketplacesProceedings of the ACM on Human-Computer Interaction10.1145/36101057:CSCW2(1-28)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610105
Demirel ÇTokuç ATekin A(2023)Click prediction boosting via Bayesian hyperparameter optimization-based ensemble learning pipelinesIntelligent Systems with Applications10.1016/j.iswa.2023.20018517(200185)Online publication date: Feb-2023
https://doi.org/10.1016/j.iswa.2023.200185
Bassani EPasi G(2022)A multi-representation re-ranking model for Personalized Product SearchInformation Fusion10.1016/j.inffus.2021.11.01081:C(240-249)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1016/j.inffus.2021.11.010
Plotnikov A(2021)Cost per click prediction in Google Ads on the example of the topic of self-employmentIV International Scientific and Practical Conference10.1145/3487757.3490815(1-6)Online publication date: 18-Mar-2021
https://dl.acm.org/doi/10.1145/3487757.3490815
Ai QNarayanan.R LDemartini GZuccon GCulpepper JHuang ZTong H(2021)Model-agnostic vs. Model-intrinsic Interpretability for Explainable Product SearchProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482276(5-15)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482276
Gharibshah ZZhu X(2021)User Response Prediction in Online AdvertisingACM Computing Surveys10.1145/344666254:3(1-43)Online publication date: 8-May-2021
https://dl.acm.org/doi/10.1145/3446662
Jiang JWu TRoumpos GCheng HYi XChi EGanapathy HJindal NCao PWang W(2020)End-to-End Deep Attentive Personalized Item Retrieval for Online Content-sharing PlatformsProceedings of The Web Conference 202010.1145/3366423.3380051(2870-2877)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380051
Liu SGu WCong GZhang Fd'Aquin MDietze SHauff CCurry ECudre Mauroux P(2020)Structural Relationship Representation Learning with Graph Embedding for Personalized Product SearchProceedings of the 29th ACM International Conference on Information & Knowledge Management10.1145/3340531.3411936(915-924)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3340531.3411936
Ai QZhang YBi KCroft W(2019)Explainable Product Search with a Dynamic Relation Embedding ModelACM Transactions on Information Systems10.1145/336173838:1(1-29)Online publication date: 18-Oct-2019
https://dl.acm.org/doi/10.1145/3361738
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents