research-article

Adaptive Experimentation with Delayed Binary Feedback

Authors:

Carlos Carrion,

Weipeng YanAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 2247 - 2255

https://doi.org/10.1145/3485447.3512097

Published: 25 April 2022 Publication History

Abstract

Conducting experiments with objectives that take significant delays to materialize (e.g. conversions, add-to-cart events, etc.) is challenging. Although the classical “split sample testing” is still valid for the delayed feedback, the experiment will take longer to complete, which also means spending more resources on worse-performing strategies due to their fixed allocation schedules. Alternatively, adaptive approaches such as “multi-armed bandits” are able to effectively reduce the cost of experimentation. But these methods generally cannot handle delayed objectives directly out of the box. This paper presents an adaptive experimentation solution tailored for delayed binary feedback objectives by estimating the real underlying objectives before they materialize and dynamically allocating variants based on the estimates. Experiments show that the proposed method is more efficient for delayed feedback compared to various other approaches and is robust in different settings. In addition, we describe an experimentation product powered by this algorithm. This product is currently deployed in the online experimentation platform of JD.com, a large e-commerce company and a publisher of digital ads.

References

[1]

Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. 2009. Explore/exploit schemes for web content optimization. In 2009 Ninth IEEE International Conference on Data Mining. IEEE, 1–10.

Digital Library

[2]

Takeshi Amemiya. 1985. Advanced econometrics. Harvard university press.

[3]

Olivier Chapelle. 2014. Modeling delayed feedback in display advertising. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 1097–1105.

Digital Library

[4]

Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of thompson sampling. Advances in neural information processing systems 24 (2011), 2249–2257.

[5]

Georges Dupret and Mounia Lalmas. 2013. Absence time and user engagement: evaluating ranking functions. In Proceedings of the sixth ACM international conference on Web search and data mining. 173–182.

Digital Library

[6]

Tong Geng, Xiliang Lin, and Harikesh S Nair. 2020. Online Evaluation of Audiences for Targeted Advertising via Bandit Experiments. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 13273–13279.

[7]

Tong Geng, Xiliang Lin, Harikesh S Nair, Jun Hao, Bin Xiang, and Shurui Fan. 2021. Comparison Lift: Bandit-based Experimentation System for Online Advertising. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 15117–15126.

[8]

David R Hunter and Kenneth Lange. 2004. A tutorial on MM algorithms. The American Statistician 58, 1 (2004), 30–37.

[9]

Ramesh Johari, Leo Pekelis, and David J Walsh. 2015. Always valid inference: Bringing sequential analysis to A/B testing. arXiv preprint arXiv:1512.04922(2015).

[10]

Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. 2013. Online learning under delayed feedback. In International Conference on Machine Learning. PMLR, 1453–1461.

[11]

John D Kalbfleisch and Ross L Prentice. 2011. The statistical analysis of failure time data. John Wiley & Sons.

[12]

Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.

[13]

Tor Lattimore and Csaba Szepesvári. 2020. Bandit algorithms. Cambridge University Press.

[14]

Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of user engagement. In International conference on user modeling, adaptation, and personalization. Springer, 164–175.

Digital Library

[15]

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web. 661–670.

Digital Library

[16]

Larkin Liu, Richard Downe, and Joshua Reid. 2019. Multi-armed bandit strategies for non-stationary reward distributions and delayed feedback processes. arXiv preprint arXiv:1902.08593(2019).

[17]

Ciara Pike-Burke, Shipra Agrawal, Csaba Szepesvari, and Steffen Grunewalder. 2018. Bandits with delayed, aggregated anonymous feedback. In International Conference on Machine Learning. PMLR, 4105–4113.

[18]

Eric M Schwartz, Eric T Bradlow, and Peter S Fader. 2017. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science 36, 4 (2017), 500–522.

Digital Library

[19]

Steven L Scott. 2010. A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry 26, 6 (2010), 639–658.

Digital Library

[20]

Steven L Scott. 2015. Multi-armed bandit experiments in the online service economy. Applied Stochastic Models in Business and Industry 31, 1 (2015), 37–45.

Digital Library

[21]

Claire Vernade, Olivier Cappé, and Vianney Perchet. 2017. Stochastic bandit models for delayed conversions. arXiv preprint arXiv:1706.09186(2017).

[22]

Claire Vernade, Alexandra Carpentier, Tor Lattimore, Giovanni Zappella, Beyza Ermis, and Michael Brueckner. 2020. Linear bandits with stochastic delayed feedback. In International Conference on Machine Learning. PMLR, 9712–9721.

[23]

Zhengyuan Zhou, Renyuan Xu, and Jose Blanchet. 2019. Learning in generalized linear contextual bandits with stochastic delays. Advances in Neural Information Processing Systems 32 (2019), 5197–5208.

Cited By

Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Dai SZhou YXu JWen JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Dually Enhanced Delayed Feedback Modeling for Streaming Conversion Rate PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614856(390-399)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614856
Deng ADu MMatlin AZhang QSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed OutcomesProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599928(3937-3946)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599928
Show More Cited By

Index Terms

Adaptive Experimentation with Delayed Binary Feedback

Index terms have been assigned to the content through auto-classification.

Recommendations

Counterfactual Reward Modification for Streaming Recommendation with Delayed Feedback
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

The user feedbacks could be delayed in many streaming recommendation scenarios. As an example, the user feedbacks to a recommended coupon consist of the immediate feedback on the click event and the delayed feedback on the resultant conversion. The ...
Transitional feedback schedules during computer-based problem-solving practice

Feedback has a strong influence on effective learning from computer-based instruction. Prior research on feedback in computer-based instruction has mainly focused on static feedback schedules that employ the same feedback schedule throughout an ...
Nonstochastic bandits with composite anonymous feedback

We investigate a nonstochastic bandit setting in which the loss of an action is not immediately charged to the player, but rather spread over the subsequent rounds in an adversarial way. The instantaneous loss observed by the player at the end of each ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
235
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)2

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Dai SZhou YXu JWen JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Dually Enhanced Delayed Feedback Modeling for Streaming Conversion Rate PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614856(390-399)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614856
Deng ADu MMatlin AZhang QSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed OutcomesProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599928(3937-3946)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599928
Liu QLi HAo XGuo YDong ZZhang RChen QTong JHe QChen HDuh WHuang HKato MMothe JPoblete B(2023)Online Conversion Rate Prediction via Neural Satellite Networks in Delayed Feedback AdvertisingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591747(1406-1415)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591747

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents