research-article

Leveraging Watch-time Feedback for Short-Video Recommendations: A Causal Labeling Framework

Authors:

Yang SongAuthors Info & Claims

CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

Pages 4952 - 4959

https://doi.org/10.1145/3583780.3615483

Published: 21 October 2023 Publication History

Abstract

With the proliferation of short video applications, the significance of short video recommendations has vastly increased. Unlike other recommendation scenarios, short video recommendation systems heavily rely on feedback from watch time. Existing approaches simply treat watch time as a direct label, failing to effectively harness its extensive semantics and introduce bias, thereby limiting the potential for modeling user interests based on watch time. To overcome this challenge, we propose a framework named Debiased Multiple-semantics-extracting Labeling (DML). DML constructs labels that encompass various semantics by utilizing quantiles derived from the distribution of watch time, prioritizing relative order rather than absolute label values. This approach facilitates easier model learning while aligning with the ranking objective of recommendations. Furthermore, we introduce a method inspired by causal adjustment to refine label definitions, thereby directly mitigating bias at the label level. We substantiate the effectiveness of our DML framework through both online and offline experiments. Extensive results demonstrate that our DML could effectively leverage watch time to discover users' real interests, enhancing their engagement in our application.

References

[1]

Aleksandr Aravkin, Aurelie Lozano, Ronny Luss, and Prabhajan Kambadur. 2014. Orthogonal Matching Pursuit for Sparse Quantile Regression. In Proceedings of the 2014 IEEE International Conference on Data Mining (ICDM '14). IEEE Computer Society, USA, 11--19. https://doi.org/10.1109/ICDM.2014.134

Digital Library

[2]

Stephen Bonner and Flavian Vasile. 2018. Causal Embeddings for Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (Vancouver, British Columbia, Canada) (RecSys '18). Association for Computing Machinery, New York, NY, USA, 104--112. https://doi.org/10.1145/3240323.3240360

Digital Library

[3]

Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. 2023 a. Reinforcing User Retention in a Billion Scale Short Video Recommender System. In Companion Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW '23 Companion). Association for Computing Machinery, New York, NY, USA, 421--426. https://doi.org/10.1145/3543873.3584640

Digital Library

[4]

Qingpeng Cai, Zhenghai Xue, Chi Zhang, Wanqi Xue, Shuchang Liu, Ruohan Zhan, Xueliang Wang, Tianyou Zuo, Wentao Xie, Dong Zheng, Peng Jiang, and Kun Gai. 2023 b. Two-Stage Constrained Actor-Critic for Short Video Recommendation. In Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW '23). Association for Computing Machinery, New York, NY, USA, 865--875. https://doi.org/10.1145/3543507.3583259

Digital Library

[5]

Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. 2021. AutoDebias: Learning to Debias for Recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 21--30. https://doi.org/10.1145/3404835.3462919

Digital Library

[6]

Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and Debias in Recommender System: A Survey and Future Directions. ACM Trans. Inf. Syst., Vol. 41, 3, Article 67 (feb 2023), 39 pages. https://doi.org/10.1145/3564284

Digital Library

[7]

Germán Cheuque, José Guzmán, and Denis Parra. 2019. Recommender Systems for Online Video Game Platforms: The Case of STEAM. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW '19). Association for Computing Machinery, New York, NY, USA, 763--771. https://doi.org/10.1145/3308560.3316457

Digital Library

[8]

Andrew Collins, Dominika Tkaczyk, Akiko Aizawa, and Joeran Beel. 2018. A Study of Position Bias in Digital Library Recommender Systems. arxiv: 1802.06565

[9]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys '16). Association for Computing Machinery, New York, NY, USA, 191--198. https://doi.org/10.1145/2959100.2959190

Digital Library

[10]

Arnaud de Myttenaere, Boris Golden, Bénédicte Le Grand, and Fabrice Rossi. 2016. Mean Absolute Percentage Error for regression models. Neurocomputing, Vol. 192 (2016), 38--48. https://doi.org/10.1016/j.neucom.2015.12.114 Advances in artificial neural networks, machine learning and computational intelligence.

[11]

Constantine E Frangakis and Donald B Rubin. 2002. Principal stratification in causal inference. Biometrics, Vol. 58, 1 (2002), 21--29. https://doi.org/10.1111/j.0006--341x.2002.00021.x

[12]

Xudong Gong, Qinlin Feng, Yuan Zhang, Jiangling Qin, Weijie Ding, Biao Li, Peng Jiang, and Kun Gai. 2022. Real-Time Short Video Recommendation on Mobile Devices. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM '22). Association for Computing Machinery, New York, NY, USA, 3103--3112. https://doi.org/10.1145/3511808.3557065

Digital Library

[13]

Ming He, Xin Chen, Xinlei Hu, and Changshu Li. 2022. Causal Intervention for Sentiment De-Biasing in Recommendation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM '22). Association for Computing Machinery, New York, NY, USA, 4014--4018. https://doi.org/10.1145/3511808.3557558

Digital Library

[14]

Xiangnan He, Yang Zhang, Fuli Feng, Chonggang Song, Lingling Yi, Guohui Ling, and Yongdong Zhang. 2023. Addressing Confounding Feature Issue for Causal Recommendation. ACM Trans. Inf. Syst., Vol. 41, 3, Article 53 (feb 2023), 23 pages. https://doi.org/10.1145/3559757

Digital Library

[15]

José Miguel Hernández-Lobato, Neil Houlsby, and Zoubin Ghahramani. 2014. Probabilistic Matrix Factorization with Non-Random Missing Data. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32 (Beijing, China) (ICML'14). JMLR.org, II--1512--II--1520. https://doi.org/10.5555/3044805.3045061

Digital Library

[16]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately Interpreting Clickthrough Data as Implicit Feedback. In Acm Sigir Forum (Salvador, Brazil) (SIGIR '05). Association for Computing Machinery, New York, NY, USA, 154--161. https://doi.org/10.1145/1076034.1076063

Digital Library

[17]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007. Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search. ACM Trans. Inf. Syst., Vol. 25, 2 (apr 2007), 7--es. https://doi.org/10.1145/1229179.1229181

Digital Library

[18]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations. http://arxiv.org/abs/1412.6980

[19]

Dingcheng Li, Xu Li, Jun Wang, and Ping Li. 2020. Video Recommendation with Multi-Gate Mixture of Experts Soft Actor Critic. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, New York, NY, USA, 1553--1556. https://doi.org/10.1145/3397271.3401238

Digital Library

[20]

Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. 2023 b. Balancing Unobserved Confounding with a Few Unbiased Ratings in Debiased Recommendations. In Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW '23). Association for Computing Machinery, New York, NY, USA, 1305--1313. https://doi.org/10.1145/3543507.3583495

Digital Library

[21]

Kunpeng Li, Guangcui Shao, Naijun Yang, Xiao Fang, and Yang Song. 2022. Billion-User Customer Lifetime Value Prediction: An Industrial-Scale Solution from Kuaishou. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM '22). Association for Computing Machinery, New York, NY, USA, 3243--3251. https://doi.org/10.1145/3511808.3557152

Digital Library

[22]

Ling Li and Hsuan-Tien Lin. 2006. Ordinal Regression by Extended Binary Classification. In Proceedings of the 19th International Conference on Neural Information Processing Systems (Canada) (NIPS'06). MIT Press, Cambridge, MA, USA, 865--872. https://doi.org/10.5555/2976456.2976565

Digital Library

[23]

Qian Li, Xiangmeng Wang, Zhichao Wang, and Guandong Xu. 2023 a. Be Causal: De-Biasing Social Network Confounding in Recommendation. ACM Trans. Knowl. Discov. Data, Vol. 17, 1, Article 14 (feb 2023), 23 pages. https://doi.org/10.1145/3533725

Digital Library

[24]

Yang Liu, Cheng Lyu, Zhiyuan Liu, and Dacheng Tao. 2019. Building Effective Short Video Recommendation. In 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). 651--656. https://doi.org/10.1109/ICMEW.2019.00126

[25]

Agnes Lydia and Sagayaraj Francis. 2019. Adagrad-an optimizer for stochastic gradient descent. Int. J. Inf. Comput. Sci, Vol. 6, 5 (2019), 566--568.

[26]

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. Modeling Task Relationships in Multi-Task Learning with Multi-Gate Mixture-of-Experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD '18). Association for Computing Machinery, New York, NY, USA, 1930--1939. https://doi.org/10.1145/3219819.3220007

Digital Library

[27]

Benjamin M. Marlin, Richard S. Zemel, Sam Roweis, and Malcolm Slaney. 2007. Collaborative Filtering and the Missing at Random Assumption. In Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence (Vancouver, BC, Canada) (UAI'07). AUAI Press, Arlington, Virginia, USA, 267--275. https://doi.org/10.5555/3020488.3020521

Digital Library

[28]

Judea Pearl. 2009. Causality. Cambridge university press.

[29]

John T Roscoe. 1975. Fundamental research statistics for the behavioral sciences [by] John T. Roscoe.

[30]

Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. 2020. Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining (Houston, TX, USA) (WSDM '20). Association for Computing Machinery, New York, NY, USA, 501--509. https://doi.org/10.1145/3336191.3371783

Digital Library

[31]

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (New York, NY, USA) (ICML'16). JMLR.org, 1670--1679. https://doi.org/10.5555/3045390.3045567

Digital Library

[32]

Harald Steck. 2013. Evaluation of Recommendations: Rating-Prediction and Ranking. In Proceedings of the 7th ACM Conference on Recommender Systems (Hong Kong, China) (RecSys '13). Association for Computing Machinery, New York, NY, USA, 213--220. https://doi.org/10.1145/2507157.2507160

Digital Library

[33]

Linpeng Tang, Qi Huang, Amit Puntambekar, Ymir Vigfusson, Wyatt Lloyd, and Kai Li. 2017. Popularity Prediction of Facebook Videos for Higher Quality Streaming. In Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC '17). USENIX Association, USA, 111--123.

[34]

Wenjie Wang, Fuli Feng, Xiangnan He, Xiang Wang, and Tat-Seng Chua. 2021a. Deconfounded Recommendation for Alleviating Bias Amplification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD '21). Association for Computing Machinery, New York, NY, USA, 1717--1725. https://doi.org/10.1145/3447548.3467249

Digital Library

[35]

Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021b. Clicks Can Be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 1288--1297. https://doi.org/10.1145/3404835.3462962

Digital Library

[36]

Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2019b. Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 6638--6647. https://proceedings.mlr.press/v97/wang19n.html

[37]

Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2021c. Combating Selection Biases in Recommender Systems with a Few Unbiased Ratings. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (Virtual Event, Israel) (WSDM '21). Association for Computing Machinery, New York, NY, USA, 427--435. https://doi.org/10.1145/3437963.3441799

Digital Library

[38]

Yixin Wang, Dawen Liang, Laurent Charlin, and David M. Blei. 2020. Causal Inference for Recommender Systems. In Proceedings of the 14th ACM Conference on Recommender Systems (Virtual Event, Brazil) (RecSys '20). Association for Computing Machinery, New York, NY, USA, 426--431. https://doi.org/10.1145/3383313.3412225

Digital Library

[39]

Yu-Huan Wang, Tian-Jun Gu, and Shyang-Yuh Wang. 2019a. Causes and Characteristics of Short Video Platform Internet Community Taking the TikTok Short Video Application as an Example. In 2019 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW). IEEE, 1--2. https://doi.org/10.1109/ICCE-TW46550.2019.8992021

[40]

Zhenlei Wang, Shiqi Shen, Zhipeng Wang, Bo Chen, Xu Chen, and Ji-Rong Wen. 2022. Unbiased Sequential Recommendation with Latent Confounders. In Proceedings of the ACM Web Conference 2022 (Virtual Event, Lyon, France) (WWW '22). Association for Computing Machinery, New York, NY, USA, 2195--2204. https://doi.org/10.1145/3485447.3512092

Digital Library

[41]

Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. 2021. Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD '21). Association for Computing Machinery, New York, NY, USA, 1791--1800. https://doi.org/10.1145/3447548.3467289

Digital Library

[42]

Siqi Wu, Marian-Andrei Rizoiu, and Lexing Xie. 2018. Beyond Views: Measuring and Predicting Engagement in Online Videos. In Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25--28, 2018. AAAI Press, 434--443. https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17892

[43]

Mengyue Yang, Quanyu Dai, Zhenhua Dong, Xu Chen, Xiuqiang He, and Jun Wang. 2021. Top-N Recommendation with Counterfactual User Preference Simulation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (Virtual Event, Queensland, Australia) (CIKM '21). Association for Computing Machinery, New York, NY, USA, 2342--2351. https://doi.org/10.1145/3459637.3482305

Digital Library

[44]

Yuxin Ying, Fuzhen Zhuang, Yongchun Zhu, Deqing Wang, and Hongwei Zheng. 2023. CAMUS: Attribute-Aware Counterfactual Augmentation for Minority Users in Recommendation. In Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW '23). Association for Computing Machinery, New York, NY, USA, 1396--1404. https://doi.org/10.1145/3543507.3583538

Digital Library

[45]

Ruohan Zhan, Changhua Pei, Qiang Su, Jianfeng Wen, Xueliang Wang, Guanyu Mu, Dong Zheng, Peng Jiang, and Kun Gai. 2022. Deconfounding Duration Bias in Watch-Time Prediction for Video Recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Washington DC, USA) (KDD '22). Association for Computing Machinery, New York, NY, USA, 4472--4481. https://doi.org/10.1145/3534678.3539092

Digital Library

[46]

Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal Intervention for Leveraging Popularity Bias in Recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 11--20. https://doi.org/10.1145/3404835.3462875

Digital Library

[47]

Yu Zheng, Chen Gao, Jingtao Ding, Lingling Yi, Depeng Jin, Yong Li, and Meng Wang. 2022. DVR: Micro-Video Recommendation Optimizing Watch-Time-Gain under Duration Bias. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM '22). Association for Computing Machinery, New York, NY, USA, 334--345. https://doi.org/10.1145/3503161.3548428

Digital Library

Cited By

Bai YZhang YFeng FLu JZang XLei CSong YBaeza-Yates RBonchi F(2024)GradCraft: Elevating Multi-task Recommendations through Holistic Gradient CraftingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671585(4774-4783)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671585
Bai YZhang YLu JChang JZang XNiu YSong YFeng FAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)LabelCraft: Empowering Short Video Recommendations with Automated Label CraftingProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635816(28-37)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635816

Index Terms

Leveraging Watch-time Feedback for Short-Video Recommendations: A Causal Labeling Framework
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Towards Trustworthy Recommender System: A Faithful and Responsible Recommendation Perspective
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
Recommender systems (RecSys) become increasingly prevalent in modern society, offering personalized information filtering to alleviate information overload and significantly impacting various human online activities. Machine learning-based recommendation ...
Causal Intervention for Leveraging Popularity Bias in Recommendation
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Recommender system usually faces popularity bias issues: from the data perspective, items exhibit uneven (usually long-tail) distribution on the interaction frequency; from the method perspective, collaborative filtering methods are prone to amplify the ...
Hybrid group recommendations for a travel service

Recommendation techniques have proven their usefulness as a tool to cope with the information overload problem in many classical domains such as movies, books, and music. Additional challenges for recommender systems emerge in the domain of tourism such ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

October 2023

5508 pages

ISBN:9798400701245

DOI:10.1145/3583780

General Chairs:
Ingo Frommholz
University of Wolverhampton, UK
,
Frank Hopfgartner
University of Koblenz, Germany
,
Mark Lee
University of Birmingham, UK
,
Michael Oakes
University of Birmingham, UK
,
Program Chairs:
Mounia Lalmas
Spotify, UK
,
Min Zhang
Tsinghua University, China
,
Rodrygo Santos
Federal University of Minas Gerais, Brazil

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

CIKM '23

Sponsor:

CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management

October 21 - 25, 2023

Birmingham, United Kingdom

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
241
Total Downloads

Downloads (Last 12 months)241
Downloads (Last 6 weeks)14

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bai YZhang YFeng FLu JZang XLei CSong YBaeza-Yates RBonchi F(2024)GradCraft: Elevating Multi-task Recommendations through Holistic Gradient CraftingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671585(4774-4783)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671585
Bai YZhang YLu JChang JZang XNiu YSong YFeng FAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)LabelCraft: Empowering Short Video Recommendations with Automated Label CraftingProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635816(28-37)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635816

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents