research-article

HD3: Distributed Dueling DQN with Discrete-Continuous Hybrid Action Spaces for Live Video Streaming

Authors:

Xiaolan Jiang,

Yusheng JiAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 2632 - 2636

https://doi.org/10.1145/3343031.3356052

Published: 15 October 2019 Publication History

Get Access

Abstract

Live streaming applications are becoming increasingly popular recently, and it exposes new technical challenges compared to regular video streaming. High video quality and low latency are two main requirements in live streaming scenarios. A live streaming application needs to make bitrate and target buffer level decisions as well as sets a continuous latency limit value to skip video frames. We formulate the live streaming task as a reinforcement learning problem with discrete-continuous hybrid action spaces, then propose a novel deep reinforcement learning (DRL) algorithm HD3 which can take hybrid actions to solve it. We compare HD3 with several state-of-the-art DRL algorithms on various network environments, and the simulation results show that HD3 can outperform all the other comparison schemes. We emphasize that HD3 generates a single agent which can perform well on different network conditions and video scenes.

References

[1]

Chang Ge, Ning Wang, Wei Koong Chai, and Hermann Hellwagner. 2018. QoE-assured 4K HTTP live streaming via transient segment holding at mobile edge. IEEE Journal on Selected Areas in Communications, Vol. 36, 8 (2018), 1816--1830.

Crossref

Google Scholar

[2]

Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. 2018. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933 (2018).

Google Scholar

[3]

Tianchi Huang, Rui-Xiao Zhang, Chao Zhou, and Lifeng Sun. 2018. Qarc: Video quality aware rate control for real-time video streaming based on deep reinforcement learning. arXiv preprint arXiv:1805.02482 (2018).

Google Scholar

[4]

Xiaolan Jiang, Yi-Han Chiang, Yang Zhao, and Yusheng Ji. 2018. Plato: Learning-based Adaptive Streaming of 360-Degree Videos. In 2018 IEEE 43rd Conference on Local Computer Networks (LCN). IEEE, 393--400.

Crossref

Google Scholar

[5]

Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. ACM, 197--210.

Digital Library

Google Scholar

[6]

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. 1928--1937.

Digital Library

Google Scholar

[7]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).

Google Scholar

[8]

Ericsson Mobility Report. 2019. https://www.ericsson.com/en/mobility-report/.

Google Scholar

[9]

ACM Multimedia 2019 Grand Challenge-(Live Video Streaming). 2019. https://www.aitrans.online/MMGC/.

Google Scholar

[10]

Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In Thirtieth AAAI Conference on Artificial Intelligence .

Digital Library

Google Scholar

[11]

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. 2015. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581 (2015).

Google Scholar

[12]

Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. 2018. Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space. arXiv preprint arXiv:1810.06394 (2018).

Google Scholar

Cited By

View all

Li ZBai HXue SJin K(2025)Intelligent Defense Decision of Aircraft Based on Rainbow AlgorithmIntelligent Robotics and Applications10.1007/978-981-96-0780-8_4(43-56)Online publication date: 21-Jan-2025
https://doi.org/10.1007/978-981-96-0780-8_4
Xu JAzad NLin Y(2024)Mixed‐Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy ManagementOptimal Control Applications and Methods10.1002/oca.3216Online publication date: 15-Oct-2024
https://doi.org/10.1002/oca.3216
Li JWang HLiu ZZhou PChen XLi QHong R(2023)Toward Optimal Real-Time Volumetric Video Streaming: A Rolling Optimization and Deep Reinforcement Learning Based ApproachIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.327789333:12(7870-7883)Online publication date: Dec-2023
https://doi.org/10.1109/TCSVT.2023.3277893
Show More Cited By

Index Terms

HD3: Distributed Dueling DQN with Discrete-Continuous Hybrid Action Spaces for Live Video Streaming
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
2. Networks
  1. Network protocols
    1. Application layer protocols

Recommendations

Continuous Bitrate & Latency Control with Deep Reinforcement Learning for Live Video Streaming
MM '19: Proceedings of the 27th ACM International Conference on Multimedia

In this paper, we introduce a continuous bitrate control and latency control model for the Live Video Streaming Challenge. Our model is based on Deep Deterministic Policy Gradient, popular on continuous control tasks. Simultaneously, it can take a fine-...
Multi-camera Live Video Streaming over Wireless Network
Advances in Mobile Computing and Multimedia Intelligence
Abstract
Due to the development of wireless communication technology, more and more streamers are using cameras mounted on mobile devices for live streaming in a wireless LAN environment. Conventional live streaming systems, which employ multiple images ...
An HTTP/2 Push-Based Approach for Low-Latency Live Streaming with Super-Short Segments

Over the last years, streaming of multimedia content has become more prominent than ever. To meet increasing user requirements, the concept of HTTP Adaptive Streaming (HAS) has recently been introduced. In HAS, video content is temporally divided into ...

Comments

Information & Contributors

Information

Published In

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
525
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Li ZBai HXue SJin K(2025)Intelligent Defense Decision of Aircraft Based on Rainbow AlgorithmIntelligent Robotics and Applications10.1007/978-981-96-0780-8_4(43-56)Online publication date: 21-Jan-2025
https://doi.org/10.1007/978-981-96-0780-8_4
Xu JAzad NLin Y(2024)Mixed‐Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Electric Vehicle Energy ManagementOptimal Control Applications and Methods10.1002/oca.3216Online publication date: 15-Oct-2024
https://doi.org/10.1002/oca.3216
Li JWang HLiu ZZhou PChen XLi QHong R(2023)Toward Optimal Real-Time Volumetric Video Streaming: A Rolling Optimization and Deep Reinforcement Learning Based ApproachIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.327789333:12(7870-7883)Online publication date: Dec-2023
https://doi.org/10.1109/TCSVT.2023.3277893
Xu YYin JYang QYang L(2022)Media Production Using Cloud and Edge Computing: Recent Progress and NBMP-Based ImplementationIEEE Transactions on Broadcasting10.1109/TBC.2022.314070468:2(545-558)Online publication date: Jun-2022
https://doi.org/10.1109/TBC.2022.3140704
Naresh MGireesh NSaxena PGupta M(2022)SAC-ABR: Soft Actor-Critic based deep reinforcement learning for Adaptive BitRate streaming2022 14th International Conference on COMmunication Systems & NETworkS (COMSNETS)10.1109/COMSNETS53615.2022.9668424(353-361)Online publication date: 4-Jan-2022
https://doi.org/10.1109/COMSNETS53615.2022.9668424
Hu KChen XXia QJin JWeng L(2021)A Control Algorithm for Sea–Air Cooperative Observation Tasks Based on a Data-Driven AlgorithmJournal of Marine Science and Engineering10.3390/jmse91111899:11(1189)Online publication date: 27-Oct-2021
https://doi.org/10.3390/jmse9111189
Sun LZong TWang SLiu YWang YAlay ÖHsu CBegen A(2021)Tightrope walking in low-latency live streamingProceedings of the 12th ACM Multimedia Systems Conference10.1145/3458305.3463382(200-213)Online publication date: 24-Jun-2021
https://dl.acm.org/doi/10.1145/3458305.3463382

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Continuous Bitrate & Latency Control with Deep Reinforcement Learning for Live Video Streaming

Multi-camera Live Video Streaming over Wireless Network

An HTTP/2 Push-Based Approach for Low-Latency Live Streaming with Super-Short Segments

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations