Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

HTTP Adaptive Streaming – Quo Vadis (2024)

•
•

Video traffic on the Internet is constantly growing; networked multimedia applications consume a predominant share of the available Internet bandwidth. A major technical breakthrough and enabler in multimedia systems research and of industrial networked multimedia services certainly was the HTTP Adaptive Streaming (HAS) technique. This resulted in the standardization of MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) which, together with HTTP Live Streaming (HLS), is widely used for multimedia delivery in today’s networks. Existing challenges in multimedia systems research deal with the trade-off between (i) the ever-increasing content complexity, (ii) various requirements with respect to time (most importantly, latency), and (iii) quality of experience (QoE). Optimizing towards one aspect usually negatively impacts at least one of the other two aspects if not both. This situation sets the stage for our research work in the ATHENA Christian Doppler (CD) Laboratory (Adaptive Streaming over HTTP and Emerging Networked Multimedia Services; https://athena.itec.aau.at/), jointly funded by public sources and industry. In this talk, we will present selected novel approaches and research results of the first year of the ATHENA CD Lab’s operation. We will highlight HAS-related research on (i) multimedia content provisioning (machine learning for video encoding); (ii) multimedia content delivery (support of edge processing and virtualized network functions for video networking); (iii) multimedia content consumption and end-to-end aspects (player-triggered segment retransmissions to improve video playout quality); and (iv) novel QoE investigations (adaptive point cloud streaming). We will also put the work into the context of international multimedia systems research.

1 of 46
Download to read offline
HTTP Adaptive Streaming – Quo Vadis? Christian Timmerer, Professor at AAU, Director at CD Lab ATHENA Hermann Hellwagner, Professor at AAU Klagenfurt, Austria June 27, 2024 1 Change log: ● DSRC Telecom Seminar Series’24 ● FEEC/UNICAMP Seminar in Comp. Eng.’23 ● IEEE MMTC DL Series’23 ● PCS’21 ● DDRC’21 ● ISM’20 ● WebMedia’20 https://www.slideshare.net/christian.timmerer
The number of transistors in an integrated circuit doubles about every two years – Moore's Law (https://en.wikipedia.org/wiki/Moore%27s_law) Users' bandwidth grows by 50% per year (10% less than Moore's Law for computer speed) – Nielsen's Law of Internet Bandwidth (https://www.nngroup.com/articles/law-of-bandwidth/) Video accounts for more than 65% of the global Internet traffic – Sandvine Global Internet Phenomena (January 2023, https://www.sandvine.com/phenomena) 2
Presenter Christian Timmerer Univ.-Prof. at Alpen-Adria-Universität Klagenfurt Director CD Lab ATHENA CIO | Head of Research and Standardization at Bitmovin 3 3 2003: MSc CS (Dipl.-Ing.) 2006: PhD CS (Dr.-techn.) 2012: Co-founded Bitmovin 2014: Habilitation (Priv.-Doz.) & Assoc. Prof. 2016: Dep. Director @ ITEC/AAU 2019: Director @ ATHENA 2022: Univ.-Prof. for Multimedia Systems Web: http://timmerer.com/ Bitmovin MPEG-DASH
4 4 My offices are here Copyright: AAU/Steinthaler
● Introduction ● ATHENA ○ Content Provisioning ○ Content Delivery ○ Content Consumption ○ End-to-End Aspects ○ Quality of Experience ● Conclusions: HAS – Quo Vadis? Agenda 5
Introduction / Motivation Sources: * Sandvine Global Internet Phenomena (January 2024). ** Cisco Annual Internet Report (2018–2023) White Paper (March 2020) 6 6 Video streaming is dominating today’s Internet traffic ● 2024*: 68% (fixed) and 64% (mobile) ● Video on-demand: 54% / 57% – Live: 14% / 7% ● Main applications: YouTube, Netflix (>10%), Tik Tok, Amazon Prime, Disney+ (<10%) ● Video and other applications continue to be of enormous demand in today’s home, but there will be significant bandwidth demands with the application requirements of the future**
HTTP Adaptive Streaming 101 Adaptation logic is within the client, not normatively specified by a standard, subject to research and development 7 7 Client
Multimedia Systems Challenges and Tradeoffs 8 8 Basic figure by Klara Nahrstedt, University of Illinois at Urbana–Champaign, IEEE MIPR 2018
“Application-oriented basic research” to address current and future research and deployment challenges of HAS and emerging streaming methods ATHENA – Adaptive Streaming over HTTP and Emerging Networked Multimedia Services Content Provisioning Content Delivery Content Consumption End-to-End Aspects ● Video encoding for HAS ● Quality-aware encoding ● Learning-based encoding ● Multi-codec HAS ● Edge computing ● Information CDN/SDN⇿clients ● Netw. assistance for/by clients ● Utility evaluation ● Bitrate adaptation schemes ● Playback improvements ● Context and user awareness ● Quality of Experience (QoE) studies ● Application/transport layer enhancements ● Quality of Experience (QoE) models ● Low-latency HAS ● Learning-based HAS https://athena.itec.aau.at/ 9 9 Funding:
Video coding for HTTP Adaptive Streaming ● Quality improvement Per-Title Encoding (PTE) et al. for live use cases: video complexity analysis, per-title/-scene/-shot/-segment, content-/context-aware, content-adaptive, quality-aware encoding ● Runtime improvement Hardware-/software-based (cloud), parallel/distributed, information reuse from reference encodings (multi-rate/-resolution) ⇨ cf. earlier versions of this talk ● Application scenarios Video on Demand (VoD incl. diff. flavors AVoD, SVoD), live (incl. diff. flavors), interactive, games, video conferencing Content Provisioning 10 10
Problem Statement / Basic Idea for Live Streaming ● Static bitrate ladder Pre-defined bitrate/resolution pairs for all contents ● Per-title encoding Optimize bitrate ladder based on the content ● UHD / HFR content Increase of spatial and temporal resolutions ● Live (is Life)[* Opus, 1985] Perceptual-based bitrate/resolution/framerate prediction algorithm based on content complexity Video Complexity Analyzer (VCA) 11 11 VoD Live * … https://www.youtube.com/watch?v=pATX-lV0VFk Vignesh V Menon, Christian Feldmann, Hadi Amirpour, Mohammad Ghanbari, and Christian Timmerer. 2022. VCA: video complexity analyzer. In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). Association for Computing Machinery, New York, NY, USA, 259–264. https://athena.itec.aau.at/2022/04/vca-video-complexity-analyzer/
Spatial (E) / Temporal (h) Content Characteristics Video Complexity Analyzer (VCA) 12 12 *) Michael King, Zinovi Tauber, and Ze-Nian Li. 2007. A New Energy Function for Segmentation and Compression. In 2007 IEEE International Conference on Multimedia and Expo. 1647–1650. https://doi.org/10.1109/ICME.2007.4284983 *) VCA 2.0 (seven features): the average ● luma texture energy EY ● gradient of the luma texture energy hY ● luma brightness LY ● chroma brightness (LU and LV ) ● chroma texture energy (EU and EV ) https://athena.itec.aau.at/2023/02/vca-v2-0-released/
VCA 2.0 Latest Results 13 13 Vignesh V Menon, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, Christian Timmerer, "Green video complexity analysis for efficient encoding in Adaptive Video Streaming," ACM Green Multimedia Systems 2023, June 2023. https://athena.itec.aau.at/2023/04/green-video-complexity-analysis-for-efficient-encoding-in-adaptive-video-streaming/
Live Per-Title Encoding Scheme 14 14 VCA: Video Complexity Analyzer (to extract/calculate E/h metrics) Github: https://github.com/cd-athena/VCA Documentation: https://cd-athena.github.io/VCA/
Live Per-Title Encoding: Example Bitrate Ladder 15 15 VCA: Video Complexity Analyzer (to extract/calculate E/h metrics) Github: https://github.com/cd-athena/VCA Documentation: https://cd-athena.github.io/VCA/
Live Per-Title Encoding: Example Frames (1) 16 16
Live Per-Title Encoding: Example Frames (2) 17 17
Results: Online Per-Title Encoding (OPTE) 18 18 V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, "OPTE: Online Per-Title Encoding for Live Video Streaming," ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022, pp. 1865-1869, doi: 10.1109/ICASSP43922.2022.9746745. https://athena.itec.aau.at/2022/01/opte-online-per-title-encoding-for-live-video-streaming/
Perceptually-aware Per-title Encoding (PPTE) 19 19 The minimum visual difference that can be perceived by HVS, i.e., the difference between two adjacent perceptual distortion levels, refers as to one Just Noticeable Difference (JND) V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, "Perceptually-Aware Per-Title Encoding for Adaptive Video Streaming," 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan, 2022, pp. 1-6, doi: 10.1109/ICME52920.2022.9859744. https://athena.itec.aau.at/2022/05/perceptually-aware-per-title-encoding-for-adaptive-video-streaming/
Results: Perceptually-aware Per-title Encoding (PPTE) 20 20 V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, "Perceptually-Aware Per-Title Encoding for Adaptive Video Streaming," 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan, 2022, pp. 1-6, doi: 10.1109/ICME52920.2022.9859744. https://athena.itec.aau.at/2022/05/perceptually-aware-per-title-encoding-for-adaptive-video-streaming/
JND-Aware Two-Pass Per-Title Encoding Scheme for Adaptive Live Streaming (JTPS) 21 21 V. V. Menon, P. T. Rajendran, C. Feldmann, K. Schoeffmann, M. Ghanbari and C. Timmerer, "JND-Aware Two-Pass Per-Title Encoding Scheme for Adaptive Live Streaming," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 2, pp. 1281-1294, Feb. 2024, doi: 10.1109/TCSVT.2023.3290725. https://athena.itec.aau.at/2023/06/jnd-aware-two-pass-per-title-encoding-scheme-for-adaptive-live-streaming/
● Efficient Content-Adaptive Feature-based Shot Detection for HTTP Adaptive Streaming (IEEE ICIP 2021) ● INCEPT: INTRA CU Depth Prediction for HEVC (IEEE MMSP 2021) ● CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming (DCC 2022) ● OPTE: Online Per-title Encoding for Live Video Streaming (IEEE ICASSP 2022) ● Live-PSTR: Live Per-title Encoding for Ultra HD Adaptive Streaming (NAB BEITC 2022) ● Perceptually-aware Per-title Encoding for Adaptive Video Streaming (IEEE ICME 2023) ● Light-weight Video Encoding Complexity Prediction using Spatio Temporal Features (IEEE MMSP 2023) ● ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming (IEEE ICIP 2023) ● Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming (PCS 2023) ● Transcoding Quality Prediction for Adaptive Video Streaming (ACM MHV 2023) ● Green video complexity analysis for efficient encoding in Adaptive Video Streaming (ACM GMSys 2023) ● JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming (IEEE TCVST 2023) ● Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming (VCIP 2023) VCA-based Applications (selection) 22 22
Network assistance for HTTP Adaptive Streaming ● Edge computing support (at CDN / cellular network edge) Functions at (or, assisted by) the edge: adaptation, analytics, (pre-)fetching, caching, transcoding, repackaging of content, request aggregation ● Server/network/CDN ↔ HAS client information exchange and collaboration IETF ALTO, MPEG SAND, MPEG NBMP, …; SDN-DASH, SDN-HAS, SABR, ... ● Use of modern network architecture features SW Defined Networking (SDN); Network Function Virtual. (NFV); MC-ABR Content Delivery 23 23
Different policies/metrics/resource utilization ● Prefetching based on last segment quality ○ Last segment quality (LSQ) ○ Last segment quality plus (LSQ+) ○ All segment qualities (ASQ) ● Prefetching based on a Markov Model ● Prefetching based on transrating ● Prefetching Based on machine Learning (buffer size, link bitrate, prev. quality, prev. link bitrate; Random Forest, Gradient Boost, AdaBoost, Decision Trees and Extremely Randomized Trees) ● Prefetching based on super resolution Segment Prefetching and Caching at the Edge (SPACE) 24 24 J. Aguilar-Armijo, C. Timmerer and H. Hellwagner, "SPACE: Segment Prefetching and Caching at the Edge for Adaptive Video Streaming," in IEEE Access, vol. 11, pp. 21783-21798, 2023 https://athena.itec.aau.at/2023/02/space-segment-prefetching-and-caching-at-the-edge-for-adaptive-video-streaming/
The best segment prefetching policy depends on the service provider’s preferences and resources ● Straightforward implementation with low resource utilization ⇨ LSQ and Markov-based ● Prioritize QoE at the expense of costs ⇨ ASQ, LSQ+ and Transrating-based ● Balance between performance and costs ⇨ ML-based ● Possible next steps: dynamically adapt; premium clients Segment Prefetching and Caching at the Edge (SPACE) 25 25
Approach: ● Mechanism: Introduce a new server/segment selection approach at the edge of the network ● Main goal: Improve the users' QoE and network utilization ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming 26 26 R. Farahani, F. Tashtarian, A. Erfanian, C. Timmerer, M. Ghanbari, and H. Hellwagner. ”ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming”. The 31st edition of the Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV’21), Sept. 28-Oct. 1, 2021, Istanbul, Turkey. https://athena.itec.aau.at/2021/04/es-has-an-edge-and-sdn-assisted-framework-for-http-adaptive-video-streaming/
Goal ● Minimizing HAS clients’ serving time & network cost, considering available resources ● Multi-layer architecture + centralized optimization model executed by SDN controller ⇨ high time complexity ● Three heuristic approaches: CG, FG-I, FG-II ● Experiments on a large-scale cloud-based testbed including 250 HAS players ⇨ improve QoE by at least 47% ⇨ decrease the streaming cost by at least 47% ⇨ enhance network utilization by at least 48% compared to state-of-the-art ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming 27 27 R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari and H. Hellwagner, "ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming," in IEEE Transactions on Network and Service Management, vol. 20, no. 1, pp. 625-643, March 2023, doi: 10.1109/TNSM.2022.3210595. https://athena.itec.aau.at/2022/09/ararat/
Extract some features as metadata during the encoding process ⇨ Reuse metadata in the transcoding process at the edge Light-weight Transcoding at the Edge (LwTE) 28 28 A. Erfanian, H. Amirpour, F. Tashtarian, C. Timmerer and H. Hellwagner, LwTE: Light-Weight Transcoding at the Edge," in IEEE Access, vol. 9, pp. 112276-112289, 2021 https://athena.itec.aau.at/2021/07/lwte-light-weight-transcoding-at-the-edge/ Delivery 3 4 Extract metadata 1 2 Determine optimal policy 3 Optimized download/ transcode 3 Variations: ● CD-LwTE ● LwTE-Live
LwTE: Binary Linear Programming (BLP) Model 29 29 Inputs & Constraints: ●Videos/Segments Size ●Metadata Size ●Resources Cost ●Available Resources ●Probability Function ●Number of Incoming Requests BLP Optimization Model Outputs: ● Segments’ Serving Policy (store/transcode/fetch) Objective function: Minimize cost (computation, storage, bandwidth) and serving delay
Performance of the proposed CD-LwTE approaches (FGH, CGH) compared with state-of-the-art approaches in terms of (a) cost, and (b) average serving delay, for various ρ values (the number of incoming requests at the edge server) CD-LwTE Comparison with State of the Art 30 30 APAC: T. X. Tran, P. Pandey, A. Hajisami, and D. Pompili, “Collaborative multibitrate video caching and processing in Mobile-Edge Computing networks,” in 2017 13th Annual Conference on Wireless On-demand Network Systems and Services (WONS), 2017, pp. 165–172. CoCache: T. X. Tran and D. Pompili, “Adaptive Bitrate Video Caching and Processing in Mobile-Edge Computing Networks,” IEEE Transactions on Mobile Computing, vol. 18, no. 9, pp. 1965–1978, 2019. PartialCache: H. Zhao, Q. Zheng, W. Zhang, B. Du, and H. Li, “A Segment-based Storage and Transcoding Trade-off Strategy for Multi-version VoD Systems in the Cloud,” IEEE Transactions on Multimedia, vol. 19, no. 1, pp. 149–159, 2016. A. Erfanian, H. Amirpour, F. Tashtarian, C. Timmerer and H. Hellwagner, "CD-LwTE: Cost-and Delay-aware Light-weight Transcoding at the Edge," in IEEE Transactions on Network and Service Management, doi: 10.1109/TNSM.2022.3229744. https://athena.itec.aau.at/2022/12/cd-lwte-cost-and-delay-aware-light-weight-transcoding-at-the-edge/ CD-LwTE CD-LwTE
LwTE Findings 31 31 ● Stores the optimal search decisions in the encoding process as metadata ● Utilizes the metadata to avoid search processes during transcoding at the edge ● Uses partial-transcoding ● LwTE does transcoding 80% faster than H.265 ● Up to 70% cost saving compared to state-of-the-art LwTE ● Extends LwTE by relaxing assumptions, new policy, and serving delay to objective ● Adds new features in metadata ● BLP model to select optimal policy to serve requests while minimizing cost and delay ● Reduces transcoding time up to 97% streaming cost up to 75% delay up to 48% compared to state-of-the-art CD-LwTE ● Investigates LwTE’s performance in live streaming context ● MBLP model to select optimal policy (fetching and transcoding) to serve requests ● Reduces streaming cost up to 34% bandwidth up to 45% compared to state-of-the-art LwTE-Live
Player Adaptation Logic and Quality of Experience ● Bitrate adaptation schemes Client-based, server-based, network-assisted, hybrid, ML-based ● Application/transport layer enhancements HTTP/2 (TCP) and HTTP/3 (QUIC), Media over QUIC (MOQ), proprietary formats (SRT, RIST, …), WebRTC, low-latency/delay ● Client playback improvements User-/client-aware playback, content-enhancement filters, super-resolution ● Low-latency live streaming Use of MPEG CMAF, HTTP/1.1 Chunked Transfer Encoding (CTE), other protocol features (e.g., HTTP/2 Push); LL-DASH/HLS; specific network functions; CDN support ● Quality of Experience Content Consumption and End-to-End Aspects 32 32
Bitrate Adaptation Schemes Bitrate Adaptation Schemes Client-based Adaptation Bandwidth -based Buffer- based Mixed adaptation Proprietary solutions MDP-based Server-based Adaptation Network- assisted Adaptation Hybrid Adaptation SDN-based Server and network- assisted A. Bentaleb, B. Taani, A.C. Begen, C. Timmerer and R. Zimmermann, "A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP," IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 562-585, First Quarter 2019. 33 33 Adaptive bitrate (ABR) algorithms, adaptation logics, ...
E-WISH: A policy driven ABR Algorithm 34 Energy Cost + 𝛅 [Data Cost] Data Cost Total Cost = ɑ Segment bitrate Throughput [Buffer Cost] Buffer Cost + β Download time Buffer occupancy [Quality Cost] Quality Cost + ɣ Video quality Distortion Instability Bitrate, Resolution, FPS (↑) M. Nguyen, E. Çetinkaya, H. Hellwagner and C. Timmerer, "WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices," 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), doi: 10.1109/MMSP53017.2021.9733605 Technology Transfer
E-WISH: An Energy-aware ABR Algorithm 35 [Data Cost] [Quality Cost] [Buffer Cost] [Energy Cost] Data Cost Buffer Cost Quality Cost Energy Cost Total Cost = ɑ + β + ɣ + 𝛅 Segment bitrate Throughput Download time Buffer occupancy Video quality Distortion Instability Bitrate, Resolution, FPS M. Nguyen, E. Çetinkaya, H. Hellwagner and C. Timmerer, "WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices," 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), doi: 10.1109/MMSP53017.2021.9733605 D.Lorenzi, M. Nguyen, F. Tashtarian, and C. Timmerer, “E-WISH: An Energy-aware ABR Algorithm For Green HTTP Adaptive Video Streaming”, 3rd ACM Mile-High Video Conference (MHV '24), https://doi.org/10.1145/3638036.3640802
E-WISH: An Energy-aware ABR Algorithm 36 D.Lorenzi, M. Nguyen, F. Tashtarian, and C. Timmerer, “E-WISH: An Energy-aware ABR Algorithm For Green HTTP Adaptive Video Streaming”, 3rd ACM Mile-High Video Conference (MHV '24), https://doi.org/10.1145/3638036.3640802
Based on ACM MMSys’22 keynote by Ali C. Begen LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation 37 37 B. Taraghi, H. Hellwagner and C. Timmerer, "LLL-CAdViSE: Live Low-Latency Cloud-Based Adaptive Video Streaming Evaluation Framework," in IEEE Access, vol. 11, pp. 25723-25734, 2023 https://athena.itec.aau.at/2023/03/lll-cadvise-live-low-latency-cloud-based-adaptive-video-streaming-evaluation-framework/ Average latency and ITU-T P.1203 MOS of MPEG-DASH (dash.js) and HLS (hls.js) using three different ABR algorithms (default, L2A-LL, LoLP) with a given target latency of 3s. https://github.com/cd-athena/LLL-CAdViSE
Live Video Streaming Pipeline 38 LIVE Origin Server (Encoding & Packaging) Live Camera Player CDN Server manifest.mpd Bitrate Resolution 5 Mbps 1920x1080 2.8 Mbps 1280x720 1.1 Mbps 960x540 System Admin Bitrate Ladder HTTP Request HTTP Response Manifest includes representations: different versions of the content optimized for various playback conditions, e.g., different bitrates and resolutions Tashtarian et al., "ARTEMIS: Adaptive Bitrate Ladder Optimization for Live Video Streaming", 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Santa Clara, CA, 2024 ● Fixed bitrate ladder ○ Content-aware, e.g., Netflix’s per-title encoding ○ Context-aware, e.g., [Lebreton and Yamagishi 2023] ○ Agnostic of the content and context (proprietary)
ARTEMIS: Adaptive Bitrate Ladder Optimization for Live Video Streaming Select representations with bitrates that closely match the desired bitrates Players Advertise a large number of representations Encoded segments CDN Server Mega-Manifest 39 Origin Server OTL Process requests and video qualities and determine OTL LIVE Live Camera Utilize the OTL OTL: Optimized Temporary Ladder Tashtarian et al., "ARTEMIS: Adaptive Bitrate Ladder Optimization for Live Video Streaming", 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Santa Clara, CA, 2024
QoE, Latency, and Stall with ARTEMIS vs. Five Static Bitrate Ladders 4 40 Average QoE up 11% Average Stall down 36% Average Latency down 18%
Quality of Experience (QoE) ... ● “... is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user’s personality and current state.”1) ● … can be easily extended to various domains, e.g., immersive media experiences.2) 41 41 1) P. Le Callet, S. Möller, A. Perkis, et al. QUALINET White Paper on Definitions of Quality of Experience. European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). 2012. 2) A. Perkis, C. Timmerer, et al. 2020. QUALINET White Paper on Definitions of Immersive Media Experience (IMEx). arXiv:2007.07032 [cs.MM] 3) Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian Timmerer, and Hermann Hellwagner. “Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression”. 27th ACM Int’l. Conf. on Multimedia (MM'19). Oct. 2019. 4) J. van der Hooft, M. T. Vega, C. Timmerer, A. C. Begen, F. De Turck and R. Schatz, "Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming," 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), Athlone, Ireland, 2020. 3) 4)
Immersive Video Delivery: From Omnidirectional Video to Holography 42 42 J. v. d. Hooft, H. Amirpour, M. Torres Vega, Y. Sanchez, R. Schatz; T. Schierl, C. Timmerer, "A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography," in IEEE Communications Surveys & Tutorials, 2023 (early access) doi: 10.1109/COMST.2023.3263252. ● 3DoF Omnidirectional Video ● 6DoF Volumetric Video ● 6DoF Imagery Video
NeRV: Neural Representations for Videos, NeurIPS 2021 Deep Learning-Based Video Coding: A Review and a Case Study, ACM CSUR 2020 ● Video Coding Learned video coding / E2E deep video coding Future of Video Streaming 43 43 An End-to-End Learning Framework for Video Compression, TPAMI 2021 ● Video Streaming DeepStream: Video Streaming Enhancements using Compressed Deep Neural Networks Deep learning-based video coding Neural implicit representation for videos DeepStream: TCSVT 2022 Generative AI for video streaming
ATHENA team Samira Afzal, Hadi Amirpour, Jesús Aguilar Armijo, Emanuele Artioli, Christian Bauer, Alexis Boniface, Ekrem Çetinkaya, Reza Ebrahimi, Alireza Erfanian, Reza Farahani, Mohammad Ghanbari (late), Milad Ghanbari, Mohammad Ghasempour, Selina Zoë Haack, Hermann Hellwagner, Manuel Hoi, Andreas Kogler, Gregor Lammer, Armin Lachini, David Langmeier, Sandro Linder, Daniele Lorenzi, Vignesh V Menon, Minh Nguyen, Engin Orhan, Lingfeng Qu, Jameson Steiner, Nina Stiller, Babak Taraghi, Farzad Tashtarian, Yuan Yuan, Yiying Wei International Collaborators (selection) ● Prof. Ali C. Begen, Ozyegin University, Turkey ● Prof. Filip De Turck, Ghent University – imec, Belgium ● Dr. Jeroen van der Hooft, Ghent University – imec, Belgium ● Dr. Maria Torres Vega, Ghent University – imec, Belgium ● Prof. Wassim Hamidouche, Technology Innovation Institute (UAE) ● Prof. Roger Zimmermann, NUS, Singapore ● Dr. Abdelhak Bentaleb, Concordia University, Canada ● Prof. Christine Guillemot, INRIA, Rennes, France ● Prof. Patrick Le Callet, Polytech Nantes-Institut Universitaire de France, France ● Prof. Junchen Jiang, University of Chicago, USA ● Dr. Sergey Gorinsky, IMDEA, Spain Acknowledgments 4 44 https://athena.itec.aau.at/
Thank you for your attention 45
46 46

More Related Content

HTTP Adaptive Streaming – Quo Vadis (2024)

  • 1. HTTP Adaptive Streaming – Quo Vadis? Christian Timmerer, Professor at AAU, Director at CD Lab ATHENA Hermann Hellwagner, Professor at AAU Klagenfurt, Austria June 27, 2024 1 Change log: ● DSRC Telecom Seminar Series’24 ● FEEC/UNICAMP Seminar in Comp. Eng.’23 ● IEEE MMTC DL Series’23 ● PCS’21 ● DDRC’21 ● ISM’20 ● WebMedia’20 https://www.slideshare.net/christian.timmerer
  • 2. The number of transistors in an integrated circuit doubles about every two years – Moore's Law (https://en.wikipedia.org/wiki/Moore%27s_law) Users' bandwidth grows by 50% per year (10% less than Moore's Law for computer speed) – Nielsen's Law of Internet Bandwidth (https://www.nngroup.com/articles/law-of-bandwidth/) Video accounts for more than 65% of the global Internet traffic – Sandvine Global Internet Phenomena (January 2023, https://www.sandvine.com/phenomena) 2
  • 3. Presenter Christian Timmerer Univ.-Prof. at Alpen-Adria-Universität Klagenfurt Director CD Lab ATHENA CIO | Head of Research and Standardization at Bitmovin 3 3 2003: MSc CS (Dipl.-Ing.) 2006: PhD CS (Dr.-techn.) 2012: Co-founded Bitmovin 2014: Habilitation (Priv.-Doz.) & Assoc. Prof. 2016: Dep. Director @ ITEC/AAU 2019: Director @ ATHENA 2022: Univ.-Prof. for Multimedia Systems Web: http://timmerer.com/ Bitmovin MPEG-DASH
  • 4. 4 4 My offices are here Copyright: AAU/Steinthaler
  • 5. ● Introduction ● ATHENA ○ Content Provisioning ○ Content Delivery ○ Content Consumption ○ End-to-End Aspects ○ Quality of Experience ● Conclusions: HAS – Quo Vadis? Agenda 5
  • 6. Introduction / Motivation Sources: * Sandvine Global Internet Phenomena (January 2024). ** Cisco Annual Internet Report (2018–2023) White Paper (March 2020) 6 6 Video streaming is dominating today’s Internet traffic ● 2024*: 68% (fixed) and 64% (mobile) ● Video on-demand: 54% / 57% – Live: 14% / 7% ● Main applications: YouTube, Netflix (>10%), Tik Tok, Amazon Prime, Disney+ (<10%) ● Video and other applications continue to be of enormous demand in today’s home, but there will be significant bandwidth demands with the application requirements of the future**
  • 7. HTTP Adaptive Streaming 101 Adaptation logic is within the client, not normatively specified by a standard, subject to research and development 7 7 Client
  • 8. Multimedia Systems Challenges and Tradeoffs 8 8 Basic figure by Klara Nahrstedt, University of Illinois at Urbana–Champaign, IEEE MIPR 2018
  • 9. “Application-oriented basic research” to address current and future research and deployment challenges of HAS and emerging streaming methods ATHENA – Adaptive Streaming over HTTP and Emerging Networked Multimedia Services Content Provisioning Content Delivery Content Consumption End-to-End Aspects ● Video encoding for HAS ● Quality-aware encoding ● Learning-based encoding ● Multi-codec HAS ● Edge computing ● Information CDN/SDN⇿clients ● Netw. assistance for/by clients ● Utility evaluation ● Bitrate adaptation schemes ● Playback improvements ● Context and user awareness ● Quality of Experience (QoE) studies ● Application/transport layer enhancements ● Quality of Experience (QoE) models ● Low-latency HAS ● Learning-based HAS https://athena.itec.aau.at/ 9 9 Funding:
  • 10. Video coding for HTTP Adaptive Streaming ● Quality improvement Per-Title Encoding (PTE) et al. for live use cases: video complexity analysis, per-title/-scene/-shot/-segment, content-/context-aware, content-adaptive, quality-aware encoding ● Runtime improvement Hardware-/software-based (cloud), parallel/distributed, information reuse from reference encodings (multi-rate/-resolution) ⇨ cf. earlier versions of this talk ● Application scenarios Video on Demand (VoD incl. diff. flavors AVoD, SVoD), live (incl. diff. flavors), interactive, games, video conferencing Content Provisioning 10 10
  • 11. Problem Statement / Basic Idea for Live Streaming ● Static bitrate ladder Pre-defined bitrate/resolution pairs for all contents ● Per-title encoding Optimize bitrate ladder based on the content ● UHD / HFR content Increase of spatial and temporal resolutions ● Live (is Life)[* Opus, 1985] Perceptual-based bitrate/resolution/framerate prediction algorithm based on content complexity Video Complexity Analyzer (VCA) 11 11 VoD Live * … https://www.youtube.com/watch?v=pATX-lV0VFk Vignesh V Menon, Christian Feldmann, Hadi Amirpour, Mohammad Ghanbari, and Christian Timmerer. 2022. VCA: video complexity analyzer. In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). Association for Computing Machinery, New York, NY, USA, 259–264. https://athena.itec.aau.at/2022/04/vca-video-complexity-analyzer/
  • 12. Spatial (E) / Temporal (h) Content Characteristics Video Complexity Analyzer (VCA) 12 12 *) Michael King, Zinovi Tauber, and Ze-Nian Li. 2007. A New Energy Function for Segmentation and Compression. In 2007 IEEE International Conference on Multimedia and Expo. 1647–1650. https://doi.org/10.1109/ICME.2007.4284983 *) VCA 2.0 (seven features): the average ● luma texture energy EY ● gradient of the luma texture energy hY ● luma brightness LY ● chroma brightness (LU and LV ) ● chroma texture energy (EU and EV ) https://athena.itec.aau.at/2023/02/vca-v2-0-released/
  • 13. VCA 2.0 Latest Results 13 13 Vignesh V Menon, Christian Feldmann, Klaus Schoeffmann, Mohammad Ghanbari, Christian Timmerer, "Green video complexity analysis for efficient encoding in Adaptive Video Streaming," ACM Green Multimedia Systems 2023, June 2023. https://athena.itec.aau.at/2023/04/green-video-complexity-analysis-for-efficient-encoding-in-adaptive-video-streaming/
  • 14. Live Per-Title Encoding Scheme 14 14 VCA: Video Complexity Analyzer (to extract/calculate E/h metrics) Github: https://github.com/cd-athena/VCA Documentation: https://cd-athena.github.io/VCA/
  • 15. Live Per-Title Encoding: Example Bitrate Ladder 15 15 VCA: Video Complexity Analyzer (to extract/calculate E/h metrics) Github: https://github.com/cd-athena/VCA Documentation: https://cd-athena.github.io/VCA/
  • 16. Live Per-Title Encoding: Example Frames (1) 16 16
  • 17. Live Per-Title Encoding: Example Frames (2) 17 17
  • 18. Results: Online Per-Title Encoding (OPTE) 18 18 V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, "OPTE: Online Per-Title Encoding for Live Video Streaming," ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022, pp. 1865-1869, doi: 10.1109/ICASSP43922.2022.9746745. https://athena.itec.aau.at/2022/01/opte-online-per-title-encoding-for-live-video-streaming/
  • 19. Perceptually-aware Per-title Encoding (PPTE) 19 19 The minimum visual difference that can be perceived by HVS, i.e., the difference between two adjacent perceptual distortion levels, refers as to one Just Noticeable Difference (JND) V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, "Perceptually-Aware Per-Title Encoding for Adaptive Video Streaming," 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan, 2022, pp. 1-6, doi: 10.1109/ICME52920.2022.9859744. https://athena.itec.aau.at/2022/05/perceptually-aware-per-title-encoding-for-adaptive-video-streaming/
  • 20. Results: Perceptually-aware Per-title Encoding (PPTE) 20 20 V. V. Menon, H. Amirpour, M. Ghanbari and C. Timmerer, "Perceptually-Aware Per-Title Encoding for Adaptive Video Streaming," 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan, 2022, pp. 1-6, doi: 10.1109/ICME52920.2022.9859744. https://athena.itec.aau.at/2022/05/perceptually-aware-per-title-encoding-for-adaptive-video-streaming/
  • 21. JND-Aware Two-Pass Per-Title Encoding Scheme for Adaptive Live Streaming (JTPS) 21 21 V. V. Menon, P. T. Rajendran, C. Feldmann, K. Schoeffmann, M. Ghanbari and C. Timmerer, "JND-Aware Two-Pass Per-Title Encoding Scheme for Adaptive Live Streaming," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 2, pp. 1281-1294, Feb. 2024, doi: 10.1109/TCSVT.2023.3290725. https://athena.itec.aau.at/2023/06/jnd-aware-two-pass-per-title-encoding-scheme-for-adaptive-live-streaming/
  • 22. ● Efficient Content-Adaptive Feature-based Shot Detection for HTTP Adaptive Streaming (IEEE ICIP 2021) ● INCEPT: INTRA CU Depth Prediction for HEVC (IEEE MMSP 2021) ● CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming (DCC 2022) ● OPTE: Online Per-title Encoding for Live Video Streaming (IEEE ICASSP 2022) ● Live-PSTR: Live Per-title Encoding for Ultra HD Adaptive Streaming (NAB BEITC 2022) ● Perceptually-aware Per-title Encoding for Adaptive Video Streaming (IEEE ICME 2023) ● Light-weight Video Encoding Complexity Prediction using Spatio Temporal Features (IEEE MMSP 2023) ● ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming (IEEE ICIP 2023) ● Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming (PCS 2023) ● Transcoding Quality Prediction for Adaptive Video Streaming (ACM MHV 2023) ● Green video complexity analysis for efficient encoding in Adaptive Video Streaming (ACM GMSys 2023) ● JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming (IEEE TCVST 2023) ● Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming (VCIP 2023) VCA-based Applications (selection) 22 22
  • 23. Network assistance for HTTP Adaptive Streaming ● Edge computing support (at CDN / cellular network edge) Functions at (or, assisted by) the edge: adaptation, analytics, (pre-)fetching, caching, transcoding, repackaging of content, request aggregation ● Server/network/CDN ↔ HAS client information exchange and collaboration IETF ALTO, MPEG SAND, MPEG NBMP, …; SDN-DASH, SDN-HAS, SABR, ... ● Use of modern network architecture features SW Defined Networking (SDN); Network Function Virtual. (NFV); MC-ABR Content Delivery 23 23
  • 24. Different policies/metrics/resource utilization ● Prefetching based on last segment quality ○ Last segment quality (LSQ) ○ Last segment quality plus (LSQ+) ○ All segment qualities (ASQ) ● Prefetching based on a Markov Model ● Prefetching based on transrating ● Prefetching Based on machine Learning (buffer size, link bitrate, prev. quality, prev. link bitrate; Random Forest, Gradient Boost, AdaBoost, Decision Trees and Extremely Randomized Trees) ● Prefetching based on super resolution Segment Prefetching and Caching at the Edge (SPACE) 24 24 J. Aguilar-Armijo, C. Timmerer and H. Hellwagner, "SPACE: Segment Prefetching and Caching at the Edge for Adaptive Video Streaming," in IEEE Access, vol. 11, pp. 21783-21798, 2023 https://athena.itec.aau.at/2023/02/space-segment-prefetching-and-caching-at-the-edge-for-adaptive-video-streaming/
  • 25. The best segment prefetching policy depends on the service provider’s preferences and resources ● Straightforward implementation with low resource utilization ⇨ LSQ and Markov-based ● Prioritize QoE at the expense of costs ⇨ ASQ, LSQ+ and Transrating-based ● Balance between performance and costs ⇨ ML-based ● Possible next steps: dynamically adapt; premium clients Segment Prefetching and Caching at the Edge (SPACE) 25 25
  • 26. Approach: ● Mechanism: Introduce a new server/segment selection approach at the edge of the network ● Main goal: Improve the users' QoE and network utilization ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming 26 26 R. Farahani, F. Tashtarian, A. Erfanian, C. Timmerer, M. Ghanbari, and H. Hellwagner. ”ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming”. The 31st edition of the Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV’21), Sept. 28-Oct. 1, 2021, Istanbul, Turkey. https://athena.itec.aau.at/2021/04/es-has-an-edge-and-sdn-assisted-framework-for-http-adaptive-video-streaming/
  • 27. Goal ● Minimizing HAS clients’ serving time & network cost, considering available resources ● Multi-layer architecture + centralized optimization model executed by SDN controller ⇨ high time complexity ● Three heuristic approaches: CG, FG-I, FG-II ● Experiments on a large-scale cloud-based testbed including 250 HAS players ⇨ improve QoE by at least 47% ⇨ decrease the streaming cost by at least 47% ⇨ enhance network utilization by at least 48% compared to state-of-the-art ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming 27 27 R. Farahani, M. Shojafar, C. Timmerer, F. Tashtarian, M. Ghanbari and H. Hellwagner, "ARARAT: A Collaborative Edge-Assisted Framework for HTTP Adaptive Video Streaming," in IEEE Transactions on Network and Service Management, vol. 20, no. 1, pp. 625-643, March 2023, doi: 10.1109/TNSM.2022.3210595. https://athena.itec.aau.at/2022/09/ararat/
  • 28. Extract some features as metadata during the encoding process ⇨ Reuse metadata in the transcoding process at the edge Light-weight Transcoding at the Edge (LwTE) 28 28 A. Erfanian, H. Amirpour, F. Tashtarian, C. Timmerer and H. Hellwagner, LwTE: Light-Weight Transcoding at the Edge," in IEEE Access, vol. 9, pp. 112276-112289, 2021 https://athena.itec.aau.at/2021/07/lwte-light-weight-transcoding-at-the-edge/ Delivery 3 4 Extract metadata 1 2 Determine optimal policy 3 Optimized download/ transcode 3 Variations: ● CD-LwTE ● LwTE-Live
  • 29. LwTE: Binary Linear Programming (BLP) Model 29 29 Inputs & Constraints: ●Videos/Segments Size ●Metadata Size ●Resources Cost ●Available Resources ●Probability Function ●Number of Incoming Requests BLP Optimization Model Outputs: ● Segments’ Serving Policy (store/transcode/fetch) Objective function: Minimize cost (computation, storage, bandwidth) and serving delay
  • 30. Performance of the proposed CD-LwTE approaches (FGH, CGH) compared with state-of-the-art approaches in terms of (a) cost, and (b) average serving delay, for various ρ values (the number of incoming requests at the edge server) CD-LwTE Comparison with State of the Art 30 30 APAC: T. X. Tran, P. Pandey, A. Hajisami, and D. Pompili, “Collaborative multibitrate video caching and processing in Mobile-Edge Computing networks,” in 2017 13th Annual Conference on Wireless On-demand Network Systems and Services (WONS), 2017, pp. 165–172. CoCache: T. X. Tran and D. Pompili, “Adaptive Bitrate Video Caching and Processing in Mobile-Edge Computing Networks,” IEEE Transactions on Mobile Computing, vol. 18, no. 9, pp. 1965–1978, 2019. PartialCache: H. Zhao, Q. Zheng, W. Zhang, B. Du, and H. Li, “A Segment-based Storage and Transcoding Trade-off Strategy for Multi-version VoD Systems in the Cloud,” IEEE Transactions on Multimedia, vol. 19, no. 1, pp. 149–159, 2016. A. Erfanian, H. Amirpour, F. Tashtarian, C. Timmerer and H. Hellwagner, "CD-LwTE: Cost-and Delay-aware Light-weight Transcoding at the Edge," in IEEE Transactions on Network and Service Management, doi: 10.1109/TNSM.2022.3229744. https://athena.itec.aau.at/2022/12/cd-lwte-cost-and-delay-aware-light-weight-transcoding-at-the-edge/ CD-LwTE CD-LwTE
  • 31. LwTE Findings 31 31 ● Stores the optimal search decisions in the encoding process as metadata ● Utilizes the metadata to avoid search processes during transcoding at the edge ● Uses partial-transcoding ● LwTE does transcoding 80% faster than H.265 ● Up to 70% cost saving compared to state-of-the-art LwTE ● Extends LwTE by relaxing assumptions, new policy, and serving delay to objective ● Adds new features in metadata ● BLP model to select optimal policy to serve requests while minimizing cost and delay ● Reduces transcoding time up to 97% streaming cost up to 75% delay up to 48% compared to state-of-the-art CD-LwTE ● Investigates LwTE’s performance in live streaming context ● MBLP model to select optimal policy (fetching and transcoding) to serve requests ● Reduces streaming cost up to 34% bandwidth up to 45% compared to state-of-the-art LwTE-Live
  • 32. Player Adaptation Logic and Quality of Experience ● Bitrate adaptation schemes Client-based, server-based, network-assisted, hybrid, ML-based ● Application/transport layer enhancements HTTP/2 (TCP) and HTTP/3 (QUIC), Media over QUIC (MOQ), proprietary formats (SRT, RIST, …), WebRTC, low-latency/delay ● Client playback improvements User-/client-aware playback, content-enhancement filters, super-resolution ● Low-latency live streaming Use of MPEG CMAF, HTTP/1.1 Chunked Transfer Encoding (CTE), other protocol features (e.g., HTTP/2 Push); LL-DASH/HLS; specific network functions; CDN support ● Quality of Experience Content Consumption and End-to-End Aspects 32 32
  • 33. Bitrate Adaptation Schemes Bitrate Adaptation Schemes Client-based Adaptation Bandwidth -based Buffer- based Mixed adaptation Proprietary solutions MDP-based Server-based Adaptation Network- assisted Adaptation Hybrid Adaptation SDN-based Server and network- assisted A. Bentaleb, B. Taani, A.C. Begen, C. Timmerer and R. Zimmermann, "A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP," IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 562-585, First Quarter 2019. 33 33 Adaptive bitrate (ABR) algorithms, adaptation logics, ...
  • 34. E-WISH: A policy driven ABR Algorithm 34 Energy Cost + 𝛅 [Data Cost] Data Cost Total Cost = ɑ Segment bitrate Throughput [Buffer Cost] Buffer Cost + β Download time Buffer occupancy [Quality Cost] Quality Cost + ɣ Video quality Distortion Instability Bitrate, Resolution, FPS (↑) M. Nguyen, E. Çetinkaya, H. Hellwagner and C. Timmerer, "WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices," 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), doi: 10.1109/MMSP53017.2021.9733605 Technology Transfer
  • 35. E-WISH: An Energy-aware ABR Algorithm 35 [Data Cost] [Quality Cost] [Buffer Cost] [Energy Cost] Data Cost Buffer Cost Quality Cost Energy Cost Total Cost = ɑ + β + ɣ + 𝛅 Segment bitrate Throughput Download time Buffer occupancy Video quality Distortion Instability Bitrate, Resolution, FPS M. Nguyen, E. Çetinkaya, H. Hellwagner and C. Timmerer, "WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices," 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), doi: 10.1109/MMSP53017.2021.9733605 D.Lorenzi, M. Nguyen, F. Tashtarian, and C. Timmerer, “E-WISH: An Energy-aware ABR Algorithm For Green HTTP Adaptive Video Streaming”, 3rd ACM Mile-High Video Conference (MHV '24), https://doi.org/10.1145/3638036.3640802
  • 36. E-WISH: An Energy-aware ABR Algorithm 36 D.Lorenzi, M. Nguyen, F. Tashtarian, and C. Timmerer, “E-WISH: An Energy-aware ABR Algorithm For Green HTTP Adaptive Video Streaming”, 3rd ACM Mile-High Video Conference (MHV '24), https://doi.org/10.1145/3638036.3640802
  • 37. Based on ACM MMSys’22 keynote by Ali C. Begen LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation 37 37 B. Taraghi, H. Hellwagner and C. Timmerer, "LLL-CAdViSE: Live Low-Latency Cloud-Based Adaptive Video Streaming Evaluation Framework," in IEEE Access, vol. 11, pp. 25723-25734, 2023 https://athena.itec.aau.at/2023/03/lll-cadvise-live-low-latency-cloud-based-adaptive-video-streaming-evaluation-framework/ Average latency and ITU-T P.1203 MOS of MPEG-DASH (dash.js) and HLS (hls.js) using three different ABR algorithms (default, L2A-LL, LoLP) with a given target latency of 3s. https://github.com/cd-athena/LLL-CAdViSE
  • 38. Live Video Streaming Pipeline 38 LIVE Origin Server (Encoding & Packaging) Live Camera Player CDN Server manifest.mpd Bitrate Resolution 5 Mbps 1920x1080 2.8 Mbps 1280x720 1.1 Mbps 960x540 System Admin Bitrate Ladder HTTP Request HTTP Response Manifest includes representations: different versions of the content optimized for various playback conditions, e.g., different bitrates and resolutions Tashtarian et al., "ARTEMIS: Adaptive Bitrate Ladder Optimization for Live Video Streaming", 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Santa Clara, CA, 2024 ● Fixed bitrate ladder ○ Content-aware, e.g., Netflix’s per-title encoding ○ Context-aware, e.g., [Lebreton and Yamagishi 2023] ○ Agnostic of the content and context (proprietary)
  • 39. ARTEMIS: Adaptive Bitrate Ladder Optimization for Live Video Streaming Select representations with bitrates that closely match the desired bitrates Players Advertise a large number of representations Encoded segments CDN Server Mega-Manifest 39 Origin Server OTL Process requests and video qualities and determine OTL LIVE Live Camera Utilize the OTL OTL: Optimized Temporary Ladder Tashtarian et al., "ARTEMIS: Adaptive Bitrate Ladder Optimization for Live Video Streaming", 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Santa Clara, CA, 2024
  • 40. QoE, Latency, and Stall with ARTEMIS vs. Five Static Bitrate Ladders 4 40 Average QoE up 11% Average Stall down 36% Average Latency down 18%
  • 41. Quality of Experience (QoE) ... ● “... is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user’s personality and current state.”1) ● … can be easily extended to various domains, e.g., immersive media experiences.2) 41 41 1) P. Le Callet, S. Möller, A. Perkis, et al. QUALINET White Paper on Definitions of Quality of Experience. European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). 2012. 2) A. Perkis, C. Timmerer, et al. 2020. QUALINET White Paper on Definitions of Immersive Media Experience (IMEx). arXiv:2007.07032 [cs.MM] 3) Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian Timmerer, and Hermann Hellwagner. “Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression”. 27th ACM Int’l. Conf. on Multimedia (MM'19). Oct. 2019. 4) J. van der Hooft, M. T. Vega, C. Timmerer, A. C. Begen, F. De Turck and R. Schatz, "Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming," 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), Athlone, Ireland, 2020. 3) 4)
  • 42. Immersive Video Delivery: From Omnidirectional Video to Holography 42 42 J. v. d. Hooft, H. Amirpour, M. Torres Vega, Y. Sanchez, R. Schatz; T. Schierl, C. Timmerer, "A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography," in IEEE Communications Surveys & Tutorials, 2023 (early access) doi: 10.1109/COMST.2023.3263252. ● 3DoF Omnidirectional Video ● 6DoF Volumetric Video ● 6DoF Imagery Video
  • 43. NeRV: Neural Representations for Videos, NeurIPS 2021 Deep Learning-Based Video Coding: A Review and a Case Study, ACM CSUR 2020 ● Video Coding Learned video coding / E2E deep video coding Future of Video Streaming 43 43 An End-to-End Learning Framework for Video Compression, TPAMI 2021 ● Video Streaming DeepStream: Video Streaming Enhancements using Compressed Deep Neural Networks Deep learning-based video coding Neural implicit representation for videos DeepStream: TCSVT 2022 Generative AI for video streaming
  • 44. ATHENA team Samira Afzal, Hadi Amirpour, Jesús Aguilar Armijo, Emanuele Artioli, Christian Bauer, Alexis Boniface, Ekrem Çetinkaya, Reza Ebrahimi, Alireza Erfanian, Reza Farahani, Mohammad Ghanbari (late), Milad Ghanbari, Mohammad Ghasempour, Selina Zoë Haack, Hermann Hellwagner, Manuel Hoi, Andreas Kogler, Gregor Lammer, Armin Lachini, David Langmeier, Sandro Linder, Daniele Lorenzi, Vignesh V Menon, Minh Nguyen, Engin Orhan, Lingfeng Qu, Jameson Steiner, Nina Stiller, Babak Taraghi, Farzad Tashtarian, Yuan Yuan, Yiying Wei International Collaborators (selection) ● Prof. Ali C. Begen, Ozyegin University, Turkey ● Prof. Filip De Turck, Ghent University – imec, Belgium ● Dr. Jeroen van der Hooft, Ghent University – imec, Belgium ● Dr. Maria Torres Vega, Ghent University – imec, Belgium ● Prof. Wassim Hamidouche, Technology Innovation Institute (UAE) ● Prof. Roger Zimmermann, NUS, Singapore ● Dr. Abdelhak Bentaleb, Concordia University, Canada ● Prof. Christine Guillemot, INRIA, Rennes, France ● Prof. Patrick Le Callet, Polytech Nantes-Institut Universitaire de France, France ● Prof. Junchen Jiang, University of Chicago, USA ● Dr. Sergey Gorinsky, IMDEA, Spain Acknowledgments 4 44 https://athena.itec.aau.at/
  • 45. Thank you for your attention 45
  • 46. 46 46