Hadoop's adolescence: An analysis of Hadoop usage in scientific workloads

K Ren, YC Kwon, M Balazinska, B Howe - Proceedings of the VLDB …, 2013 - dl.acm.org
Proceedings of the VLDB Endowment, 2013dl.acm.org
We analyze Hadoop workloads from three di? erent research clusters from a user-centric
perspective. The goal is to better understand data scientists' use of the system and how well
the use of the system matches its design. Our analysis suggests that Hadoop usage is still in
its adolescence. We see underuse of Hadoop features, extensions, and tools. We see
significant diversity in resource usage and application styles, including some interactive and
iterative workloads, motivating new tools in the ecosystem. We also observe significant …
We analyze Hadoop workloads from three di?erent research clusters from a user-centric perspective. The goal is to better understand data scientists' use of the system and how well the use of the system matches its design. Our analysis suggests that Hadoop usage is still in its adolescence. We see underuse of Hadoop features, extensions, and tools. We see significant diversity in resource usage and application styles, including some interactive and iterative workloads, motivating new tools in the ecosystem. We also observe significant opportunities for optimizations of these workloads. We find that job customization and configuration are used in a narrow scope, suggesting the future pursuit of automatic tuning systems. Overall, we present the first user-centered measurement study of Hadoop and find significant opportunities for improving its efficient use for data scientists.
ACM Digital Library