FEBench: A Benchmark for Real-Time Relational Data Feature Extraction
Authors: 
Xuanhe Zhou, Cheng Chen, Kunyi Li, Bingsheng He, Mian Lu, Qiaosheng Liu, Wei Huang, Guoliang Li, Zhao Zheng, Yuqiang ChenAuthors Info & Claims
Xuanhe Zhou, Cheng Chen, Kunyi Li, Bingsheng He, Mian Lu, Qiaosheng Liu, Wei Huang, Guoliang Li, Zhao Zheng, Yuqiang ChenAuthors Info & ClaimsPages 3597 - 3609
Abstract
As the use of online AI inference services rapidly expands in various applications (e.g., fraud detection in banking, product recommendation in e-commerce), real-time feature extraction (RTFE) systems have been developed to compute the requested features from incoming data tuples in ultra-low latency. Similar to relational databases, these RTFE procedures can be expressed using SQL-like languages. However, there is a lack of research on the workload characteristics and specialized benchmarks for RTFE, especially in comparison with existing database workloads and benchmarks (e.g., concurrent transactions in TPC-C). In this paper, we study the RTFE workload characteristics using over one hundred real datasets from open repositories (e.g. Kaggle, Tianchi, UCI ML, KiltHub) and those from 4Paradigm. The study highlights the significant differences between RTFE workloads and existing database benchmarks in terms of application scenarios, operator distributions, and query structures. Based on these findings, we propose to develop a realtime feature extraction benchmark named FEBench based on the four important criteria for a domain-specific benchmark proposed by Jim Gray. FEBench consists of selected representative datasets, query templates, and an online request simulator. We use FEBench to evaluate the effectiveness of feature extraction systems including OpenMLDB and Flink and find that each system exhibits distinct advantages and limitations in terms of overall latency, tail latency, and concurrency performance.
References
[1]
https://archive.ics.uci.edu/ml/index.php. Last accessed on 2023-2.
[2]
https://github.com/4paradigm/openmldb. Last accessed on 2023-2.
[3]
https://github.com/akopytov/sysbench. Last accessed on 2023-2.
[4]
https://github.com/alibaba/feathub. Last accessed on 2023-2.
[5]
https://github.com/feathr-ai/feathr. Last accessed on 2023-2.
[6]
https://kilthub.cmu.edu/. Last accessed on 2023-2.
[7]
https://medium.com/engineering-varo/feature-store-challenges-and-considerations-d1d59c070634. Last accessed on 2023-2.
[8]
https://tianchi.aliyun.com/. Last accessed on 2023-2.
[9]
https://www.irs.gov/pub/irs-prior/p3415--2021.pdf. Last accessed on 2023-2.
[10]
https://www.kaggle.com/competitions. Last accessed on 2023-2.
[11]
https://www.tecton.ai/. Last accessed on 2023-2.
[12]
https://www.tpc.org. Last accessed on 2023-2.
[13]
https://www.tpc.org/tpcds/. Last accessed on 2023-2.
[14]
https://www.tpc.org/tpch/. Last accessed on 2023-2.
[15]
Forecast: The business value of artificial intelligence. In Gartner, 2018.
[16]
R. Ahmed, A. W. Lee, A. Witkowski, D. Das, H. Su, M. Zaït, and T. Cruanes. Cost-based query transformation in oracle. In Proc. VLDB Endow., pages 1026--1036. ACM, 2006.
[17]
S. P. Anderson. Advertising on the internet. The Oxford handbook of the digital economy, pages 355--396, 2012.
[18]
T. G. Armstrong, V. Ponnekanti, D. Borthakur, and M. Callaghan. Linkbench: a database benchmark based on the facebook social graph. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 1185--1196, 2013.
[19]
J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez. Recommender systems survey. Knowledge-based systems, 46:109--132, 2013.
[20]
R. J. Bolton and D. J. Hand. Statistical fraud detection: A review. Statistical science, 17(3):235--255, 2002.
[21]
J. Cai, J. Luo, S. Wang, and S. Yang. Feature selection in machine learning: A new perspective. Neurocomputing, 300:70--79, 2018.
[22]
P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4), 2015.
[23]
S. Charrington. Machine learning platforms.
[24]
C. Chen, J. Yang, M. Lu, and et al. Optimizing in-memory database engine for ai-powered on-line decision augmentation using persistent memory. Proceedings of the VLDB Endowment, 14(5):799--812, 2021.
[25]
R. L. Cole, F. Funke, L. Giakoumakis, W. Guy, and et al. The mixed workload ch-benchmark. In Proceedings of the Fourth International Workshop on Testing Database Systems, DBTest 2011, Athens, Greece, June 13, 2011, page 8. ACM, 2011.
[26]
E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, pages 837--845, 1988.
[27]
D. E. Difallah, A. Pavlo, C. Curino, and P. Cudré-Mauroux. Oltp-bench: An extensible testbed for benchmarking relational databases. Proc. VLDB Endow., 7(4):277--288, 2013.
[28]
D. S. Evans. The online advertising industry: Economics, evolution, and privacy. Journal of economic perspectives, 23(3):37--60, 2009.
[29]
J. Gray, editor. The Benchmark Handbook for Database and Transaction Systems (1st Edition). Morgan Kaufmann, 1991.
[30]
I. Guyon, L. Sun-Hosoya, M. Boullé, H. J. Escalante, S. Escalera, Z. Liu, D. Jajetic, B. Ray, M. Saeed, M. Sebag, et al. Analysis of the automl challenge series. Automated Machine Learning, 177, 2019.
[31]
M. A. Hall. Correlation-based feature selection for machine learning. PhD thesis, The University of Waikato, 1999.
[32]
M. A. Hall and L. A. Smith. Practical feature subset selection for machine learning. 1998.
[33]
S. Hur and J. Kim. A survey on feature store. Electronics and Telecommunications Trends, 36(2):65--74, 2021.
[34]
G. Kang, L. Wang, W. Gao, F. Tang, and J. Zhan. Olxpbench: Real-time, semantically consistent, and domain-specific are essential in benchmarking, designing, and implementing htap systems. arXiv preprint arXiv:2203.16095, 2022.
[35]
K. Khan, S. U. Rehman, K. Aziz, S. Fong, and S. Sarasvady. Dbscan: Past, present and future. In The fifth international conference on the applications of digital information and web technologies (ICADIWT 2014), pages 232--238. IEEE, 2014.
[36]
I. Kononenko. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine, 23(1):89--109, 2001.
[37]
H. Lan, Z. data s, and Y. Peng. A survey on advancing the DBMS query optimizer: Cardinality estimation, cost model, and plan enumeration. Data Sci. Eng., 6(1):86--101, 2021.
[38]
V. Leis, A. Gubichev, A. Mirchev, P. A. Boncz, A. Kemper, and T. Neumann. How good are query optimizers, really? Proc. VLDB Endow., 9(3):204--215, 2015.
[39]
G. Li, X. Zhou, and L. Cao. AI meets database: AI4DB and DB4AI. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021, pages 2859--2866. ACM, 2021.
[40]
G. Li, X. Zhou, J. Sun, and et al. opengauss: An autonomous database system. Proc. VLDB Endow., 14(12):3028--3041, 2021.
[41]
Y. Luo, M. Wang, H. Zhou, Q. Yao, W.-W. Tu, Y. Chen, W. Dai, and Q. Yang. Autocross: Automatic feature crossing for tabular data in real-world applications. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1936--1945, 2019.
[42]
OpenJDK, 2013. https://openjdk.java.net/projects/code-tools/jmh/, Last accessed on 2020-11-15.
[43]
L. Orr, A. Sanyal, X. Ling, K. Goel, and M. Leszczynski. Managing ml pipelines: feature stores and the coming wave of embedding ecosystems. arXiv preprint arXiv:2108.05053, 2021.
[44]
T. percentile. Tp-x. https://support.huaweicloud.com/intl/en-us/productdesc-apm/apm_06_0002.html, 2019.
[45]
V. Steinbiss, B.-H. Tran, and H. Ney. Improvements in beam search. In Third international conference on spoken language processing, 1994.
[46]
C. Sun, N. Azari, and C. Turakhia. Gallery: A machine learning model management system at uber. In EDBT, pages 474--485, 2020.
[47]
J. Sun and G. Li. An end-to-end learning-based cost estimator. Proc. VLDB Endow., 13(3):307--319, 2019.
[48]
Z. Sun, X. Zhou, and G. Li. Learned index: A comprehensive experimental evaluation. Proc. VLDB Endow., 16(8):1992--2004, 2023.
[49]
Y. Tay. Data generation for application-specific benchmarking. Proceedings of the VLDB Endowment, 4(12):1470--1473, 2011.
[50]
T. Tsai. Competitive landscape: Ai startups in china. In Technical Report.
[51]
J. Wang, C. Chai, J. Liu, and G. Li. FACE: A normalizing flow based cardinality estimator. VLDB, 15(1):72--84, 2021.
[52]
S. Wang. A comprehensive survey of data mining-based accounting-fraud detection research. In 2010 International Conference on Intelligent Computation Technology and Automation, volume 1, pages 50--53. IEEE, 2010.
[53]
Wikipedia. LLVM, 2019. [Online; accessed 02-July-2022].
[54]
S. Wu, Y. Li, H. Zhu, J. Zhao, and G. Chen. Dynamic index construction with deep reinforcement learning. Data Sci. Eng., 7(2):87--101, 2022.
[55]
A. Yasin. A top-down method for performance analysis and counters architecture. In ISPASS, pages 35--44. IEEE Computer Society, 2014.
[56]
X. Yu, C. Chai, G. Li, and J. Liu. Cost-based or learning-based? A hybrid query optimizer for query plan selection. Proc. VLDB Endow., 15(13):3924--3936, 2022.
[57]
S. Zeuch, B. D. Monte, J. Karimov, C. Lutz, M. Renz, J. Traub, S. Breß, T. Rabl, and V. Markl. Analyzing efficient stream processing on modern hardware. Proceedings of the VLDB Endowment, 12(5):516--530, 2019.
[58]
S. Zhang, B. He, D. Dahlmeier, A. C. Zhou, and T. Heinze. Revisiting the design of data stream processing systems on multi-core processors. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pages 659--670, 2017.
[59]
S. Zhang, J. He, A. C. Zhou, and B. He. Briskstream: Scaling data stream processing on shared-memory multicore architectures. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD '19, page 705--722, New York, NY, USA, 2019. Association for Computing Machinery.
[60]
X. Zhou, C. Chai, G. Li, and J. Sun. Database meets artificial intelligence: A survey. IEEE Trans. Knowl. Data Eng., 34(3):1096--1116, 2022.
[61]
X. Zhou, G. Li, C. Chai, and J. Feng. A learned query rewrite system using monte carlo tree search. Proc. VLDB Endow., 15(1):46--58, 2021.
[62]
X. Zhou, G. Li, J. Wu, and et al. A learned query rewrite system. Proc. VLDB Endow., 16(12), 2023.
Recommendations
Subsetting the SPEC CPU2006 benchmark suite
On August 24, 2006, the Standard Performance Evaluation Corporation (SPEC) announced CPU2006 -- the next generation of industry-standardized CPU-intensive benchmark suite. The SPEC CPU benchmark suite has become the most frequently used suite for ...
Requirements-driven database systems benchmark method
Benchmarks are the vital tools in the performance measurement, evaluation, and comparison of relational database management systems (RDBMS). Standard benchmarks such as the TP1, TPC-A, TPC-B, TPC-C, TPC-D, TPC-H, TPC-R, TPC-W, Wisconsin, and AS3 Ap ...
A real-time benchmark for Java™
JTRES '07: Proceedings of the 5th international workshop on Java technologies for real-time and embedded systemsThe introduction of pause time sensitive garbage collectors in various Java™ Standard Edition runtime implementations and the availability of real-time Java implementations based on Real-Time Specification for Java (RTSJ, aka JSR-1) has opened the door ...
Comments
Information & Contributors
Information
Published In
Publisher
VLDB Endowment
Publication History
Published: 01 August 2023
Published in PVLDB Volume 16, Issue 12
Check for updates
Badges
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 126Total Downloads
- Downloads (Last 12 months)126
- Downloads (Last 6 weeks)4
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in