research-article

Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware

Authors:

Xiaolan J. Zhang,

Henrique Andrade,

Senthil Nathan,

Randall Schnier,

Michael Spicer,

Chitra VenkatramaniAuthors Info & Claims

WHPCF '09: Proceedings of the 2nd Workshop on High Performance Computational Finance

Article No.: 7, Pages 1 - 8

https://doi.org/10.1145/1645413.1645420

Published: 15 November 2009 Publication History

Abstract

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. We show that such a system can be built with general-purpose middleware and run on commodity hardware. The middleware we use is IBM System S, which has been augmented with transport technology from IBM WebSphere MQ Low Latency Messaging. Using eight commodity x86 blades connected with Ethernet and Infiniband, this system can achieve 80 μsec average latency at 3 times the February 2008 options market data rate and 206 μsec average latency at 15 times the February 2008 rate.

References

[1]

FAST Protocol. http://www.fixprotocol.org/fast.

[2]

IBM BladeCenter Blade Servers. http://www-03.ibm.com/systems/bladecenter/hardware/servers.

[3]

IBM Blue Gene. http://www-03.ibm.com/systems/deepcomputing/bluegene.

[4]

IBM Cell Broadband Engine. http://www-03.ibm.com/technology/cell.

[5]

IBM Wall Street Center of Excellence. http://www-03.ibm.com/systems/services/briefingcenter/wscoe.

[6]

IBM WebSphere MQ Low Latency Messaging. http://www-01.ibm.com/software/integration/wmq/llm.

[7]

IBM z/OS. http://www-03.ibm.com/systems/z/os/zos.

[8]

Infiniband Trade Association. http://www.infinibandta.org.

[9]

Options Price Reporting Authority (OPRA). http://www.opradata.com.

[10]

Red Hat Enterprise Linux 5. http://www.redhat.com/rhel.

[11]

Securities Technology Analysis Center (STAC). http://www.stacresearch.com.

[12]

Security Industry Automation Corporation (SIAC). http://www.siac.com.

[13]

V. Agarwal, D. A. Bader, L. Dan, L.-K. Liu, D. Pasetto, M. Perrone, and F. Petrini. Faster fast: multicore acceleration of streaming financial data. Computer Science - Research and Development, 23(3--4):249--257, 2009.

[14]

L. Amini, N. Jain, A. Sehgal, J. Silber, and O. Verscheure. Adaptive control of extreme-scale stream processing systems. In ICDCS. IEEE, 2006.

Digital Library

[15]

H. Andrade, B. Gedik, K.-L. Wu, and P. S. Yu. Scale-up strategies for processing high-rate data streams in System S. In ICDE. IEEE, 2009.

Digital Library

[16]

M. Feldman. High-frequency traders get boost from FPGA acceleration. HPCWire, June 2007.

[17]

S. Froehlich, M. Hack, X. Meng, and L. Zhang. Achieving precise coordinated cluster time in a cluster environment. In ISPCS. IEEE, 2008.

[18]

B. Gedik, H. Andrade, K.-L. Wu, P. S. Yu, and M. Doo. SPADE: The System S declarative stream processing engine. In SIGMOD, pages 1123--1134. ACM, 2008.

Digital Library

[19]

M. Hirzel, H. Andrade, B. Gedik, V. Kumar, G. Losa, M. Mendell, H. Nasgaard, R. Soulé, and K.-L. Wu. SPADE language specification. Technical Report RC24830, IBM, 2009.

[20]

G. Jacques-Silva, J. Challenger, L. Degenaro, J. Giles, and R. Wagle. Towards autonomic fault recovery in System S. In ICAC. IEEE, 2007.

Digital Library

[21]

N. Jain, L. Amini, H. Andrade, R. King, Y. Park, P. Selo, and C. Venkatramani. Design, implementation, and evaluation of the Linear Road Benchmark on the Stream Processing Core. In SIGMOD. ACM, 2006.

Digital Library

[22]

G. MacSweeny. Low-latency spending moves full speed ahead. Wall Street and Technology, December 2008.

[23]

J. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, and K.-L. Wu. SODA: An optimizing scheduler for large-scale stream-based distributed computer systems. In Middleware. ACM/IFIP/USENIX, 2008.

Digital Library

[24]

K.-L. Wu, P. S. Yu, B. Gedik, K. Hildrum, C. C. Aggarwal, E. Bouillet, W. Fan, D. George, X. Gu, G. Luo, and H. Wang. Challenges and experience in prototyping a multi-modal stream analytic and monitoring application on System S. In VLDB, 2007.

Digital Library

[25]

L. Zhang, Z. Liu, and C. H. Xia. Clock synchronization algorithms for network measurements. In INFOCOM. IEEE, 2002.

Cited By

Şahin SGedik B(2018)C-StreamACM Transactions on Parallel Computing10.1145/31841204:3(1-27)Online publication date: 27-Apr-2018
https://dl.acm.org/doi/10.1145/3184120
Gedik BÖzsema HÖztürk Ö(2016)Pipelined fission for stream programs with dynamic selectivity and partitioned stateJournal of Parallel and Distributed Computing10.1016/j.jpdc.2016.05.00396:C(106-120)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1016/j.jpdc.2016.05.003
Lodi GAniello LDi Luna GBaldoni R(2014)An event-based platform for collaborative threats detection and monitoringInformation Systems10.1016/j.is.2013.07.00539(175-195)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1016/j.is.2013.07.005
Show More Cited By

Index Terms

Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware
1. Computer systems organization
  1. Embedded and cyber-physical systems
  2. Real-time systems

Recommendations

Evaluation of a high-volume, low-latency market data processing system implemented with IBM middleware

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. The ...
Implementing systems management solutions using ibm director
Ibm websphere portal v4.1 handbook volume 1

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WHPCF '09: Proceedings of the 2nd Workshop on High Performance Computational Finance

November 2009

54 pages

ISBN:9781605587165

DOI:10.1145/1645413

Conference Chairs:
David Daly
IBM Thomas J. Watson Research Center
,
Maria Eleftheriou
IBM Thomas J. Watson Research Center
,
José Moreira
IBM Thomas J. Watson Research Center
,
Kyung Ryu
IBM Thomas J. Watson Research Center

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC '09

Sponsor:

SIGARCH

SC '09: International Conference for High Performance Computing, Networking, Storage and Analysis

November 15, 2009

Oregon, Portland

Acceptance Rates

Overall Acceptance Rate 8 of 10 submissions, 80%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
485
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Şahin SGedik B(2018)C-StreamACM Transactions on Parallel Computing10.1145/31841204:3(1-27)Online publication date: 27-Apr-2018
https://dl.acm.org/doi/10.1145/3184120
Gedik BÖzsema HÖztürk Ö(2016)Pipelined fission for stream programs with dynamic selectivity and partitioned stateJournal of Parallel and Distributed Computing10.1016/j.jpdc.2016.05.00396:C(106-120)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1016/j.jpdc.2016.05.003
Lodi GAniello LDi Luna GBaldoni R(2014)An event-based platform for collaborative threats detection and monitoringInformation Systems10.1016/j.is.2013.07.00539(175-195)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1016/j.is.2013.07.005
Mukherjee ABhattacharjee PMukherjee DMisra P(2012)Data Stream Management System and Capital Market SurveillanceInternational Journal of Computer Theory and Engineering10.7763/IJCTE.2012.V4.494(410-414)Online publication date: 2012
https://doi.org/10.7763/IJCTE.2012.V4.494
Aggarwal CTuraga D(2012)Mining Data StreamsMachine Learning and Knowledge Discovery for Engineering Systems Health Management10.1201/b11580-3(3-38)Online publication date: 9-Mar-2012
https://doi.org/10.1201/b11580-3
Pasetto DFranke HSchleupen KMaze DPenner HAchilles HCrawford CPurcell M(2012)Design and Implementation of a Network Centric Appliance PlatformProceedings of the 2012 Brazilian Symposium on Computing System Engineering10.1109/SBESC.2012.46(204-207)Online publication date: 5-Nov-2012
https://dl.acm.org/doi/10.1109/SBESC.2012.46
Esteves Verssimo PAniello LDi Luna GLodi GBaldoni R(2012)Collaborative Inter-domain Stealthy Port Scan Detection Using Esper Complex Event ProcessingCollaborative Financial Infrastructure Protection10.1007/978-3-642-20420-3_7(139-156)Online publication date: 2012
https://doi.org/10.1007/978-3-642-20420-3_7
Gedik BAndrade H(2012)A model-based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere StreamsSoftware—Practice & Experience10.1002/spe.113942:11(1363-1391)Online publication date: 1-Nov-2012
https://dl.acm.org/doi/10.1002/spe.1139
Park YKing RNathan SMost WAndrade H(2012)Evaluation of a high-volume, low-latency market data processing system implemented with IBM middlewareSoftware—Practice & Experience10.1002/spe.104742:1(37-56)Online publication date: 1-Jan-2012
https://dl.acm.org/doi/10.1002/spe.1047
Wagle RAndrade HHildrum KVenkatramani CSpicer MEyers DEtzion OGal AZdonik SVincent P(2011)Distributed middleware reliability and fault tolerance support in system SProceedings of the 5th ACM international conference on Distributed event-based system10.1145/2002259.2002304(335-346)Online publication date: 11-Jul-2011
https://dl.acm.org/doi/10.1145/2002259.2002304
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten