research-article

Faster Base64 Encoding and Decoding Using AVX2 Instructions

Authors:

Wojciech Muła,

Daniel LemireAuthors Info & Claims

ACM Transactions on the Web (TWEB), Volume 12, Issue 3

Article No.: 20, Pages 1 - 26

https://doi.org/10.1145/3132709

Published: 17 July 2018 Publication History

Abstract

Web developers use base64 formats to include images, fonts, sounds, and other resources directly inside HTML, JavaScript, JSON, and XML files. We estimate that billions of base64 messages are decoded every day. We are motivated to improve the efficiency of base64 encoding and decoding. Compared to state-of-the-art implementations, we multiply the speeds of both the encoding (≈ 10 ×0) and the decoding (≈ 7 ×). We achieve these good results by using the single-instruction-multiple-data instructions available on recent Intel processors (AVX2). Our accelerated software abides by the specification and reports errors when encountering characters outside of the base64 set. It is available online as free software under a liberal license.

References

[1]

Amazon 2015. Amazon SimpleDB. Retrieved from http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/Welcome.html.

[2]

Amazon 2017. Amazon DynamoDB. Retrieved from http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html.

[3]

ARM 2017. The Scalable Vector Extension (SVE), for ARMv8-A. Technical Report. ARM Holdings, Cambridge, UK. Retrieved from https://static.docs.arm.com/ddi0584/a/DDI0584A_a_SVE_supp_armv8A.pdf.

[4]

David Calhoun. 2011. When to base64 encode images (and when not to). Retrieved from http://davidbcalhoun.com/2011/when-to-base64-encode-images-and-when-not-to/.

[5]

Hanson Char. 2014. A fast and correct base 64 codec. Retrieved from https://aws.amazon.com/blogs/developer/a-fast-and-correct-base-64-codec/.

[6]

Matt Crane and Jimmy Lin. 2017. An exploration of serverless architectures for information retrieval. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR’17). ACM, New York, NY, 241--244.

Digital Library

[7]

Mark Davis. 2012. Unicode over 60 percent of the web. Retrieved from https://googleblog.blogspot.ca/2012/02/unicode-over-60-percent-of-web.html.

[8]

Elastic 2017. Elasticsearch reference-Binary datatype. Retrieved from https://www.elastic.co/guide/en/elasticsearch/reference/current/binary.html.

[9]

Pierre Estérie, Joel Falcou, Mathias Gaunard, and Jean-Thierry Lapresté. 2014. Boost.SIMD: Generic programming for portable SIMDization. In Proceedings of the Workshop on Programming Models for SIMD/Vector Processing (WPMVP’14). ACM, New York, NY, 1--8.

Digital Library

[10]

Tammy Everts. 2013. Rules for mobile performance optimization. Commun. ACM 56, 8 (Aug. 2013), 52--59.

Digital Library

[11]

Roy T. Fielding, James Gettys, Jeffrey C. Mogul, Henrik Frystyk Nielsen, Larry Masinter, Paul J. Leach, and Tim Berners-Lee. 1999. Hypertext transfer protocol-HTTP/1.1. Retrieved from https://tools.ietf.org/html/rfc2616.

Digital Library

[12]

Agner Fog. 2016. Instruction Tables: Lists of Instruction Latencies, Throughputs and Micro-operation Breakdowns for Intel, AMD and VIA CPUs. Technical Report. Copenhagen University College of Engineering, Copenhagen, Denmark. Retrieved from http://www.agner.org/optimize/instruction_tables.pdf.

[13]

Ned Freed and Nathaniel S. Borenstein. 1996. Multipurpose internet mail extensions (MIME) Part One: Format of internet message bodies. Retrieved from https://tools.ietf.org/html/rfc2045.

Digital Library

[14]

Nick Galbreath. 2016. Fast c-string transformations. Retrieved from https://github.com/client9/stringencoders.

[15]

Ian Hickson. 2016. Web storage. Retrieved from https://www.w3.org/TR/webstorage/.

[16]

Richard Duchatsch Johansen, Talita Cristina Pagani Britto, and Cesar Augusto Cusin. 2013. CSS browser selector plus: A JavaScript library to support cross-browser responsive design. In Proceedings of the 22nd International Conference on World Wide Web (WWW’13). ACM, New York, NY, 27--30.

Digital Library

[17]

Michael B. Jones, John Bradley, and Nat Sakimura. 2015. JSON web signature (JWS). Retrieved from https://tools.ietf.org/html/rfc7515.

[18]

Simon Josefsson. 2006. The base16, base32, and base64 data encodings. Retrieved from https://tools.ietf.org/html/rfc4648.

[19]

Alfred Klomp. 2014a. Fast base64 encoding/decoding with SSE vectorization. Retrieved from http://www.alfredklomp.com/programming/sse-base64/.

[20]

Alfred Klomp. 2014b. Fast base64 stream encoder/decoder in C99, with SIMD acceleration. Retrieved from https://github.com/aklomp/base64.

[21]

Nick Kopp. 2013. Base64 encoding on a GPU. Retrieved from https://www.codeproject.com/Articles/276993/Base-Encoding-on-a-GPU.

[22]

John Linn. 1993. Privacy enhancement for internet electronic mail: Part I: Message encryption and authentication procedures. Retrieved from https://tools.ietf.org/html/rfc1421.

[23]

Larry Masinter. 1998. The “data” URL scheme. Retrieved from https://tools.ietf.org/html/rfc2397.

[24]

Microsoft 2017. XML data (SQL server)-Use the binary base64 option. Retrieved from https://www.elastic.co/guide/en/elasticsearch/reference/current/binary.html.

[25]

MongoDB 2017. MongoDB extended JSON. Retrieved from https://docs.mongodb.com/manual/reference/mongodb-extended-json/.

[26]

Thomas Nägele. 2015. Client-side Performance Profiling of JavaScript for Web Applications. Master’s thesis. Radboud University Nijmegen, The Netherlands.

[27]

Gabriele Paoloni. 2010. How to Benchmark Code Execution Times on Intel IA-32 and IA-64 Instruction Set Architectures. Intel Corporation, Santa Clara, CA.

[28]

Guru Prasad Srinivasa, Rizwana Begum, Scott Haseley, Mark Hempstead, and Geoffrey Challen. 2017. Separated by birth: Hidden differences between seemingly-identical smartphone CPUs. In Proceedings of the 18th International Workshop on Mobile Computing Systems and Applications (HotMobile’17). ACM, New York, NY, 103--108.

Digital Library

[29]

Yi Tang and Manjia Lin. 2015. EQPO: Obscuring encrypted web traffic with equal-sized pseudo-objects. In Proceedings of the International Conference on Information Security and Cryptology. Springer, New York, 227--245.

Digital Library

[30]

Matt Tierney, Ian Spiro, Christoph Bregler, and Lakshminarayanan Subramanian. 2013. Cryptagram: Photo privacy for online social media. In Proceedings of the 1st ACM Conference on Online Social Networks (COSN’13). ACM, New York, NY, 75--88.

Digital Library

[31]

Haichuan Wang, Peng Wu, Ilie Gabriel Tanase, Mauricio J. Serrano, and José E. Moreira. 2014. Simple, portable, and fast SIMD intrinsic programming: Generic SIMD library. In Proceedings of the Workshop on Programming Models for SIMD/Vector Processing (WPMVP’14). ACM, New York, NY, 9--16.

Digital Library

Cited By

Schröder MMachmeier SMaeng SHeuveline V(2024)Validating CESU-8 Encoded Text Utilising SIMD InstructionsProceedings of the 2024 13th International Conference on Software and Computer Applications10.1145/3651781.3651797(102-111)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3651781.3651797
Tmeizeh MRodríguez-Domínguez CHurtado-Torres M(2024)File chunking towards on-chain storage: a blockchain-based data preservation frameworkCluster Computing10.1007/s10586-024-04646-6Online publication date: 29-Jun-2024
https://doi.org/10.1007/s10586-024-04646-6
Goswami SPatel K(2023)Network Security for IoT Device using Blockchain Technology2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT)10.1109/ICCCNT56998.2023.10308186(1-8)Online publication date: 6-Jul-2023
https://doi.org/10.1109/ICCCNT56998.2023.10308186
Show More Cited By

Index Terms

Faster Base64 Encoding and Decoding Using AVX2 Instructions
1. Theory of computation
  1. Design and analysis of algorithms
    1. Parallel algorithms
      1. Vector / streaming algorithms

Recommendations

Retargetable code optimization with SIMD instructions
CODES+ISSS '06: Proceedings of the 4th international conference on Hardware/software codesign and system synthesis

Retargetable C compilers are nowadays widely used to quickly obtain compiler support for new embedded processors and to perform early processor architecture exploration. One frequent concern about retargetable compilers, though, is their lack of machine-...
Using SIMD registers and instructions to enable instruction-level parallelism in sorting algorithms
SPAA '07: Proceedings of the nineteenth annual ACM symposium on Parallel algorithms and architectures

Most contemporary processors offer some version of Single Instruction Multiple Data (SIMD) machinery - vector registers and instructions to manipulate data stored in such registers. The central idea of this paper is to use these SIMD resources to ...
Compiler optimizations for processors with SIMD instructions

To achieve maximum efficiency, modern embedded processors for media applications exploit single instruction multiple data (SIMD) instructions. SIMD instructions provide a form of vectorization where a large machine word is viewed as a vector of subwords ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on the Web

ACM Transactions on the Web Volume 12, Issue 3

August 2018

207 pages

ISSN:1559-1131

EISSN:1559-114X

DOI:10.1145/3240924

Editor:
Brian D. Davison
Lehigh University, USA

Issue’s Table of Contents

Copyright © 2018 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2018

Accepted: 01 January 2018

Revised: 01 August 2017

Received: 01 March 2017

Published in TWEB Volume 12, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
242
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schröder MMachmeier SMaeng SHeuveline V(2024)Validating CESU-8 Encoded Text Utilising SIMD InstructionsProceedings of the 2024 13th International Conference on Software and Computer Applications10.1145/3651781.3651797(102-111)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3651781.3651797
Tmeizeh MRodríguez-Domínguez CHurtado-Torres M(2024)File chunking towards on-chain storage: a blockchain-based data preservation frameworkCluster Computing10.1007/s10586-024-04646-6Online publication date: 29-Jun-2024
https://doi.org/10.1007/s10586-024-04646-6
Goswami SPatel K(2023)Network Security for IoT Device using Blockchain Technology2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT)10.1109/ICCCNT56998.2023.10308186(1-8)Online publication date: 6-Jul-2023
https://doi.org/10.1109/ICCCNT56998.2023.10308186
Puthiyidam JJoseph SBhushan B(2023)Enhanced authentication security for IoT client nodes through T-ECDSA integrated into MQTT brokerThe Journal of Supercomputing10.1007/s11227-023-05789-w80:7(8898-8932)Online publication date: 1-Dec-2023
https://doi.org/10.1007/s11227-023-05789-w
Nizipli YLemire D(2023)Parsing millions of URLs per secondSoftware: Practice and Experience10.1002/spe.329654:5(744-758)Online publication date: 9-Dec-2023
https://doi.org/10.1002/spe.3296
Liu XGuo LYang HWei X(2022)Visible Light Positioning Based on Collaborative LEDs and Edge ComputingIEEE Transactions on Computational Social Systems10.1109/TCSS.2021.31096319:1(324-335)Online publication date: Feb-2022
https://doi.org/10.1109/TCSS.2021.3109631
Bagha AWoungang IDhurandher STraore I(2022)An Elliptic Curve Biometric Based User Authentication Protocol for Smart Homes Using SmartphonePan-African Artificial Intelligence and Smart Systems10.1007/978-3-030-93314-2_14(219-236)Online publication date: 12-Mar-2022
https://doi.org/10.1007/978-3-030-93314-2_14
Lemire DMuła W(2021)Transcoding billions of Unicode characters per second with SIMD instructionsSoftware: Practice and Experience10.1002/spe.303652:2(555-575)Online publication date: 13-Oct-2021
https://doi.org/10.1002/spe.3036
Trigo JRubio ÓMartínez-Espronceda MAlesanco ÁGarcía JSerrano-Arriezu L(2020)Building Standardized and Secure Mobile Health Services Based on Social MediaElectronics10.3390/electronics91222089:12(2208)Online publication date: 21-Dec-2020
https://doi.org/10.3390/electronics9122208
P MA B(2020)QR code based color image stego-crypto technique using dynamic bit replacement and logistic mapOptik10.1016/j.ijleo.2020.165838(165838)Online publication date: Nov-2020
https://doi.org/10.1016/j.ijleo.2020.165838
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents